1 line
523 KiB
JSON
1 line
523 KiB
JSON
{"repo": "christopherpaquin/vllm-local-developer-stack", "files": {"README.md": {"0eeb35af2b411046fbaae1a66a530f62b258e9c5": {"t": 1785746603.6561377, "n": 48487, "h": "412fbfb442b45f51", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen3-Coder-30B-A3B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ` (MoE, 30B total / ~3B active) |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens — measured, not estimated; see [Display Server Impact](#display-server-impact) before raising this |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n - [OpenCode](#opencode)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [LiteLLM Proxy Support (Optional)](#litellm-proxy-support-optional)\n - [Authentication (Optional)](#authentication-optional)\n - [Database](#database)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ ├── docker-compose.litellm.yml # Optional LiteLLM proxy service definition\n│ ├── litellm-config.yaml.example # Annotated reference for the auto-generated LiteLLM model_list\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── setup-opencode.sh # OpenCode CLI integration hook (via LiteLLM proxy)\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | Recommended value for 12 GiB RTX 3060 cards (see MEMORY & CONTEXT in `.env.example`) — needed for real headroom with this model; see [Display Server Impact](#display-server-impact) |\n| `MAX_MODEL_LEN` | 16384 | Measured, not estimated — see [Display Server Impact](#display-server-impact) for the full story (an earlier 8192/0.85-util config booted but was too small for real agentic traffic) before raising it further |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen3-coder-30b-a3b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen3-coder-30b-a3b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `qwen3-coder-30b-a3b-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n> [!TIP]\n> Zed's **Agent panel** (`Ctrl+Enter`) is a general chat window — it only\n> prints copy-paste code blocks, it doesn't touch your files. To have the\n> model edit the active file directly, highlight the target lines and\n> invoke the **Inline Assistant** with `Ctrl+I` instead.\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen3-coder-30b-a3b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and `edit-format: diff`.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\n> [!IMPORTANT]\n> The `edit-format: diff` setting is required, not optional. Aider doesn't\n> recognize local/unregistered models in its built-in model list, so\n> without it Aider silently falls back to read-only `chat` mode — it'll\n> print code suggestions but never actually write them to disk. Forcing\n> `diff` format makes the model emit unified diffs that Aider's parser can\n> apply directly.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is a terminal-native AI coding agent. Unlike\nZed/Continue/Aider above, it doesn't connect to vLLM directly — it connects\nto the [LiteLLM proxy](#litellm-proxy-support-optional), which fronts vLLM\n(and any other configured provider, e.g. DeepSeek) behind one\nOpenAI-compatible endpoint. LiteLLM must be deployed first\n(`ENABLE_LITELLM=true` in `scripts/deploy/.env`, then `deploy.sh`).\n\nRun this on whichever workstation has (or should have) OpenCode installed —\nit does not need to be the machine hosting vLLM/LiteLLM:\n\n```bash\nbash scripts/deploy/setup-opencode.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-opencode.sh 192.168.1.50:4000 # or pass the LiteLLM host directly\n```\n\nThis script:\n1. Installs OpenCode if it isn't already on `PATH` (via the official\n `curl -fsSL https://opencode.ai/install | bash` installer, after\n confirmation).\n2. Resolves the LiteLLM proxy address (command-line argument, or\n `scripts/deploy/.env`'s `LITELLM_HOST`/`LITELLM_PORT` when run on the\n LiteLLM host itself, or an interactive prompt).\n3. If the proxy has [`LITELLM_MASTER_KEY`](#authentication-optional) set,\n resolves it automatically (environment variable, or a local\n `scripts/deploy/.env` when run on the LiteLLM host itself) or prompts\n for it interactively if it detects a `401` while probing the proxy.\n4. Writes `~/.config/opencode/opencode.json`, adding the\n [`opencode-plugin-litellm`](https://github.com/yuseferi/opencode-litellm)\n plugin and a `provider.litellm` block pointed at the proxy's LAN address.\n Any existing config is backed up first\n (`opencode.json.bak.<timestamp>`), so it's safe to re-run.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `litellm` (`@ai-sdk/openai-compatible`, custom `baseURL`) |\n| API Base | `http://<litellm-host>:<litellm-port>/v1` |\n| Models | **Not hand-listed.** `opencode-plugin-litellm` queries `GET /v1/models` at every OpenCode startup and populates the model picker automatically — whatever is in LiteLLM's `model_list` (vLLM, DeepSeek, or anything else added later) shows up with zero config edits on the client side. |\n| API Key | The resolved `LITELLM_MASTER_KEY` if the proxy has auth enabled, otherwise the literal string `dummy` (unenforced, just satisfies the field) |\n\nOnce configured, run `opencode` and select a model from `/models` — no\nmanual config editing required, even after the server-side `model_list`\nchanges (re-run `deploy.sh` on the server, then just restart `opencode`).\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window (current default: 16384)\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\nAlready at `GPU_MEMORY_UTILIZATION=0.90` (the recommended value for these\ncards) — don't raise it further as a fix for OOM; halving `MAX_MODEL_LEN` is\nthe lever to pull. vLLM's own startup error reports the exact maximum\ncontext length that would have worked (e.g. `\"estimated maximum model\nlength is 14208\"`) — faster than guessing a halved value.\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nA desktop environment competing for GPU 0's VRAM can push KV cache\nallocation into an OOM at boot — this is why `GPU_MEMORY_UTILIZATION=0.90`\n(not higher) and `MAX_MODEL_LEN=16384` are the shipped defaults rather than\nmore aggressive values the hardware can otherwise support. If GPU 0 shows\n>800 MiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n`Qwen3-Coder-30B-A3B-Instruct-AWQ` genuinely has the tightest KV-cache\nheadroom of any model this repo has shipped — measured, not estimated, and\nit took two rounds of tuning to land on values that actually work in\npractice, not just at server boot:\n\n1. **`GPU_MEMORY_UTILIZATION=0.85`, `MAX_MODEL_LEN=16384`** — failed outright\n at vLLM startup:\n ```\n ValueError: To serve at least one request with the model's max seq len (16384),\n 0.75 GiB KV cache is needed, which is larger than the available KV cache\n memory (0.65 GiB). Based on the available memory, the estimated maximum\n model length is 14208.\n ```\n2. **Dropped to `MAX_MODEL_LEN=8192`** at the same `0.85` utilization —\n booted fine (vLLM reported a 14,208-token KV cache pool, 1.73x\n concurrency), but broke in real use: `MAX_MODEL_LEN` is a *combined*\n prompt+completion budget, and OpenCode requests its full declared\n `max_output_tokens` (4096) on essentially every call — any prompt over\n ~4K tokens (routine for agentic coding: tool schemas, file contents,\n conversation history) blew the 8192 ceiling, producing a\n `ContextWindowExceededError` on nearly every real request.\n3. **Raised `GPU_MEMORY_UTILIZATION` to `0.90`** — the value this repo's own\n `.env.example` already recommends for 12 GiB RTX 3060 cards; `0.85` had\n been a leftover conservative choice from tuning earlier, smaller-headroom\n models. The KV cache pool nearly doubled (1.23 GiB/GPU available, up from\n 0.65). Restored `MAX_MODEL_LEN=16384` on top of that: now boots with real\n margin — `GPU KV cache size: 26,928 tokens`, `1.64x concurrency at 16384\n tokens/request`.\n\nVerified live at the final config: tool calling with a real\n`max_tokens=4096` request (OpenCode's actual pattern) succeeds; requests\nover the ceiling still get a clean `400`, not a crash.\n\nIf GPU 0 is already headless (no idle usage), there may be a little more\nroom to raise `MAX_MODEL_LEN` further — but given how little headroom this\nmodel leaves even at `0.90`, don't guess. Run\n`bash scripts/deploy/validate-vram.sh` after any change and watch actual\nVRAM usage before committing to a higher value; if vLLM fails to start, its\nown error message reports the exact maximum context length that would have\nworked, which is the fastest way to find the real ceiling.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## LiteLLM Proxy Support (Optional)\n\nThis repository includes optional support for [LiteLLM](https://github.com/BerriAI/litellm), which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind a single OpenAI-compatible endpoint on the LAN. This is what enables [OpenCode](#opencode)'s automatic multi-model discovery — clients query LiteLLM's `/v1/models` instead of each backend individually, and don't need a hand-maintained model list.\n\nThis section covers only what this repo automates: enabling the proxy, its `model_list`, and its optional master-key auth. LiteLLM is a large project with its own extensive feature set (routing/load-balancing strategies, per-key budgets and rate limits, spend tracking, caching, guardrails, a Postgres-backed virtual-key store, Prometheus metrics, and more) — none of that is wired up here. For anything beyond basic install/config, see the [official LiteLLM documentation](https://docs.litellm.ai/) rather than this README; `deploy-artifacts/litellm-config.yaml.example` links back to the relevant docs pages inline where it matters.\n\n### Key Features\n- **One endpoint, multiple models**: `GET /v1/models` on the LiteLLM proxy lists every model in its `model_list` — the local vLLM model, plus DeepSeek when `DEEPSEEK_API_KEY` is configured.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_LITELLM=false`) to preserve the core vLLM-only focus, same as Open WebUI.\n- **Reuses vLLM's healthcheck**: The proxy container's `depends_on: vllm: condition: service_healthy` means LiteLLM only starts once vLLM's own `/health` check passes — no separate readiness mechanism to keep in sync.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled (`restart: unless-stopped`, same as every other service in this stack).\n- **Secrets never touch a generated file**: `DEEPSEEK_API_KEY` and `LITELLM_MASTER_KEY` are read by LiteLLM directly from its container environment (`os.environ/...` references in the generated config) — neither is ever written into `deploy-artifacts/litellm-config.yaml` itself, same secrets pattern as `HF_TOKEN` for vLLM.\n- **Optional auth**: no authentication by default (matches vLLM/Open WebUI), or set one master key to require `Authorization: Bearer <key>` on every request — see [Authentication](#authentication-optional) below.\n- **Always includes a Postgres database**: not independently toggleable — enables the Admin UI and virtual keys with no extra config; see [Database](#database) below.\n\n### Configuration\n\nAll LiteLLM settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable the LiteLLM proxy deployment (true/false)\nENABLE_LITELLM=true\n\n# Port on the host network where the LiteLLM proxy will listen\nLITELLM_PORT=4000\n\n# DeepSeek API key — omit to run a vLLM-only proxy\nDEEPSEEK_API_KEY=sk-your_key_here\n\n# Optional: require auth on every proxy request — see Authentication below\nLITELLM_MASTER_KEY=your-password-here\n```\n\n`deploy-artifacts/litellm-config.yaml` (the file LiteLLM actually loads) is auto-generated by `deploy.sh` from these values every run — see `deploy-artifacts/litellm-config.yaml.example` for the annotated reference. It's gitignored; don't hand-edit it.\n\n### Authentication (Optional)\n\nBy default the proxy has **no authentication** — same trusted-home-LAN posture as vLLM and Open WebUI. Set `LITELLM_MASTER_KEY` in `scripts/deploy/.env` to require `Authorization: Bearer <key>` on every proxy request, including `GET /v1/models`.\n\n- **Any string works as the key.** LiteLLM does a plain comparison — there's no required `sk-` prefix or format, despite that being a common convention in examples (including its own docs).\n- **`/health` and `/health/liveliness` stay exempt** — Docker's healthcheck doesn't send credentials, so enabling auth doesn't break container health status.\n- **Takes effect on the next `deploy.sh` run** — the key is baked into the container's environment and the generated config's `general_settings.master_key` at deploy time, not read live.\n- **Log into the LiteLLM Admin UI** at `http://<host>:<port>/ui` with any username and the master key as the password — this requires the [database](#database) below to actually be connected; without it, `/ui` login fails outright with `Authentication Error, Not connected to DB!` (not just \"no key set\"). This repo always deploys the database alongside LiteLLM, so this works out of the box. See the [official Admin UI docs](https://docs.litellm.ai/docs/proxy/ui) for what's in there.\n- **`scripts/deploy/setup-opencode.sh` resolves it automatically** — from the environment, from a local `scripts/deploy/.env` when run on the proxy host itself, or by prompting interactively if it detects a `401` while probing the proxy.\n\n```env\nLITELLM_MASTER_KEY=your-password-here\n```\n\nVerify it's working:\n```bash\n# Unauthenticated — should now fail with 401\ncurl -o /dev/null -w '%{http_code}\\n' http://10.1.10.17:4000/v1/models\n\n# Authenticated — should succeed\ncurl -H \"Authorization: Bearer your-password-here\" http://10.1.10.17:4000/v1/models\n```\n\nThis repo only wires up the single master key. LiteLLM's virtual keys (per-user/per-app keys with their own budgets, rate limits, and model access lists, issued via the master key) are a real LiteLLM feature but not something `deploy.sh` generates or manages — see the [virtual keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys) if you need that.\n\n### Database\n\nLiteLLM is always deployed with a Postgres database (`litellm-db`, `postgres:16-alpine`) — it's not independently toggleable, only `ENABLE_LITELLM` gates the whole feature. Without it, LiteLLM runs in \"DB-less mode\": model proxying and `LITELLM_MASTER_KEY` auth on API requests still work fine, but `/ui` login fails and there's no way to create virtual keys.\n\n- **Not exposed on any host port.** Only the `litellm` container can reach it, over the Docker Compose network (`litellm-db:5432`) — there's no `LITELLM_DB_PORT` to open a firewall rule for.\n- **`LITELLM_DB_PASSWORD` is auto-generated** by `deploy.sh` on first run and persisted to `scripts/deploy/.env` — an internal container-to-container credential only, never typed by a human. Generated once and kept stable across redeploys deliberately: the password is baked into the Postgres volume's own user table on first init, so a fresh random value on a later run would lock LiteLLM out of its own already-provisioned database.\n- **Persistent volume** (`LITELLM_DB_DATA_VOLUME`, default `litellm-db-data`) — survives container restarts, redeployments, and plain `teardown.sh`. Only removed by `teardown.sh --purge` (same flag that also removes Open WebUI's data — there's no way to purge just one).\n- **First boot runs LiteLLM's own Prisma migrations** against the fresh database before the proxy starts serving — this can take significantly longer than a normal restart (observed ~40s on top of regular startup on this deployment's hardware). `deploy.sh`'s post-deployment validation retries for up to 90s to account for this; subsequent redeploys against an already-migrated database are fast.\n- **Verify it's actually connected**, not just that the container is healthy (container health only checks Postgres itself, not that LiteLLM successfully attached to it) — try a DB-backed feature like minting a virtual key:\n ```bash\n curl -X POST http://10.1.10.17:4000/key/generate \\\n -H \"Authorization: Bearer your-master-key\" \\\n -H \"Content-Type: application/json\" -d '{}'\n ```\n A JSON key object back means the DB is genuinely wired up; `Authentication Error, Not connected to DB!` means it isn't.\n\nBeyond standing the database up, this repo doesn't configure anything that uses it — budgets, rate limits, teams/orgs, spend dashboards, and everything else backed by it are managed through the Admin UI or LiteLLM's own API once you're logged in; see the [official LiteLLM proxy docs](https://docs.litellm.ai/docs/proxy/) for all of that.\n\n### Deployment\n\nSimply set `ENABLE_LITELLM=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script automatically detects that LiteLLM is enabled, generates `litellm-config.yaml` from `MODEL`/`SERVED_MODEL_NAME`/`DEEPSEEK_API_KEY`, performs port availability checks, opens UFW/firewalld rules (restricted to `LAN_CIDR` if set), launches the container stack in dependency order (vLLM → `litellm-db` → LiteLLM), and validates that all of them are running, correctly configured with their restart policies, and that `/v1/models` lists the expected model(s).\n\nAfter successful deployment, the script outputs the connection URLs alongside vLLM's and Open WebUI's:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nLiteLLM Proxy : http://10.1.10.17:4000/v1\nLiteLLM UI : http://10.1.10.17:4000/ui\n```\n\nFrom there, run [`setup-opencode.sh`](#opencode) on any workstation to point OpenCode at it.\n\n### Troubleshooting\n\n- **`/v1/models` doesn't list the vLLM model**: Check `deploy-artifacts/litellm-config.yaml` (auto-generated — confirm `SERVED_MODEL_NAME` in `scripts/deploy/.env` matches) and `docker logs litellm-proxy`.\n- **`/v1/models` doesn't list DeepSeek**: `DEEPSEEK_API_KEY` is unset or invalid — `deploy.sh` prints a warning and omits the DeepSeek entry entirely rather than shipping a broken one.\n- **Cannot reach LiteLLM from another LAN host**: Ensure `LITELLM_HOST` is set to `0.0.0.0` in `scripts/deploy/.env`, and that the firewall (UFW/firewalld) is allowing `LITELLM_PORT`.\n- **LiteLLM cannot reach vLLM**: LiteLLM connects to vLLM inside the Docker network via the `vllm` service name (`http://vllm:8000/v1`), same as Open WebUI — this is hardcoded into the generated config and isn't user-configurable.\n- **`401 Unauthorized` on `/v1/models` or chat completions**: `LITELLM_MASTER_KEY` is set — every request needs `Authorization: Bearer <key>`. See [Authentication](#authentication-optional). If you didn't mean to enable auth, remove `LITELLM_MASTER_KEY` from `scripts/deploy/.env` and re-run `deploy.sh`.\n- **`/ui` login fails with `Authentication Error, Not connected to DB!`**: the `litellm-db` container isn't healthy, or `deploy.sh` hasn't run since it was added. Check `docker inspect --format='{{.State.Health.Status}}' litellm-db` and `docker logs litellm-db`; see [Database](#database) for the virtual-key verification command that confirms the connection independently of container health status.\n- **Deploy seems to hang for ~30–60s longer than usual on `litellm-proxy`**: expected on the *first* deploy after enabling LiteLLM (or after a fresh `litellm-db` volume) — LiteLLM runs its own Prisma migrations against the database before it starts serving. Subsequent redeploys are fast; `deploy.sh` already budgets for this (retries its `/v1/models` check for up to 90s).\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- LiteLLM (when enabled) also has no auth configured by default — same\n trusted-home-LAN assumption as vLLM and Open WebUI. It does, however,\n hold your `DEEPSEEK_API_KEY` in its container environment; anyone who can\n reach `LITELLM_PORT` on the LAN can spend against that key through the\n proxy. Restrict `LAN_CIDR` in `scripts/deploy/.env`, and/or set\n `LITELLM_MASTER_KEY` to require `Authorization: Bearer <key>` on every\n request — see [Authentication](#authentication-optional).\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "639e09ff3872e61d1c72aa37c0b6b50501e6e210": {"t": 1785766217.7870927, "n": 47427, "h": "1a6b492614a032c0", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen3-Coder-30B-A3B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ` (MoE, 30B total / ~3B active) |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 8,192 tokens — measured, not estimated; see [Display Server Impact](#display-server-impact) before raising this |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n - [OpenCode](#opencode)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [LiteLLM Proxy Support (Optional)](#litellm-proxy-support-optional)\n - [Authentication (Optional)](#authentication-optional)\n - [Database](#database)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ ├── docker-compose.litellm.yml # Optional LiteLLM proxy service definition\n│ ├── litellm-config.yaml.example # Annotated reference for the auto-generated LiteLLM model_list\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── setup-opencode.sh # OpenCode CLI integration hook (via LiteLLM proxy)\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.85 | Headroom for CUDA overhead + a display server on GPU 0 |\n| `MAX_MODEL_LEN` | 8192 | Measured, not estimated: weights alone leave only 0.65 GiB/GPU for KV cache — a first attempt at 16384 failed outright at vLLM startup; see [Display Server Impact](#display-server-impact) before raising it |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen3-coder-30b-a3b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen3-coder-30b-a3b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `qwen3-coder-30b-a3b-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n> [!TIP]\n> Zed's **Agent panel** (`Ctrl+Enter`) is a general chat window — it only\n> prints copy-paste code blocks, it doesn't touch your files. To have the\n> model edit the active file directly, highlight the target lines and\n> invoke the **Inline Assistant** with `Ctrl+I` instead.\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen3-coder-30b-a3b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and `edit-format: diff`.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\n> [!IMPORTANT]\n> The `edit-format: diff` setting is required, not optional. Aider doesn't\n> recognize local/unregistered models in its built-in model list, so\n> without it Aider silently falls back to read-only `chat` mode — it'll\n> print code suggestions but never actually write them to disk. Forcing\n> `diff` format makes the model emit unified diffs that Aider's parser can\n> apply directly.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is a terminal-native AI coding agent. Unlike\nZed/Continue/Aider above, it doesn't connect to vLLM directly — it connects\nto the [LiteLLM proxy](#litellm-proxy-support-optional), which fronts vLLM\n(and any other configured provider, e.g. DeepSeek) behind one\nOpenAI-compatible endpoint. LiteLLM must be deployed first\n(`ENABLE_LITELLM=true` in `scripts/deploy/.env`, then `deploy.sh`).\n\nRun this on whichever workstation has (or should have) OpenCode installed —\nit does not need to be the machine hosting vLLM/LiteLLM:\n\n```bash\nbash scripts/deploy/setup-opencode.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-opencode.sh 192.168.1.50:4000 # or pass the LiteLLM host directly\n```\n\nThis script:\n1. Installs OpenCode if it isn't already on `PATH` (via the official\n `curl -fsSL https://opencode.ai/install | bash` installer, after\n confirmation).\n2. Resolves the LiteLLM proxy address (command-line argument, or\n `scripts/deploy/.env`'s `LITELLM_HOST`/`LITELLM_PORT` when run on the\n LiteLLM host itself, or an interactive prompt).\n3. If the proxy has [`LITELLM_MASTER_KEY`](#authentication-optional) set,\n resolves it automatically (environment variable, or a local\n `scripts/deploy/.env` when run on the LiteLLM host itself) or prompts\n for it interactively if it detects a `401` while probing the proxy.\n4. Writes `~/.config/opencode/opencode.json`, adding the\n [`opencode-plugin-litellm`](https://github.com/yuseferi/opencode-litellm)\n plugin and a `provider.litellm` block pointed at the proxy's LAN address.\n Any existing config is backed up first\n (`opencode.json.bak.<timestamp>`), so it's safe to re-run.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `litellm` (`@ai-sdk/openai-compatible`, custom `baseURL`) |\n| API Base | `http://<litellm-host>:<litellm-port>/v1` |\n| Models | **Not hand-listed.** `opencode-plugin-litellm` queries `GET /v1/models` at every OpenCode startup and populates the model picker automatically — whatever is in LiteLLM's `model_list` (vLLM, DeepSeek, or anything else added later) shows up with zero config edits on the client side. |\n| API Key | The resolved `LITELLM_MASTER_KEY` if the proxy has auth enabled, otherwise the literal string `dummy` (unenforced, just satisfies the field) |\n\nOnce configured, run `opencode` and select a model from `/models` — no\nmanual config editing required, even after the server-side `model_list`\nchanges (re-run `deploy.sh` on the server, then just restart `opencode`).\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=4096 # Halve the context window (current default: 8192)\nGPU_MEMORY_UTILIZATION=0.85 # Already conservative for this model — see Display Server Impact\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\nvLLM's own startup error reports the exact maximum context length that\nwould have worked (e.g. `\"estimated maximum model length is 14208\"`) —\nfaster than guessing a halved value.\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nA desktop environment competing for GPU 0's VRAM can push KV cache\nallocation into an OOM at boot — this is why the shipped defaults use\n`GPU_MEMORY_UTILIZATION=0.85` and `MAX_MODEL_LEN=8192` rather than more\naggressive values the hardware can otherwise support. If GPU 0 shows >800\nMiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n`Qwen3-Coder-30B-A3B-Instruct-AWQ` genuinely has the tightest KV-cache\nheadroom of any model this repo has shipped — measured, not estimated.\nWeights alone (7.95 GiB/GPU) leave only 0.65 GiB/GPU for KV cache at\n`GPU_MEMORY_UTILIZATION=0.85`. A first deploy attempt at `MAX_MODEL_LEN=16384`\nfailed outright at vLLM startup:\n\n```\nValueError: To serve at least one request with the model's max seq len (16384),\n0.75 GiB KV cache is needed, which is larger than the available KV cache\nmemory (0.65 GiB). Based on the available memory, the estimated maximum\nmodel length is 14208.\n```\n\n`8192` is confirmed working with real margin, not just barely fitting one\nrequest — vLLM itself reports `GPU KV cache size: 14,208 tokens` /\n`Maximum concurrency for 8,192 tokens per request: 1.73x` (room for close to\ntwo full-length concurrent requests). Verified live: a 7,944-token request\nserved successfully; a request over the ceiling was rejected cleanly with a\n`400`, not a crash.\n\nIf GPU 0 is already headless (no idle usage), there may be a little room to\nraise `GPU_MEMORY_UTILIZATION` and/or `MAX_MODEL_LEN` in\n`scripts/deploy/.env` — but given how little headroom this model leaves,\ndon't guess. Run `bash scripts/deploy/validate-vram.sh` after any change and\nwatch actual VRAM usage before committing to a higher value; if vLLM fails\nto start, its own error message reports the exact maximum context length\nthat would have worked, which is the fastest way to find the real ceiling.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## LiteLLM Proxy Support (Optional)\n\nThis repository includes optional support for [LiteLLM](https://github.com/BerriAI/litellm), which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind a single OpenAI-compatible endpoint on the LAN. This is what enables [OpenCode](#opencode)'s automatic multi-model discovery — clients query LiteLLM's `/v1/models` instead of each backend individually, and don't need a hand-maintained model list.\n\nThis section covers only what this repo automates: enabling the proxy, its `model_list`, and its optional master-key auth. LiteLLM is a large project with its own extensive feature set (routing/load-balancing strategies, per-key budgets and rate limits, spend tracking, caching, guardrails, a Postgres-backed virtual-key store, Prometheus metrics, and more) — none of that is wired up here. For anything beyond basic install/config, see the [official LiteLLM documentation](https://docs.litellm.ai/) rather than this README; `deploy-artifacts/litellm-config.yaml.example` links back to the relevant docs pages inline where it matters.\n\n### Key Features\n- **One endpoint, multiple models**: `GET /v1/models` on the LiteLLM proxy lists every model in its `model_list` — the local vLLM model, plus DeepSeek when `DEEPSEEK_API_KEY` is configured.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_LITELLM=false`) to preserve the core vLLM-only focus, same as Open WebUI.\n- **Reuses vLLM's healthcheck**: The proxy container's `depends_on: vllm: condition: service_healthy` means LiteLLM only starts once vLLM's own `/health` check passes — no separate readiness mechanism to keep in sync.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled (`restart: unless-stopped`, same as every other service in this stack).\n- **Secrets never touch a generated file**: `DEEPSEEK_API_KEY` and `LITELLM_MASTER_KEY` are read by LiteLLM directly from its container environment (`os.environ/...` references in the generated config) — neither is ever written into `deploy-artifacts/litellm-config.yaml` itself, same secrets pattern as `HF_TOKEN` for vLLM.\n- **Optional auth**: no authentication by default (matches vLLM/Open WebUI), or set one master key to require `Authorization: Bearer <key>` on every request — see [Authentication](#authentication-optional) below.\n- **Always includes a Postgres database**: not independently toggleable — enables the Admin UI and virtual keys with no extra config; see [Database](#database) below.\n\n### Configuration\n\nAll LiteLLM settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable the LiteLLM proxy deployment (true/false)\nENABLE_LITELLM=true\n\n# Port on the host network where the LiteLLM proxy will listen\nLITELLM_PORT=4000\n\n# DeepSeek API key — omit to run a vLLM-only proxy\nDEEPSEEK_API_KEY=sk-your_key_here\n\n# Optional: require auth on every proxy request — see Authentication below\nLITELLM_MASTER_KEY=your-password-here\n```\n\n`deploy-artifacts/litellm-config.yaml` (the file LiteLLM actually loads) is auto-generated by `deploy.sh` from these values every run — see `deploy-artifacts/litellm-config.yaml.example` for the annotated reference. It's gitignored; don't hand-edit it.\n\n### Authentication (Optional)\n\nBy default the proxy has **no authentication** — same trusted-home-LAN posture as vLLM and Open WebUI. Set `LITELLM_MASTER_KEY` in `scripts/deploy/.env` to require `Authorization: Bearer <key>` on every proxy request, including `GET /v1/models`.\n\n- **Any string works as the key.** LiteLLM does a plain comparison — there's no required `sk-` prefix or format, despite that being a common convention in examples (including its own docs).\n- **`/health` and `/health/liveliness` stay exempt** — Docker's healthcheck doesn't send credentials, so enabling auth doesn't break container health status.\n- **Takes effect on the next `deploy.sh` run** — the key is baked into the container's environment and the generated config's `general_settings.master_key` at deploy time, not read live.\n- **Log into the LiteLLM Admin UI** at `http://<host>:<port>/ui` with any username and the master key as the password — this requires the [database](#database) below to actually be connected; without it, `/ui` login fails outright with `Authentication Error, Not connected to DB!` (not just \"no key set\"). This repo always deploys the database alongside LiteLLM, so this works out of the box. See the [official Admin UI docs](https://docs.litellm.ai/docs/proxy/ui) for what's in there.\n- **`scripts/deploy/setup-opencode.sh` resolves it automatically** — from the environment, from a local `scripts/deploy/.env` when run on the proxy host itself, or by prompting interactively if it detects a `401` while probing the proxy.\n\n```env\nLITELLM_MASTER_KEY=your-password-here\n```\n\nVerify it's working:\n```bash\n# Unauthenticated — should now fail with 401\ncurl -o /dev/null -w '%{http_code}\\n' http://10.1.10.17:4000/v1/models\n\n# Authenticated — should succeed\ncurl -H \"Authorization: Bearer your-password-here\" http://10.1.10.17:4000/v1/models\n```\n\nThis repo only wires up the single master key. LiteLLM's virtual keys (per-user/per-app keys with their own budgets, rate limits, and model access lists, issued via the master key) are a real LiteLLM feature but not something `deploy.sh` generates or manages — see the [virtual keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys) if you need that.\n\n### Database\n\nLiteLLM is always deployed with a Postgres database (`litellm-db`, `postgres:16-alpine`) — it's not independently toggleable, only `ENABLE_LITELLM` gates the whole feature. Without it, LiteLLM runs in \"DB-less mode\": model proxying and `LITELLM_MASTER_KEY` auth on API requests still work fine, but `/ui` login fails and there's no way to create virtual keys.\n\n- **Not exposed on any host port.** Only the `litellm` container can reach it, over the Docker Compose network (`litellm-db:5432`) — there's no `LITELLM_DB_PORT` to open a firewall rule for.\n- **`LITELLM_DB_PASSWORD` is auto-generated** by `deploy.sh` on first run and persisted to `scripts/deploy/.env` — an internal container-to-container credential only, never typed by a human. Generated once and kept stable across redeploys deliberately: the password is baked into the Postgres volume's own user table on first init, so a fresh random value on a later run would lock LiteLLM out of its own already-provisioned database.\n- **Persistent volume** (`LITELLM_DB_DATA_VOLUME`, default `litellm-db-data`) — survives container restarts, redeployments, and plain `teardown.sh`. Only removed by `teardown.sh --purge` (same flag that also removes Open WebUI's data — there's no way to purge just one).\n- **First boot runs LiteLLM's own Prisma migrations** against the fresh database before the proxy starts serving — this can take significantly longer than a normal restart (observed ~40s on top of regular startup on this deployment's hardware). `deploy.sh`'s post-deployment validation retries for up to 90s to account for this; subsequent redeploys against an already-migrated database are fast.\n- **Verify it's actually connected**, not just that the container is healthy (container health only checks Postgres itself, not that LiteLLM successfully attached to it) — try a DB-backed feature like minting a virtual key:\n ```bash\n curl -X POST http://10.1.10.17:4000/key/generate \\\n -H \"Authorization: Bearer your-master-key\" \\\n -H \"Content-Type: application/json\" -d '{}'\n ```\n A JSON key object back means the DB is genuinely wired up; `Authentication Error, Not connected to DB!` means it isn't.\n\nBeyond standing the database up, this repo doesn't configure anything that uses it — budgets, rate limits, teams/orgs, spend dashboards, and everything else backed by it are managed through the Admin UI or LiteLLM's own API once you're logged in; see the [official LiteLLM proxy docs](https://docs.litellm.ai/docs/proxy/) for all of that.\n\n### Deployment\n\nSimply set `ENABLE_LITELLM=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script automatically detects that LiteLLM is enabled, generates `litellm-config.yaml` from `MODEL`/`SERVED_MODEL_NAME`/`DEEPSEEK_API_KEY`, performs port availability checks, opens UFW/firewalld rules (restricted to `LAN_CIDR` if set), launches the container stack in dependency order (vLLM → `litellm-db` → LiteLLM), and validates that all of them are running, correctly configured with their restart policies, and that `/v1/models` lists the expected model(s).\n\nAfter successful deployment, the script outputs the connection URLs alongside vLLM's and Open WebUI's:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nLiteLLM Proxy : http://10.1.10.17:4000/v1\nLiteLLM UI : http://10.1.10.17:4000/ui\n```\n\nFrom there, run [`setup-opencode.sh`](#opencode) on any workstation to point OpenCode at it.\n\n### Troubleshooting\n\n- **`/v1/models` doesn't list the vLLM model**: Check `deploy-artifacts/litellm-config.yaml` (auto-generated — confirm `SERVED_MODEL_NAME` in `scripts/deploy/.env` matches) and `docker logs litellm-proxy`.\n- **`/v1/models` doesn't list DeepSeek**: `DEEPSEEK_API_KEY` is unset or invalid — `deploy.sh` prints a warning and omits the DeepSeek entry entirely rather than shipping a broken one.\n- **Cannot reach LiteLLM from another LAN host**: Ensure `LITELLM_HOST` is set to `0.0.0.0` in `scripts/deploy/.env`, and that the firewall (UFW/firewalld) is allowing `LITELLM_PORT`.\n- **LiteLLM cannot reach vLLM**: LiteLLM connects to vLLM inside the Docker network via the `vllm` service name (`http://vllm:8000/v1`), same as Open WebUI — this is hardcoded into the generated config and isn't user-configurable.\n- **`401 Unauthorized` on `/v1/models` or chat completions**: `LITELLM_MASTER_KEY` is set — every request needs `Authorization: Bearer <key>`. See [Authentication](#authentication-optional). If you didn't mean to enable auth, remove `LITELLM_MASTER_KEY` from `scripts/deploy/.env` and re-run `deploy.sh`.\n- **`/ui` login fails with `Authentication Error, Not connected to DB!`**: the `litellm-db` container isn't healthy, or `deploy.sh` hasn't run since it was added. Check `docker inspect --format='{{.State.Health.Status}}' litellm-db` and `docker logs litellm-db`; see [Database](#database) for the virtual-key verification command that confirms the connection independently of container health status.\n- **Deploy seems to hang for ~30–60s longer than usual on `litellm-proxy`**: expected on the *first* deploy after enabling LiteLLM (or after a fresh `litellm-db` volume) — LiteLLM runs its own Prisma migrations against the database before it starts serving. Subsequent redeploys are fast; `deploy.sh` already budgets for this (retries its `/v1/models` check for up to 90s).\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- LiteLLM (when enabled) also has no auth configured by default — same\n trusted-home-LAN assumption as vLLM and Open WebUI. It does, however,\n hold your `DEEPSEEK_API_KEY` in its container environment; anyone who can\n reach `LITELLM_PORT` on the LAN can spend against that key through the\n proxy. Restrict `LAN_CIDR` in `scripts/deploy/.env`, and/or set\n `LITELLM_MASTER_KEY` to require `Authorization: Bearer <key>` on every\n request — see [Authentication](#authentication-optional).\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "8ff935c9595600e21e4ed4915e96214a81eacc27": {"t": 1785766220.3395038, "n": 46352, "h": "baa94e28e5ba3530", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Mistral-Small-Instruct-2409-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `AMead10/Mistral-Small-Instruct-2409-awq` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (see [Display Server Impact](#display-server-impact) before raising this — this model has less KV-cache headroom than earlier smaller models this repo has shipped) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n - [OpenCode](#opencode)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [LiteLLM Proxy Support (Optional)](#litellm-proxy-support-optional)\n - [Authentication (Optional)](#authentication-optional)\n - [Database](#database)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ ├── docker-compose.litellm.yml # Optional LiteLLM proxy service definition\n│ ├── litellm-config.yaml.example # Annotated reference for the auto-generated LiteLLM model_list\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── setup-opencode.sh # OpenCode CLI integration hook (via LiteLLM proxy)\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.85 | Headroom for CUDA overhead + a display server on GPU 0 |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget — this model's ~13 GiB weight footprint leaves less headroom than earlier smaller models this repo has shipped; see [Display Server Impact](#display-server-impact) before raising it |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"mistral-small-2409-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"mistral-small-2409-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `mistral-small-2409-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n> [!TIP]\n> Zed's **Agent panel** (`Ctrl+Enter`) is a general chat window — it only\n> prints copy-paste code blocks, it doesn't touch your files. To have the\n> model edit the active file directly, highlight the target lines and\n> invoke the **Inline Assistant** with `Ctrl+I` instead.\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `mistral-small-2409-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and `edit-format: diff`.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\n> [!IMPORTANT]\n> The `edit-format: diff` setting is required, not optional. Aider doesn't\n> recognize local/unregistered models in its built-in model list, so\n> without it Aider silently falls back to read-only `chat` mode — it'll\n> print code suggestions but never actually write them to disk. Forcing\n> `diff` format makes the model emit unified diffs that Aider's parser can\n> apply directly.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is a terminal-native AI coding agent. Unlike\nZed/Continue/Aider above, it doesn't connect to vLLM directly — it connects\nto the [LiteLLM proxy](#litellm-proxy-support-optional), which fronts vLLM\n(and any other configured provider, e.g. DeepSeek) behind one\nOpenAI-compatible endpoint. LiteLLM must be deployed first\n(`ENABLE_LITELLM=true` in `scripts/deploy/.env`, then `deploy.sh`).\n\nRun this on whichever workstation has (or should have) OpenCode installed —\nit does not need to be the machine hosting vLLM/LiteLLM:\n\n```bash\nbash scripts/deploy/setup-opencode.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-opencode.sh 192.168.1.50:4000 # or pass the LiteLLM host directly\n```\n\nThis script:\n1. Installs OpenCode if it isn't already on `PATH` (via the official\n `curl -fsSL https://opencode.ai/install | bash` installer, after\n confirmation).\n2. Resolves the LiteLLM proxy address (command-line argument, or\n `scripts/deploy/.env`'s `LITELLM_HOST`/`LITELLM_PORT` when run on the\n LiteLLM host itself, or an interactive prompt).\n3. If the proxy has [`LITELLM_MASTER_KEY`](#authentication-optional) set,\n resolves it automatically (environment variable, or a local\n `scripts/deploy/.env` when run on the LiteLLM host itself) or prompts\n for it interactively if it detects a `401` while probing the proxy.\n4. Writes `~/.config/opencode/opencode.json`, adding the\n [`opencode-plugin-litellm`](https://github.com/yuseferi/opencode-litellm)\n plugin and a `provider.litellm` block pointed at the proxy's LAN address.\n Any existing config is backed up first\n (`opencode.json.bak.<timestamp>`), so it's safe to re-run.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `litellm` (`@ai-sdk/openai-compatible`, custom `baseURL`) |\n| API Base | `http://<litellm-host>:<litellm-port>/v1` |\n| Models | **Not hand-listed.** `opencode-plugin-litellm` queries `GET /v1/models` at every OpenCode startup and populates the model picker automatically — whatever is in LiteLLM's `model_list` (vLLM, DeepSeek, or anything else added later) shows up with zero config edits on the client side. |\n| API Key | The resolved `LITELLM_MASTER_KEY` if the proxy has auth enabled, otherwise the literal string `dummy` (unenforced, just satisfies the field) |\n\nOnce configured, run `opencode` and select a model from `/models` — no\nmanual config editing required, even after the server-side `model_list`\nchanges (re-run `deploy.sh` on the server, then just restart `opencode`).\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nA desktop environment competing for GPU 0's VRAM can push KV cache\nallocation into an OOM at boot — this is why the shipped defaults use\n`GPU_MEMORY_UTILIZATION=0.85` and `MAX_MODEL_LEN=16384` rather than more\naggressive values the hardware can otherwise support. If GPU 0 shows >800\nMiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\nIf GPU 0 is already headless (no idle usage), there may be room to raise\n`GPU_MEMORY_UTILIZATION` and/or `MAX_MODEL_LEN` in `scripts/deploy/.env` —\nbut unlike earlier, smaller models this repo has shipped, `16384` is not a\nconservative floor here: `Mistral-Small-Instruct-2409-awq`'s ~13 GiB weight\nfootprint (~6.5 GiB/GPU after `TENSOR_PARALLEL_SIZE=2` sharding) leaves\nmeaningfully less KV-cache headroom per 12 GiB card than earlier, smaller\nmodels did. Don't assume the old headroom — run\n`bash scripts/deploy/validate-vram.sh` after any change and watch actual\nVRAM usage before committing to a higher value.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## LiteLLM Proxy Support (Optional)\n\nThis repository includes optional support for [LiteLLM](https://github.com/BerriAI/litellm), which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind a single OpenAI-compatible endpoint on the LAN. This is what enables [OpenCode](#opencode)'s automatic multi-model discovery — clients query LiteLLM's `/v1/models` instead of each backend individually, and don't need a hand-maintained model list.\n\nThis section covers only what this repo automates: enabling the proxy, its `model_list`, and its optional master-key auth. LiteLLM is a large project with its own extensive feature set (routing/load-balancing strategies, per-key budgets and rate limits, spend tracking, caching, guardrails, a Postgres-backed virtual-key store, Prometheus metrics, and more) — none of that is wired up here. For anything beyond basic install/config, see the [official LiteLLM documentation](https://docs.litellm.ai/) rather than this README; `deploy-artifacts/litellm-config.yaml.example` links back to the relevant docs pages inline where it matters.\n\n### Key Features\n- **One endpoint, multiple models**: `GET /v1/models` on the LiteLLM proxy lists every model in its `model_list` — the local vLLM model, plus DeepSeek when `DEEPSEEK_API_KEY` is configured.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_LITELLM=false`) to preserve the core vLLM-only focus, same as Open WebUI.\n- **Reuses vLLM's healthcheck**: The proxy container's `depends_on: vllm: condition: service_healthy` means LiteLLM only starts once vLLM's own `/health` check passes — no separate readiness mechanism to keep in sync.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled (`restart: unless-stopped`, same as every other service in this stack).\n- **Secrets never touch a generated file**: `DEEPSEEK_API_KEY` and `LITELLM_MASTER_KEY` are read by LiteLLM directly from its container environment (`os.environ/...` references in the generated config) — neither is ever written into `deploy-artifacts/litellm-config.yaml` itself, same secrets pattern as `HF_TOKEN` for vLLM.\n- **Optional auth**: no authentication by default (matches vLLM/Open WebUI), or set one master key to require `Authorization: Bearer <key>` on every request — see [Authentication](#authentication-optional) below.\n- **Always includes a Postgres database**: not independently toggleable — enables the Admin UI and virtual keys with no extra config; see [Database](#database) below.\n\n### Configuration\n\nAll LiteLLM settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable the LiteLLM proxy deployment (true/false)\nENABLE_LITELLM=true\n\n# Port on the host network where the LiteLLM proxy will listen\nLITELLM_PORT=4000\n\n# DeepSeek API key — omit to run a vLLM-only proxy\nDEEPSEEK_API_KEY=sk-your_key_here\n\n# Optional: require auth on every proxy request — see Authentication below\nLITELLM_MASTER_KEY=your-password-here\n```\n\n`deploy-artifacts/litellm-config.yaml` (the file LiteLLM actually loads) is auto-generated by `deploy.sh` from these values every run — see `deploy-artifacts/litellm-config.yaml.example` for the annotated reference. It's gitignored; don't hand-edit it.\n\n### Authentication (Optional)\n\nBy default the proxy has **no authentication** — same trusted-home-LAN posture as vLLM and Open WebUI. Set `LITELLM_MASTER_KEY` in `scripts/deploy/.env` to require `Authorization: Bearer <key>` on every proxy request, including `GET /v1/models`.\n\n- **Any string works as the key.** LiteLLM does a plain comparison — there's no required `sk-` prefix or format, despite that being a common convention in examples (including its own docs).\n- **`/health` and `/health/liveliness` stay exempt** — Docker's healthcheck doesn't send credentials, so enabling auth doesn't break container health status.\n- **Takes effect on the next `deploy.sh` run** — the key is baked into the container's environment and the generated config's `general_settings.master_key` at deploy time, not read live.\n- **Log into the LiteLLM Admin UI** at `http://<host>:<port>/ui` with any username and the master key as the password — this requires the [database](#database) below to actually be connected; without it, `/ui` login fails outright with `Authentication Error, Not connected to DB!` (not just \"no key set\"). This repo always deploys the database alongside LiteLLM, so this works out of the box. See the [official Admin UI docs](https://docs.litellm.ai/docs/proxy/ui) for what's in there.\n- **`scripts/deploy/setup-opencode.sh` resolves it automatically** — from the environment, from a local `scripts/deploy/.env` when run on the proxy host itself, or by prompting interactively if it detects a `401` while probing the proxy.\n\n```env\nLITELLM_MASTER_KEY=your-password-here\n```\n\nVerify it's working:\n```bash\n# Unauthenticated — should now fail with 401\ncurl -o /dev/null -w '%{http_code}\\n' http://10.1.10.17:4000/v1/models\n\n# Authenticated — should succeed\ncurl -H \"Authorization: Bearer your-password-here\" http://10.1.10.17:4000/v1/models\n```\n\nThis repo only wires up the single master key. LiteLLM's virtual keys (per-user/per-app keys with their own budgets, rate limits, and model access lists, issued via the master key) are a real LiteLLM feature but not something `deploy.sh` generates or manages — see the [virtual keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys) if you need that.\n\n### Database\n\nLiteLLM is always deployed with a Postgres database (`litellm-db`, `postgres:16-alpine`) — it's not independently toggleable, only `ENABLE_LITELLM` gates the whole feature. Without it, LiteLLM runs in \"DB-less mode\": model proxying and `LITELLM_MASTER_KEY` auth on API requests still work fine, but `/ui` login fails and there's no way to create virtual keys.\n\n- **Not exposed on any host port.** Only the `litellm` container can reach it, over the Docker Compose network (`litellm-db:5432`) — there's no `LITELLM_DB_PORT` to open a firewall rule for.\n- **`LITELLM_DB_PASSWORD` is auto-generated** by `deploy.sh` on first run and persisted to `scripts/deploy/.env` — an internal container-to-container credential only, never typed by a human. Generated once and kept stable across redeploys deliberately: the password is baked into the Postgres volume's own user table on first init, so a fresh random value on a later run would lock LiteLLM out of its own already-provisioned database.\n- **Persistent volume** (`LITELLM_DB_DATA_VOLUME`, default `litellm-db-data`) — survives container restarts, redeployments, and plain `teardown.sh`. Only removed by `teardown.sh --purge` (same flag that also removes Open WebUI's data — there's no way to purge just one).\n- **First boot runs LiteLLM's own Prisma migrations** against the fresh database before the proxy starts serving — this can take significantly longer than a normal restart (observed ~40s on top of regular startup on this deployment's hardware). `deploy.sh`'s post-deployment validation retries for up to 90s to account for this; subsequent redeploys against an already-migrated database are fast.\n- **Verify it's actually connected**, not just that the container is healthy (container health only checks Postgres itself, not that LiteLLM successfully attached to it) — try a DB-backed feature like minting a virtual key:\n ```bash\n curl -X POST http://10.1.10.17:4000/key/generate \\\n -H \"Authorization: Bearer your-master-key\" \\\n -H \"Content-Type: application/json\" -d '{}'\n ```\n A JSON key object back means the DB is genuinely wired up; `Authentication Error, Not connected to DB!` means it isn't.\n\nBeyond standing the database up, this repo doesn't configure anything that uses it — budgets, rate limits, teams/orgs, spend dashboards, and everything else backed by it are managed through the Admin UI or LiteLLM's own API once you're logged in; see the [official LiteLLM proxy docs](https://docs.litellm.ai/docs/proxy/) for all of that.\n\n### Deployment\n\nSimply set `ENABLE_LITELLM=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script automatically detects that LiteLLM is enabled, generates `litellm-config.yaml` from `MODEL`/`SERVED_MODEL_NAME`/`DEEPSEEK_API_KEY`, performs port availability checks, opens UFW/firewalld rules (restricted to `LAN_CIDR` if set), launches the container stack in dependency order (vLLM → `litellm-db` → LiteLLM), and validates that all of them are running, correctly configured with their restart policies, and that `/v1/models` lists the expected model(s).\n\nAfter successful deployment, the script outputs the connection URLs alongside vLLM's and Open WebUI's:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nLiteLLM Proxy : http://10.1.10.17:4000/v1\nLiteLLM UI : http://10.1.10.17:4000/ui\n```\n\nFrom there, run [`setup-opencode.sh`](#opencode) on any workstation to point OpenCode at it.\n\n### Troubleshooting\n\n- **`/v1/models` doesn't list the vLLM model**: Check `deploy-artifacts/litellm-config.yaml` (auto-generated — confirm `SERVED_MODEL_NAME` in `scripts/deploy/.env` matches) and `docker logs litellm-proxy`.\n- **`/v1/models` doesn't list DeepSeek**: `DEEPSEEK_API_KEY` is unset or invalid — `deploy.sh` prints a warning and omits the DeepSeek entry entirely rather than shipping a broken one.\n- **Cannot reach LiteLLM from another LAN host**: Ensure `LITELLM_HOST` is set to `0.0.0.0` in `scripts/deploy/.env`, and that the firewall (UFW/firewalld) is allowing `LITELLM_PORT`.\n- **LiteLLM cannot reach vLLM**: LiteLLM connects to vLLM inside the Docker network via the `vllm` service name (`http://vllm:8000/v1`), same as Open WebUI — this is hardcoded into the generated config and isn't user-configurable.\n- **`401 Unauthorized` on `/v1/models` or chat completions**: `LITELLM_MASTER_KEY` is set — every request needs `Authorization: Bearer <key>`. See [Authentication](#authentication-optional). If you didn't mean to enable auth, remove `LITELLM_MASTER_KEY` from `scripts/deploy/.env` and re-run `deploy.sh`.\n- **`/ui` login fails with `Authentication Error, Not connected to DB!`**: the `litellm-db` container isn't healthy, or `deploy.sh` hasn't run since it was added. Check `docker inspect --format='{{.State.Health.Status}}' litellm-db` and `docker logs litellm-db`; see [Database](#database) for the virtual-key verification command that confirms the connection independently of container health status.\n- **Deploy seems to hang for ~30–60s longer than usual on `litellm-proxy`**: expected on the *first* deploy after enabling LiteLLM (or after a fresh `litellm-db` volume) — LiteLLM runs its own Prisma migrations against the database before it starts serving. Subsequent redeploys are fast; `deploy.sh` already budgets for this (retries its `/v1/models` check for up to 90s).\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- LiteLLM (when enabled) also has no auth configured by default — same\n trusted-home-LAN assumption as vLLM and Open WebUI. It does, however,\n hold your `DEEPSEEK_API_KEY` in its container environment; anyone who can\n reach `LITELLM_PORT` on the LAN can spend against that key through the\n proxy. Restrict `LAN_CIDR` in `scripts/deploy/.env`, and/or set\n `LITELLM_MASTER_KEY` to require `Authorization: Bearer <key>` on every\n request — see [Authentication](#authentication-optional).\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "03367814a0c2cb7482d47fab9ecb9ff13aa54521": {"t": 1785766221.9995933, "n": 42641, "h": "4d6801ef0c44ffec", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Mistral-Small-Instruct-2409-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `AMead10/Mistral-Small-Instruct-2409-awq` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (see [Display Server Impact](#display-server-impact) before raising this — this model has less KV-cache headroom than earlier smaller models this repo has shipped) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n - [OpenCode](#opencode)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [LiteLLM Proxy Support (Optional)](#litellm-proxy-support-optional)\n - [Authentication (Optional)](#authentication-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ ├── docker-compose.litellm.yml # Optional LiteLLM proxy service definition\n│ ├── litellm-config.yaml.example # Annotated reference for the auto-generated LiteLLM model_list\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── setup-opencode.sh # OpenCode CLI integration hook (via LiteLLM proxy)\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.85 | Headroom for CUDA overhead + a display server on GPU 0 |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget — this model's ~13 GiB weight footprint leaves less headroom than earlier smaller models this repo has shipped; see [Display Server Impact](#display-server-impact) before raising it |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"mistral-small-2409-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"mistral-small-2409-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `mistral-small-2409-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n> [!TIP]\n> Zed's **Agent panel** (`Ctrl+Enter`) is a general chat window — it only\n> prints copy-paste code blocks, it doesn't touch your files. To have the\n> model edit the active file directly, highlight the target lines and\n> invoke the **Inline Assistant** with `Ctrl+I` instead.\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `mistral-small-2409-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and `edit-format: diff`.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\n> [!IMPORTANT]\n> The `edit-format: diff` setting is required, not optional. Aider doesn't\n> recognize local/unregistered models in its built-in model list, so\n> without it Aider silently falls back to read-only `chat` mode — it'll\n> print code suggestions but never actually write them to disk. Forcing\n> `diff` format makes the model emit unified diffs that Aider's parser can\n> apply directly.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is a terminal-native AI coding agent. Unlike\nZed/Continue/Aider above, it doesn't connect to vLLM directly — it connects\nto the [LiteLLM proxy](#litellm-proxy-support-optional), which fronts vLLM\n(and any other configured provider, e.g. DeepSeek) behind one\nOpenAI-compatible endpoint. LiteLLM must be deployed first\n(`ENABLE_LITELLM=true` in `scripts/deploy/.env`, then `deploy.sh`).\n\nRun this on whichever workstation has (or should have) OpenCode installed —\nit does not need to be the machine hosting vLLM/LiteLLM:\n\n```bash\nbash scripts/deploy/setup-opencode.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-opencode.sh 192.168.1.50:4000 # or pass the LiteLLM host directly\n```\n\nThis script:\n1. Installs OpenCode if it isn't already on `PATH` (via the official\n `curl -fsSL https://opencode.ai/install | bash` installer, after\n confirmation).\n2. Resolves the LiteLLM proxy address (command-line argument, or\n `scripts/deploy/.env`'s `LITELLM_HOST`/`LITELLM_PORT` when run on the\n LiteLLM host itself, or an interactive prompt).\n3. If the proxy has [`LITELLM_MASTER_KEY`](#authentication-optional) set,\n resolves it automatically (environment variable, or a local\n `scripts/deploy/.env` when run on the LiteLLM host itself) or prompts\n for it interactively if it detects a `401` while probing the proxy.\n4. Writes `~/.config/opencode/opencode.json`, adding the\n [`opencode-plugin-litellm`](https://github.com/yuseferi/opencode-litellm)\n plugin and a `provider.litellm` block pointed at the proxy's LAN address.\n Any existing config is backed up first\n (`opencode.json.bak.<timestamp>`), so it's safe to re-run.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `litellm` (`@ai-sdk/openai-compatible`, custom `baseURL`) |\n| API Base | `http://<litellm-host>:<litellm-port>/v1` |\n| Models | **Not hand-listed.** `opencode-plugin-litellm` queries `GET /v1/models` at every OpenCode startup and populates the model picker automatically — whatever is in LiteLLM's `model_list` (vLLM, DeepSeek, or anything else added later) shows up with zero config edits on the client side. |\n| API Key | The resolved `LITELLM_MASTER_KEY` if the proxy has auth enabled, otherwise the literal string `dummy` (unenforced, just satisfies the field) |\n\nOnce configured, run `opencode` and select a model from `/models` — no\nmanual config editing required, even after the server-side `model_list`\nchanges (re-run `deploy.sh` on the server, then just restart `opencode`).\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nA desktop environment competing for GPU 0's VRAM can push KV cache\nallocation into an OOM at boot — this is why the shipped defaults use\n`GPU_MEMORY_UTILIZATION=0.85` and `MAX_MODEL_LEN=16384` rather than more\naggressive values the hardware can otherwise support. If GPU 0 shows >800\nMiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\nIf GPU 0 is already headless (no idle usage), there may be room to raise\n`GPU_MEMORY_UTILIZATION` and/or `MAX_MODEL_LEN` in `scripts/deploy/.env` —\nbut unlike earlier, smaller models this repo has shipped, `16384` is not a\nconservative floor here: `Mistral-Small-Instruct-2409-awq`'s ~13 GiB weight\nfootprint (~6.5 GiB/GPU after `TENSOR_PARALLEL_SIZE=2` sharding) leaves\nmeaningfully less KV-cache headroom per 12 GiB card than earlier, smaller\nmodels did. Don't assume the old headroom — run\n`bash scripts/deploy/validate-vram.sh` after any change and watch actual\nVRAM usage before committing to a higher value.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## LiteLLM Proxy Support (Optional)\n\nThis repository includes optional support for [LiteLLM](https://github.com/BerriAI/litellm), which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind a single OpenAI-compatible endpoint on the LAN. This is what enables [OpenCode](#opencode)'s automatic multi-model discovery — clients query LiteLLM's `/v1/models` instead of each backend individually, and don't need a hand-maintained model list.\n\nThis section covers only what this repo automates: enabling the proxy, its `model_list`, and its optional master-key auth. LiteLLM is a large project with its own extensive feature set (routing/load-balancing strategies, per-key budgets and rate limits, spend tracking, caching, guardrails, a Postgres-backed virtual-key store, Prometheus metrics, and more) — none of that is wired up here. For anything beyond basic install/config, see the [official LiteLLM documentation](https://docs.litellm.ai/) rather than this README; `deploy-artifacts/litellm-config.yaml.example` links back to the relevant docs pages inline where it matters.\n\n### Key Features\n- **One endpoint, multiple models**: `GET /v1/models` on the LiteLLM proxy lists every model in its `model_list` — the local vLLM model, plus DeepSeek when `DEEPSEEK_API_KEY` is configured.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_LITELLM=false`) to preserve the core vLLM-only focus, same as Open WebUI.\n- **Reuses vLLM's healthcheck**: The proxy container's `depends_on: vllm: condition: service_healthy` means LiteLLM only starts once vLLM's own `/health` check passes — no separate readiness mechanism to keep in sync.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled (`restart: unless-stopped`, same as every other service in this stack).\n- **Secrets never touch a generated file**: `DEEPSEEK_API_KEY` and `LITELLM_MASTER_KEY` are read by LiteLLM directly from its container environment (`os.environ/...` references in the generated config) — neither is ever written into `deploy-artifacts/litellm-config.yaml` itself, same secrets pattern as `HF_TOKEN` for vLLM.\n- **Optional auth**: no authentication by default (matches vLLM/Open WebUI), or set one master key to require `Authorization: Bearer <key>` on every request — see [Authentication](#authentication-optional) below.\n\n### Configuration\n\nAll LiteLLM settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable the LiteLLM proxy deployment (true/false)\nENABLE_LITELLM=true\n\n# Port on the host network where the LiteLLM proxy will listen\nLITELLM_PORT=4000\n\n# DeepSeek API key — omit to run a vLLM-only proxy\nDEEPSEEK_API_KEY=sk-your_key_here\n\n# Optional: require auth on every proxy request — see Authentication below\nLITELLM_MASTER_KEY=your-password-here\n```\n\n`deploy-artifacts/litellm-config.yaml` (the file LiteLLM actually loads) is auto-generated by `deploy.sh` from these values every run — see `deploy-artifacts/litellm-config.yaml.example` for the annotated reference. It's gitignored; don't hand-edit it.\n\n### Authentication (Optional)\n\nBy default the proxy has **no authentication** — same trusted-home-LAN posture as vLLM and Open WebUI. Set `LITELLM_MASTER_KEY` in `scripts/deploy/.env` to require `Authorization: Bearer <key>` on every proxy request, including `GET /v1/models`.\n\n- **Any string works as the key.** LiteLLM does a plain comparison — there's no required `sk-` prefix or format, despite that being a common convention in examples (including its own docs).\n- **`/health` and `/health/liveliness` stay exempt** — Docker's healthcheck doesn't send credentials, so enabling auth doesn't break container health status.\n- **Takes effect on the next `deploy.sh` run** — the key is baked into the container's environment and the generated config's `general_settings.master_key` at deploy time, not read live.\n- **Also unlocks the LiteLLM Admin UI** at `http://<host>:<port>/ui` — log in with any username and the master key as the password. This UI is where LiteLLM's virtual-key/budget/rate-limit features (out of scope here) actually live; see the [official Admin UI docs](https://docs.litellm.ai/docs/proxy/ui) if you want to use them.\n- **`scripts/deploy/setup-opencode.sh` resolves it automatically** — from the environment, from a local `scripts/deploy/.env` when run on the proxy host itself, or by prompting interactively if it detects a `401` while probing the proxy.\n\n```env\nLITELLM_MASTER_KEY=your-password-here\n```\n\nVerify it's working:\n```bash\n# Unauthenticated — should now fail with 401\ncurl -o /dev/null -w '%{http_code}\\n' http://10.1.10.17:4000/v1/models\n\n# Authenticated — should succeed\ncurl -H \"Authorization: Bearer your-password-here\" http://10.1.10.17:4000/v1/models\n```\n\nThis repo only wires up the single master key. LiteLLM's virtual keys (per-user/per-app keys with their own budgets, rate limits, and model access lists, issued via the master key) are a real LiteLLM feature but not something `deploy.sh` generates or manages — see the [virtual keys docs](https://docs.litellm.ai/docs/proxy/virtual_keys) if you need that.\n\n### Deployment\n\nSimply set `ENABLE_LITELLM=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script automatically detects that LiteLLM is enabled, generates `litellm-config.yaml` from `MODEL`/`SERVED_MODEL_NAME`/`DEEPSEEK_API_KEY`, performs port availability checks, opens UFW/firewalld rules (restricted to `LAN_CIDR` if set), launches the container stack in dependency order (vLLM → LiteLLM), and validates that both are running, correctly configured with their restart policies, and that `/v1/models` lists the expected model(s).\n\nAfter successful deployment, the script outputs the connection URL alongside vLLM's and Open WebUI's:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nLiteLLM Proxy : http://10.1.10.17:4000/v1\n```\n\nFrom there, run [`setup-opencode.sh`](#opencode) on any workstation to point OpenCode at it.\n\n### Troubleshooting\n\n- **`/v1/models` doesn't list the vLLM model**: Check `deploy-artifacts/litellm-config.yaml` (auto-generated — confirm `SERVED_MODEL_NAME` in `scripts/deploy/.env` matches) and `docker logs litellm-proxy`.\n- **`/v1/models` doesn't list DeepSeek**: `DEEPSEEK_API_KEY` is unset or invalid — `deploy.sh` prints a warning and omits the DeepSeek entry entirely rather than shipping a broken one.\n- **Cannot reach LiteLLM from another LAN host**: Ensure `LITELLM_HOST` is set to `0.0.0.0` in `scripts/deploy/.env`, and that the firewall (UFW/firewalld) is allowing `LITELLM_PORT`.\n- **LiteLLM cannot reach vLLM**: LiteLLM connects to vLLM inside the Docker network via the `vllm` service name (`http://vllm:8000/v1`), same as Open WebUI — this is hardcoded into the generated config and isn't user-configurable.\n- **`401 Unauthorized` on `/v1/models` or chat completions**: `LITELLM_MASTER_KEY` is set — every request needs `Authorization: Bearer <key>`. See [Authentication](#authentication-optional). If you didn't mean to enable auth, remove `LITELLM_MASTER_KEY` from `scripts/deploy/.env` and re-run `deploy.sh`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- LiteLLM (when enabled) also has no auth configured by default — same\n trusted-home-LAN assumption as vLLM and Open WebUI. It does, however,\n hold your `DEEPSEEK_API_KEY` in its container environment; anyone who can\n reach `LITELLM_PORT` on the LAN can spend against that key through the\n proxy. Restrict `LAN_CIDR` in `scripts/deploy/.env`, and/or set\n `LITELLM_MASTER_KEY` to require `Authorization: Bearer <key>` on every\n request — see [Authentication](#authentication-optional).\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "cbfecdf128ce904c348075646b01265d7255f66a": {"t": 1785766223.447313, "n": 38354, "h": "fb83d0a7718780db", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-14B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-14B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (can be raised to 32,768 on a headless GPU 0 — see [Display Server Impact](#display-server-impact)) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n - [OpenCode](#opencode)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [LiteLLM Proxy Support (Optional)](#litellm-proxy-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ ├── docker-compose.litellm.yml # Optional LiteLLM proxy service definition\n│ ├── litellm-config.yaml.example # Annotated reference for the auto-generated LiteLLM model_list\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── setup-opencode.sh # OpenCode CLI integration hook (via LiteLLM proxy)\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.85 | Headroom for CUDA overhead + a display server on GPU 0 |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget (14B model; see [Display Server Impact](#display-server-impact) for pushing this to 32768) |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-14b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-14b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `qwen2.5-coder-14b-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n> [!TIP]\n> Zed's **Agent panel** (`Ctrl+Enter`) is a general chat window — it only\n> prints copy-paste code blocks, it doesn't touch your files. To have the\n> model edit the active file directly, highlight the target lines and\n> invoke the **Inline Assistant** with `Ctrl+I` instead.\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-14b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and `edit-format: diff`.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\n> [!IMPORTANT]\n> The `edit-format: diff` setting is required, not optional. Aider doesn't\n> recognize local/unregistered models in its built-in model list, so\n> without it Aider silently falls back to read-only `chat` mode — it'll\n> print code suggestions but never actually write them to disk. Forcing\n> `diff` format makes the model emit unified diffs that Aider's parser can\n> apply directly.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n### OpenCode\n\n[OpenCode](https://opencode.ai) is a terminal-native AI coding agent. Unlike\nZed/Continue/Aider above, it doesn't connect to vLLM directly — it connects\nto the [LiteLLM proxy](#litellm-proxy-support-optional), which fronts vLLM\n(and any other configured provider, e.g. DeepSeek) behind one\nOpenAI-compatible endpoint. LiteLLM must be deployed first\n(`ENABLE_LITELLM=true` in `scripts/deploy/.env`, then `deploy.sh`).\n\nRun this on whichever workstation has (or should have) OpenCode installed —\nit does not need to be the machine hosting vLLM/LiteLLM:\n\n```bash\nbash scripts/deploy/setup-opencode.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-opencode.sh 192.168.1.50:4000 # or pass the LiteLLM host directly\n```\n\nThis script:\n1. Installs OpenCode if it isn't already on `PATH` (via the official\n `curl -fsSL https://opencode.ai/install | bash` installer, after\n confirmation).\n2. Resolves the LiteLLM proxy address (command-line argument, or\n `scripts/deploy/.env`'s `LITELLM_HOST`/`LITELLM_PORT` when run on the\n LiteLLM host itself, or an interactive prompt).\n3. Writes `~/.config/opencode/opencode.json`, adding the\n [`opencode-plugin-litellm`](https://github.com/yuseferi/opencode-litellm)\n plugin and a `provider.litellm` block pointed at the proxy's LAN address.\n Any existing config is backed up first\n (`opencode.json.bak.<timestamp>`), so it's safe to re-run.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `litellm` (`@ai-sdk/openai-compatible`, custom `baseURL`) |\n| API Base | `http://<litellm-host>:<litellm-port>/v1` |\n| Models | **Not hand-listed.** `opencode-plugin-litellm` queries `GET /v1/models` at every OpenCode startup and populates the model picker automatically — whatever is in LiteLLM's `model_list` (vLLM, DeepSeek, or anything else added later) shows up with zero config edits on the client side. |\n| API Key | Not enforced by this LiteLLM deployment — the script writes `dummy` |\n\nOnce configured, run `opencode` and select a model from `/models` — no\nmanual config editing required, even after the server-side `model_list`\nchanges (re-run `deploy.sh` on the server, then just restart `opencode`).\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nEven with the smaller 14B model, a desktop environment competing for GPU\n0's VRAM can push KV cache allocation into an OOM at boot — this is why the\nshipped defaults use `GPU_MEMORY_UTILIZATION=0.85` and `MAX_MODEL_LEN=16384`\nrather than the more aggressive `0.90`/`32768` the hardware can otherwise\nsupport. If GPU 0 shows >800 MiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\nIf GPU 0 is already headless (no idle usage), you can raise\n`GPU_MEMORY_UTILIZATION` back to `0.90` and `MAX_MODEL_LEN` to `32768` in\n`scripts/deploy/.env` for the full context window.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## LiteLLM Proxy Support (Optional)\n\nThis repository includes optional support for [LiteLLM](https://github.com/BerriAI/litellm), which fronts vLLM (and any other configured provider, e.g. DeepSeek) behind a single OpenAI-compatible endpoint on the LAN. This is what enables [OpenCode](#opencode)'s automatic multi-model discovery — clients query LiteLLM's `/v1/models` instead of each backend individually, and don't need a hand-maintained model list.\n\n### Key Features\n- **One endpoint, multiple models**: `GET /v1/models` on the LiteLLM proxy lists every model in its `model_list` — the local vLLM model, plus DeepSeek when `DEEPSEEK_API_KEY` is configured.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_LITELLM=false`) to preserve the core vLLM-only focus, same as Open WebUI.\n- **Reuses vLLM's healthcheck**: The proxy container's `depends_on: vllm: condition: service_healthy` means LiteLLM only starts once vLLM's own `/health` check passes — no separate readiness mechanism to keep in sync.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled (`restart: unless-stopped`, same as every other service in this stack).\n- **Secrets never touch a generated file**: `DEEPSEEK_API_KEY` is read by LiteLLM directly from its container environment (`os.environ/DEEPSEEK_API_KEY` in the generated config) — the key itself is never written into `deploy-artifacts/litellm-config.yaml`, same secrets pattern as `HF_TOKEN` for vLLM.\n\n### Configuration\n\nAll LiteLLM settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable the LiteLLM proxy deployment (true/false)\nENABLE_LITELLM=true\n\n# Port on the host network where the LiteLLM proxy will listen\nLITELLM_PORT=4000\n\n# DeepSeek API key — omit to run a vLLM-only proxy\nDEEPSEEK_API_KEY=sk-your_key_here\n```\n\n`deploy-artifacts/litellm-config.yaml` (the file LiteLLM actually loads) is auto-generated by `deploy.sh` from these values every run — see `deploy-artifacts/litellm-config.yaml.example` for the annotated reference. It's gitignored; don't hand-edit it.\n\n### Deployment\n\nSimply set `ENABLE_LITELLM=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script automatically detects that LiteLLM is enabled, generates `litellm-config.yaml` from `MODEL`/`SERVED_MODEL_NAME`/`DEEPSEEK_API_KEY`, performs port availability checks, opens UFW/firewalld rules (restricted to `LAN_CIDR` if set), launches the container stack in dependency order (vLLM → LiteLLM), and validates that both are running, correctly configured with their restart policies, and that `/v1/models` lists the expected model(s).\n\nAfter successful deployment, the script outputs the connection URL alongside vLLM's and Open WebUI's:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nLiteLLM Proxy : http://10.1.10.17:4000/v1\n```\n\nFrom there, run [`setup-opencode.sh`](#opencode) on any workstation to point OpenCode at it.\n\n### Troubleshooting\n\n- **`/v1/models` doesn't list the vLLM model**: Check `deploy-artifacts/litellm-config.yaml` (auto-generated — confirm `SERVED_MODEL_NAME` in `scripts/deploy/.env` matches) and `docker logs litellm-proxy`.\n- **`/v1/models` doesn't list DeepSeek**: `DEEPSEEK_API_KEY` is unset or invalid — `deploy.sh` prints a warning and omits the DeepSeek entry entirely rather than shipping a broken one.\n- **Cannot reach LiteLLM from another LAN host**: Ensure `LITELLM_HOST` is set to `0.0.0.0` in `scripts/deploy/.env`, and that the firewall (UFW/firewalld) is allowing `LITELLM_PORT`.\n- **LiteLLM cannot reach vLLM**: LiteLLM connects to vLLM inside the Docker network via the `vllm` service name (`http://vllm:8000/v1`), same as Open WebUI — this is hardcoded into the generated config and isn't user-configurable.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- LiteLLM (when enabled) also has no `master_key`/auth configured by\n default — same trusted-home-LAN assumption as vLLM and Open WebUI. It\n does, however, hold your `DEEPSEEK_API_KEY` in its container environment;\n anyone who can reach `LITELLM_PORT` on the LAN can spend against that key\n through the proxy. Restrict `LAN_CIDR` in `scripts/deploy/.env` if that's\n a concern, or set `litellm_settings.master_key` in\n `deploy-artifacts/litellm-config.yaml.example` / regenerate with auth if\n you need it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "0c2e21a0cd84bd7aed949a01732d050c0cbcee48": {"t": 1785766225.277513, "n": 31372, "h": "acaff289218caeb4", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-14B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-14B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (can be raised to 32,768 on a headless GPU 0 — see [Display Server Impact](#display-server-impact)) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.85 | Headroom for CUDA overhead + a display server on GPU 0 |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget (14B model; see [Display Server Impact](#display-server-impact) for pushing this to 32768) |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-14b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-14b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `qwen2.5-coder-14b-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n> [!TIP]\n> Zed's **Agent panel** (`Ctrl+Enter`) is a general chat window — it only\n> prints copy-paste code blocks, it doesn't touch your files. To have the\n> model edit the active file directly, highlight the target lines and\n> invoke the **Inline Assistant** with `Ctrl+I` instead.\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-14b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, patching the OpenAI-compatible API base URL, API key, model, and `edit-format: diff`.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\n> [!IMPORTANT]\n> The `edit-format: diff` setting is required, not optional. Aider doesn't\n> recognize local/unregistered models in its built-in model list, so\n> without it Aider silently falls back to read-only `chat` mode — it'll\n> print code suggestions but never actually write them to disk. Forcing\n> `diff` format makes the model emit unified diffs that Aider's parser can\n> apply directly.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nEven with the smaller 14B model, a desktop environment competing for GPU\n0's VRAM can push KV cache allocation into an OOM at boot — this is why the\nshipped defaults use `GPU_MEMORY_UTILIZATION=0.85` and `MAX_MODEL_LEN=16384`\nrather than the more aggressive `0.90`/`32768` the hardware can otherwise\nsupport. If GPU 0 shows >800 MiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\nIf GPU 0 is already headless (no idle usage), you can raise\n`GPU_MEMORY_UTILIZATION` back to `0.90` and `MAX_MODEL_LEN` to `32768` in\n`scripts/deploy/.env` for the full context window.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "eefae2cdc0b17e37bb2fb4d6f3ad58ba41f805dd": {"t": 1785766226.8602877, "n": 30698, "h": "36a6ca83d7b6260c", "c": "# vllm-local-developer-stack\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-14B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-14B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (can be raised to 32,768 on a headless GPU 0 — see [Display Server Impact](#display-server-impact)) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect editor/CLI tools such as Zed, Continue (VS Code/JetBrains), or Aider\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-scriptsdeployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Client & Editor Integrations](#client--editor-integrations)\n - [Zed](#zed)\n - [Continue (VS Code & JetBrains)](#continue-vs-code--jetbrains)\n - [Aider](#aider)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-local-developer-stack/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-zed.sh # Zed IDE integration hook (primary supported IDE)\n │ ├── setup-continue.sh # Continue extension hook (VS Code & JetBrains)\n │ ├── setup-aider.sh # Aider CLI integration hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.85 | Headroom for CUDA overhead + a display server on GPU 0 |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget (14B model; see [Display Server Impact](#display-server-impact) for pushing this to 32768) |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Client & Editor Integrations](#client--editor-integrations) below.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-14b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-14b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Client & Editor Integrations\n\nThese are client-side setup steps, separate from deploying, tuning, or\nbenchmarking the server. Each script below configures one editor/tool to\npoint at your running vLLM endpoint — run interactively (it prompts, or\nreads `BIND_HOST:PORT` from `.env` if present) or with the host passed\ndirectly as an argument. Run a given script on **whichever workstation has\nthat tool installed** — it does not need to be the machine hosting vLLM.\n\n### Zed\n\nZed is the primary supported editor for this stack's AI assistant integration.\n\n**Step 1 — Run the configuration script**:\n\n```bash\nbash scripts/deploy/setup-zed.sh # reads host/port from .env or prompts\nbash scripts/deploy/setup-zed.sh 192.168.1.50:8000 # or pass the host directly\n```\n\nThis configures `~/.config/zed/settings.json`, injecting the vLLM endpoint\nas a custom OpenAI-compatible provider. It resolves the live model ID and\ncontext window from `GET /v1/models` if the server is reachable, falling\nback to `scripts/deploy/.env`'s `SERVED_MODEL_NAME`/`MAX_MODEL_LEN`\notherwise. Any existing `settings.json` is backed up first\n(`settings.json.bak.<timestamp>`), so it's safe to re-run whenever the\nserver's model or context length changes.\n\n**Step 2 — Set the API key placeholder**:\n\nvLLM doesn't enforce an API key, but Zed's OpenAI provider requires one to\nbe present:\n\n1. In Zed, open the Agent panel and click the model selector in the\n bottom-right corner → **Configure...** (or run the `agent: settings`\n command via `Ctrl+Shift+A`).\n2. Under the **OpenAI** provider section, enter `dummy` as the API key.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (custom endpoint via `api_url`) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if reachable (typically `qwen2.5-coder-14b-awq`) — falls back to `.env`'s `SERVED_MODEL_NAME` |\n| API Key | Not enforced by vLLM — use `dummy` in Zed's provider settings |\n\n### Continue (VS Code & JetBrains)\n\nInstall the **Continue** extension/plugin ([VS Code Marketplace](https://marketplace.visualstudio.com/) or [JetBrains Marketplace](https://plugins.jetbrains.com/), for IDEs like PyCharm/IntelliJ/WebStorm/CLion), then run:\n\n```bash\nbash scripts/deploy/setup-continue.sh\n# or: bash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n#### Supported IDEs\n\n* **VS Code** — run the script, then reload the window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n* **JetBrains IDEs** (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.) — run the script (Continue in JetBrains shares the same `~/.continue/config.json` global path on Linux/macOS), then restart the IDE or click the gear icon in the Continue sidebar to refresh.\n\n#### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-14b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n### Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n# or: bash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, specifically patching only the OpenAI-compatible API base URL, API key, and model parameters.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nEven with the smaller 14B model, a desktop environment competing for GPU\n0's VRAM can push KV cache allocation into an OOM at boot — this is why the\nshipped defaults use `GPU_MEMORY_UTILIZATION=0.85` and `MAX_MODEL_LEN=16384`\nrather than the more aggressive `0.90`/`32768` the hardware can otherwise\nsupport. If GPU 0 shows >800 MiB idle usage, free it before deploying:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\nIf GPU 0 is already headless (no idle usage), you can raise\n`GPU_MEMORY_UTILIZATION` back to `0.90` and `MAX_MODEL_LEN` to `32768` in\n`scripts/deploy/.env` for the full context window.\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## License\n\nMIT\n"}, "451f8612d97c75b5872534ef3c7db3494be4d5be": {"t": 1785766229.3774576, "n": 28665, "h": "9e38badb622859da", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect coding tools such as Continue in VS Code\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## IDE Integration: Continue Extension (VS Code & JetBrains)\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has your IDE and\nthe [Continue](https://continue.dev) extension installed** (e.g., VS Code or JetBrains IDEs like PyCharm/IntelliJ). That machine does not need to be the one hosting vLLM.\n\n```bash\n# Run interactively (will prompt to use IP/port from .env if present, or let you enter one manually):\nbash scripts/deploy/setup-continue.sh\n\n# Or pass the host's BIND_HOST:PORT directly as an argument:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n### Supported IDEs\n\n#### 1. VS Code\n* Install the **Continue** extension from the VS Code Marketplace.\n* Run the `setup-continue.sh` script above to configure the endpoint.\n* Once the script finishes, reload the VS Code window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n#### 2. JetBrains IDEs (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.)\n* Install the **Continue** plugin from the JetBrains Marketplace.\n* Run the `setup-continue.sh` script above. (Continue in JetBrains shares the same `~/.continue/config.json` global configuration path on Linux/macOS).\n* Restart your JetBrains IDE or click the gear icon in the Continue sidebar to refresh the configuration.\n\n### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n## Client Integration: Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\n# Run interactively (will prompt to use IP/port from .env if present, or let you enter one manually):\nbash scripts/deploy/setup-aider.sh\n\n# Or pass the host's BIND_HOST:PORT directly as an argument:\nbash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, specifically patching only the OpenAI-compatible API base URL, API key, and model parameters.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n## License\n\nMIT\n"}, "8c655c2632b4d8c9b23a3d818b124349a1cd4ea2": {"t": 1785766230.9228156, "n": 29527, "h": "056b8881cbb5e786", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect coding tools such as Continue in VS Code\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## IDE Integration: Continue Extension (VS Code & JetBrains)\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has your IDE and\nthe [Continue](https://continue.dev) extension installed** (e.g., VS Code or JetBrains IDEs like PyCharm/IntelliJ). That machine does not need to be the one hosting vLLM.\n\n```bash\n# Run interactively (will prompt to use IP/port from .env if present, or let you enter one manually):\nbash scripts/deploy/setup-continue.sh\n\n# Or pass the host's BIND_HOST:PORT directly as an argument:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\n### Supported IDEs\n\n#### 1. VS Code\n* Install the **Continue** extension from the VS Code Marketplace.\n* Run the `setup-continue.sh` script above to configure the endpoint.\n* Once the script finishes, reload the VS Code window (`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n#### 2. JetBrains IDEs (PyCharm, IntelliJ IDEA, WebStorm, CLion, etc.)\n* Install the **Continue** plugin from the JetBrains Marketplace.\n* Run the `setup-continue.sh` script above. (Continue in JetBrains shares the same `~/.continue/config.json` global configuration path on Linux/macOS).\n* Restart your JetBrains IDE or click the gear icon in the Continue sidebar to refresh the configuration.\n\n### How it Works / Custom Configurations\nThe setup script injects the vLLM endpoint into `~/.continue/config.yaml` on the machine you run it on, setting it as both the **chat model** and the **tab-autocomplete model**. It will create the file with full defaults if it doesn't exist, or patch it safely (with a backup) if it does.\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n\n## IDE Integration: VS Code Native Chat\n\nVS Code's native Copilot/Chat interface supports connecting to custom endpoints. You can run the setup script to register your local vLLM server:\n\n```bash\n# Run interactively (will prompt to use IP/port from .env if present, or let you enter one manually):\nbash scripts/deploy/setup-vscode-chat.sh\n\n# Or pass the host's BIND_HOST:PORT directly as an argument:\nbash scripts/deploy/setup-vscode-chat.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Resolves the vLLM server address (prompting, reading from `scripts/deploy/.env`, or using the command-line argument).\n2. Updates or creates `~/.config/Code/User/chatLanguageModels.json` to register the local server as a custom endpoint vendor, avoiding duplicates if already configured.\n3. Reload VS Code to enable selecting the local vLLM model directly in the native chat UI.\n\n## Client Integration: Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\n# Run interactively (will prompt to use IP/port from .env if present, or let you enter one manually):\nbash scripts/deploy/setup-aider.sh\n\n# Or pass the host's BIND_HOST:PORT directly as an argument:\nbash scripts/deploy/setup-aider.sh 192.168.1.50:8000\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Resolves the vLLM server address (interactively prompting for IP and port, reading from `scripts/deploy/.env`, or using the command-line argument).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, specifically patching only the OpenAI-compatible API base URL, API key, and model parameters.\n4. Generates or updates `.aider.model.metadata.json` alongside your Aider config to register the correct context window size (based on the server's `MAX_MODEL_LEN`) and token cost structures, suppressing any \"Unknown context window size and costs\" warnings.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n## License\n\nMIT\n"}, "937518b16e3c61c3b25bc92b14a8faa9a0fd0042": {"t": 1785766232.5393298, "n": 27558, "h": "0a807b3df1727ef6", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect coding tools such as Continue in VS Code\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## Editor Integration: VS Code Continue Extension\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has VS Code and\nthe [Continue](https://continue.dev) extension installed** — that machine\ndoes not need to be the one hosting vLLM. A common setup is a GPU host\nrunning the server on the LAN and one or more separate laptops/workstations\neach pointing their own VS Code at it.\n\n```bash\n# On the vLLM host itself (server reachable at localhost):\nbash scripts/deploy/setup-continue.sh\n\n# From any other workstation on the network — pass the host's BIND_HOST:PORT:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\nThis injects the vLLM endpoint into `~/.continue/config.json` **on the\nmachine you run it on**, as both a chat model and a tab-autocomplete model\n— creating the file with full defaults if it doesn't exist, or patching it\nsafely (with backup) if it does. Reload VS Code afterward\n(`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` (`localhost:8000` by default) |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n## Client Integration: Aider\n\nYou can also use [Aider](https://aider.chat) as a command-line coding assistant powered by the vLLM instance. A setup script is provided to automate Aider installation and configuration.\n\n```bash\nbash scripts/deploy/setup-aider.sh\n```\n\nThis script:\n1. Detects if Aider is installed. If it is not, it stops to confirm if you want to install it (supporting installation via `pipx` or `pip`).\n2. Reads the vLLM endpoint configuration from `scripts/deploy/.env` (using the defined `BIND_HOST` and `PORT`).\n3. Safely updates or creates Aider configuration files (`.aider.conf.yml` at the project root or `~/.aider.conf.yml` in your home directory) to use the local vLLM endpoint, specifically patching only the OpenAI-compatible API base URL, API key, and model parameters.\n\nOnce configured, simply run:\n```bash\naider\n```\n\n## License\n\nMIT\n"}, "e0fadba35c4b3e58a1da3ec29055de49638bdca4": {"t": 1785766235.1858966, "n": 26710, "h": "83176917ca33d08a", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect coding tools such as Continue in VS Code\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── teardown.sh # Graceful server teardown wrapper (stops and deletes containers)\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before teardown.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/teardown.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `teardown.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/teardown.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/teardown.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## Editor Integration: VS Code Continue Extension\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has VS Code and\nthe [Continue](https://continue.dev) extension installed** — that machine\ndoes not need to be the one hosting vLLM. A common setup is a GPU host\nrunning the server on the LAN and one or more separate laptops/workstations\neach pointing their own VS Code at it.\n\n```bash\n# On the vLLM host itself (server reachable at localhost):\nbash scripts/deploy/setup-continue.sh\n\n# From any other workstation on the network — pass the host's BIND_HOST:PORT:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\nThis injects the vLLM endpoint into `~/.continue/config.json` **on the\nmachine you run it on**, as both a chat model and a tab-autocomplete model\n— creating the file with full defaults if it doesn't exist, or patching it\nsafely (with backup) if it does. Reload VS Code afterward\n(`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` (`localhost:8000` by default) |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n## License\n\nMIT\n"}, "f45c2797e370d3fc988bde239098a927d0161756": {"t": 1785766237.3726885, "n": 26659, "h": "6b8c195a07106e74", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect coding tools such as Continue in VS Code\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `scripts/deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, scripts/deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── deploy-artifacts/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.open-webui.yml # Optional WebUI service and volume definition\n│ └── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── .env.example # Annotated parameter reference — copy to .env\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ ├── smoke-test.sh # Smoke test verification for endpoints\n │ └── stop.sh # Graceful server teardown wrapper\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before stop.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `scripts/deploy/.env`\n\n```bash\ncp scripts/deploy/.env.example scripts/deploy/.env\n$EDITOR scripts/deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `scripts/deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `scripts/deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `scripts/deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `scripts/deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`scripts/deploy/.env`. If `scripts/deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `scripts/deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy-artifacts/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit scripts/deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy-artifacts/docker-compose.yml down\n# or: bash scripts/deploy/stop.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy-artifacts/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy-artifacts/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `stop.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `scripts/deploy/.env` file. Copy these values from `scripts/deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `scripts/deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/stop.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/stop.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `scripts/deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `scripts/deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `scripts/deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `scripts/deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `scripts/deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `scripts/deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy-artifacts/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## Editor Integration: VS Code Continue Extension\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has VS Code and\nthe [Continue](https://continue.dev) extension installed** — that machine\ndoes not need to be the one hosting vLLM. A common setup is a GPU host\nrunning the server on the LAN and one or more separate laptops/workstations\neach pointing their own VS Code at it.\n\n```bash\n# On the vLLM host itself (server reachable at localhost):\nbash scripts/deploy/setup-continue.sh\n\n# From any other workstation on the network — pass the host's BIND_HOST:PORT:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\nThis injects the vLLM endpoint into `~/.continue/config.json` **on the\nmachine you run it on**, as both a chat model and a tab-autocomplete model\n— creating the file with full defaults if it doesn't exist, or patching it\nsafely (with backup) if it does. Reload VS Code afterward\n(`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` (`localhost:8000` by default) |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `scripts/deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `scripts/deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n## License\n\nMIT\n"}, "763b0a41f01727075c69f110edaf8ebafa12e41e": {"t": 1785766238.9433715, "n": 26228, "h": "6a184ba8627002d8", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Deployment Scope\n\nThis repository deploys a single-node vLLM OpenAI-compatible API server for use on a private homelab network.\n\nThe intended use case is:\n\n- Run vLLM on one GPU-equipped Ubuntu host\n- Expose the vLLM API on the private LAN\n- Connect coding tools such as Continue in VS Code\n- Optionally connect a local web UI for direct chat interaction\n\nThis repo does not currently target:\n\n- Multi-node vLLM\n- Ray-based distributed inference\n- Kubernetes\n- Public internet exposure\n- Production authentication or TLS\n\n## Table of Contents\n\n- [Deployment Scope](#deployment-scope)\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Open WebUI Support (Optional)](#open-webui-support-optional)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n├── deploy/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n│ └── .env.example # Annotated parameter reference — copy to .env\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ └── stop.sh # Graceful vLLM server teardown wrapper\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before stop.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `deploy/.env`\n\n```bash\ncp deploy/.env.example deploy/.env\n$EDITOR deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`deploy/.env`. If `deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy/docker-compose.yml down\n# or: bash scripts/deploy/stop.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Open WebUI Support (Optional)\n\nThis repository includes optional support for [Open WebUI](https://github.com/open-webui/open-webui), allowing you to interact with the hosted vLLM model through a beautiful ChatGPT-like browser interface.\n\n### Key Features\n- **Browser Access**: Use the model from any device (phone, laptop, tablet) on your local network.\n- **Optional & Disabled by Default**: Kept disabled by default (`ENABLE_OPEN_WEBUI=false`) to preserve the core vLLM-only focus.\n- **Data Persistence**: Open WebUI's database, user accounts, and chat history are saved in a persistent Docker volume, preserving your data across container restarts, redeployments, and normal `stop.sh` operations.\n- **Auto-Boot**: Starts automatically at host reboot alongside vLLM when enabled.\n\n### Configuration\n\nAll Open WebUI settings live in your `deploy/.env` file. Copy these values from `deploy/.env.example` if they are not already in your configuration:\n\n```env\n# Enable Open WebUI deployment (true/false)\nENABLE_OPEN_WEBUI=true\n\n# Port on the host network where Open WebUI will listen\nOPEN_WEBUI_PORT=3000\n\n# Subnet CIDR of your private LAN to restrict firewall access (optional)\n# Example: LAN_CIDR=10.1.10.0/24\nLAN_CIDR=10.1.10.0/24\n```\n\n### Deployment\n\nSimply set `ENABLE_OPEN_WEBUI=true` in `deploy/.env` and run the deployment script:\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\nThe script will automatically detect that Open WebUI is enabled, perform port availability checks, open UFW/firewalld rules (restricted to `LAN_CIDR` if set), launch the container stack, and validate that both vLLM and Open WebUI are running and configured with their restart policies.\n\nAfter successful deployment, the script outputs the connection URLs:\n```text\nvLLM API : http://10.1.10.17:8000/v1\nOpen WebUI : http://10.1.10.17:3000\n```\n\n> [!NOTE]\n> On the first run of Open WebUI, you will need to sign up to create the admin account. This account is entirely local and does not send any data outside your network. Since this setup is intended for a trusted home LAN, there is no TLS or external authentication configured by default.\n\n### Verification & Smoke Testing\n\nTo verify both services are running and accessible from either the host itself or another LAN client:\n\n```bash\n# Run the validation smoke test\nbash scripts/deploy/smoke-test.sh\n```\n\nYou can also pass overrides to verify connectivity from another machine on your LAN:\n```bash\n# Usage: bash scripts/deploy/smoke-test.sh [host-ip] [vllm-port] [open-webui-port] [enable-webui]\nbash scripts/deploy/smoke-test.sh 10.1.10.17 8000 3000 true\n```\n\nThe smoke test validates:\n1. vLLM `/v1/models` endpoint responds successfully.\n2. Open WebUI HTTP endpoint responds on the configured port.\n3. Both containers are configured with `unless-stopped` (or your configured) restart policies.\n\n### Teardown and Data Preservation\n\nTo stop the services and release GPU VRAM:\n```bash\nbash scripts/deploy/stop.sh\n```\nThis stops both vLLM and Open WebUI containers. **Your Open WebUI chat history, user accounts, and settings are preserved.**\n\nTo perform a deep-clean and delete all Open WebUI data/volumes, pass the `--purge` flag:\n```bash\n# WARNING: This deletes the Open WebUI database/volume permanently!\nbash scripts/deploy/stop.sh --purge\n```\n\n### Troubleshooting\n\n- **Container fails to start or port already in use**: The deployment script checks port availability and fails-fast. If the port is in use, verify with `ss -tlnp` (as root) or configure a different `OPEN_WEBUI_PORT` in `deploy/.env`.\n- **Cannot reach Open WebUI from another LAN host**: Ensure `OPEN_WEBUI_HOST` is set to `0.0.0.0` (all interfaces) in `deploy/.env`. Verify that the firewall (UFW/firewalld) is allowing the port and that `LAN_CIDR` matches your client's subnet.\n- **Open WebUI cannot reach vLLM**: Open WebUI connects to vLLM inside the Docker network. Ensure `OPEN_WEBUI_OPENAI_API_BASE_URL` in `deploy/.env` points to `http://vllm:8000/v1` (using the container service name `vllm` rather than `localhost`).\n- **Services not starting after reboot**: Check if the Docker service is enabled to start at boot (`systemctl is-enabled docker`). Verify the restart policies in `deploy/.env` (`VLLM_RESTART_POLICY` and `OPEN_WEBUI_RESTART_POLICY`) are set to `unless-stopped` or `always`.\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## Editor Integration: VS Code Continue Extension\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has VS Code and\nthe [Continue](https://continue.dev) extension installed** — that machine\ndoes not need to be the one hosting vLLM. A common setup is a GPU host\nrunning the server on the LAN and one or more separate laptops/workstations\neach pointing their own VS Code at it.\n\n```bash\n# On the vLLM host itself (server reachable at localhost):\nbash scripts/deploy/setup-continue.sh\n\n# From any other workstation on the network — pass the host's BIND_HOST:PORT:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\nThis injects the vLLM endpoint into `~/.continue/config.json` **on the\nmachine you run it on**, as both a chat model and a tab-autocomplete model\n— creating the file with full defaults if it doesn't exist, or patching it\nsafely (with backup) if it does. Reload VS Code afterward\n(`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` (`localhost:8000` by default) |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n## License\n\nMIT\n"}, "1e5d8085ee5d110e9f31a10742485f1d871fb30e": {"t": 1785766241.5123596, "n": 21281, "h": "f32af3f6aff5f152", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Table of Contents\n\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── .secrets.baseline # detect-secrets baseline — known false positives\n├── README.md\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n├── deploy/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n│ └── .env.example # Annotated parameter reference — copy to .env\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ └── stop.sh # Graceful vLLM server teardown wrapper\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before stop.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `deploy/.env`\n\n```bash\ncp deploy/.env.example deploy/.env\n$EDITOR deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | If `BIND_HOST` is a non-loopback address (e.g. `10.1.10.17`, `192.168.0.x`), check for an active firewall (`ufw`/`firewalld`) and open the API port if it isn't already allowed; if no firewall is active, or `BIND_HOST` is `127.0.0.1`, skip — nothing to do | — |\n| 4 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 5 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 6 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 7 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `deploy/.env` in place, diffing against any existing values first | ✅ |\n| 8 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 9 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 10 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 11 | `docker compose up -d` | — |\n| 12 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`deploy/.env`. If `deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy/docker-compose.yml down\n# or: bash scripts/deploy/stop.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- If `BIND_HOST` is set to a LAN address, `deploy.sh` opens the API port on\n `ufw`/`firewalld` automatically (only if one of them is active — it never\n installs or enables a firewall for you). It only ever opens the single\n `PORT` from `deploy/.env`, never a broad range.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `detect-secrets` — scans staged changes for hardcoded secrets, API keys, or credentials using [detect-secrets](https://github.com/Yelp/detect-secrets) (no account/registration required, unlike some hosted secret-scanning services). Known false positives (e.g. the literal placeholder `api_key=\"dummy\"` used since vLLM doesn't enforce API keys by default) are tracked in `.secrets.baseline` — if you intentionally add a new one, regenerate it with `detect-secrets scan > .secrets.baseline` and mark it as a false positive.\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## Editor Integration: VS Code Continue Extension\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has VS Code and\nthe [Continue](https://continue.dev) extension installed** — that machine\ndoes not need to be the one hosting vLLM. A common setup is a GPU host\nrunning the server on the LAN and one or more separate laptops/workstations\neach pointing their own VS Code at it.\n\n```bash\n# On the vLLM host itself (server reachable at localhost):\nbash scripts/deploy/setup-continue.sh\n\n# From any other workstation on the network — pass the host's BIND_HOST:PORT:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\nThis injects the vLLM endpoint into `~/.continue/config.json` **on the\nmachine you run it on**, as both a chat model and a tab-autocomplete model\n— creating the file with full defaults if it doesn't exist, or patching it\nsafely (with backup) if it does. Reload VS Code afterward\n(`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` (`localhost:8000` by default) |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n## License\n\nMIT\n"}, "941db4532ca44ba46d804e6ec36196f0ebeb50a4": {"t": 1785766243.006571, "n": 20263, "h": "b89efb0d07892340", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n## Table of Contents\n\n- [Repository Structure](#repository-structure)\n- [Deploying](#deploying)\n - [1. Configure `deploy/.env`](#1-configure-deployenv)\n - [2. Run the deploy script](#2-run-the-deploy-script)\n - [3. Verify performance](#3-verify-performance)\n- [Manual Step-by-Step Setup (Advanced)](#manual-step-by-step-setup-advanced)\n- [API Usage](#api-usage)\n- [Tuning Reference](#tuning-reference)\n- [Server Management](#server-management)\n- [Security Notes](#security-notes)\n- [Development & Code Quality](#development--code-quality)\n- [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension)\n- [License](#license)\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Ignores WORKLOG.md, deploy/.env, generated override files, benchmark results\n├── .pre-commit-config.yaml # Pre-commit framework configuration\n├── README.md\n├── git-hooks/\n│ └── check-commit-msg-secrets.py # commit-msg hook: scans commit messages for secrets\n├── deploy/\n│ ├── docker-compose.yml # vLLM service definition (image, GPU reservation, healthcheck)\n│ ├── docker-compose.override.yml # Auto-generated by deploy.sh — gitignored, do not edit\n│ └── .env.example # Annotated parameter reference — copy to .env\n└── scripts/\n ├── prereqs/\n │ └── install-prereqs.sh # Idempotent dependency installer (drivers, Docker, toolkit)\n ├── deploy/\n │ ├── deploy.sh # Single entry point — orchestrates the full deployment\n │ ├── validate-system.sh # Pre-flight hardware validation\n │ ├── validate-vram.sh # Live startup telemetry monitor\n │ ├── setup-continue.sh # VS Code Continue extension hook\n │ └── stop.sh # Graceful vLLM server teardown wrapper\n └── tuning/\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before stop.sh when debugging)\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n## Deploying\n\n### 1. Configure `deploy/.env`\n\n```bash\ncp deploy/.env.example deploy/.env\n$EDITOR deploy/.env # set BIND_HOST, HF_CACHE_DIR at minimum\n```\n\nSee `deploy/.env.example` for every available option and its explanation.\n\n### 2. Run the deploy script\n\n```bash\nsudo bash scripts/deploy/deploy.sh\n```\n\n`deploy.sh` is the **single entry point** for standing up the stack. It is\nnot something you run *after* validating and tuning the system — it runs\nthose steps *for you*, in this order. Root is required end-to-end (package\ninstallation and enabling the Docker service both need it):\n\n| Step | What happens | Blocking? |\n|------|--------------|-----------|\n| 1 | Load and validate `deploy/.env` | ✅ Aborts if `.env` is missing |\n| 2 | Resolve `BIND_HOST` (auto-detects your LAN IP if unset) | — |\n| 3 | Run `install-prereqs.sh` — drivers, Docker, nvidia-container-toolkit (idempotent; prompts before installing anything missing, never touches what's already there) | ✅ |\n| 4 | Run `validate-system.sh` — GPU ↔ Docker connectivity, PCIe link quality | ✅ |\n| 5 | Run `check-bottlenecks.sh` — performance advisory | ⚠️ Never blocks |\n| 6 | Run `tune-inference.sh` — updates only the GPU-tuned keys in `deploy/.env` in place, diffing against any existing values first | ✅ |\n| 7 | Re-apply your user-set values on top (your settings always win over auto-tuning) | — |\n| 8 | Generate `docker-compose.override.yml` with the fully-resolved vLLM command | — |\n| 9 | Ensure `docker.service` is enabled at boot, so the container (`restart: unless-stopped`) comes back up after a host reboot, not just a plain restart | — |\n| 10 | `docker compose up -d` | — |\n| 11 | Monitor startup — VRAM telemetry + log tailing until the server reports ready, or an OOM is detected | — |\n\nA single successful run leaves you with a running, health-checked server at\n`http://<BIND_HOST>:<PORT>/v1`. If you want to run any of these steps\nindividually — for debugging, or because you want finer control — see\n[Manual Step-by-Step Setup](#manual-step-by-step-setup-advanced) below.\n\n> ⚠ If NVIDIA drivers were just installed by `install-prereqs.sh` for the\n> first time, **reboot**, then re-run `deploy.sh`.\n\n### 3. Verify performance\n\n**Benchmark** — single-stream throughput, using a multi-step Rust/Tokio\nprogramming prompt:\n\n```bash\nbash scripts/tuning/benchmark.sh\n```\n\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\n**Load test** (optional) — `benchmark.sh` measures one request at a time;\nreal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/tuning/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/tuning/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of\nthe run and reports any that go active — distinct from\n`check-bottlenecks.sh`'s point-in-time power-cap check, this catches\n*actual* throttling as it happens under sustained concurrent load (e.g. a\ncard that's fine at idle but thermal-throttles a minute into real traffic).\n\n**Tracking changes over time** — every `benchmark.sh` and `load-test.sh`\nrun is saved as a timestamped JSON record in `benchmark-results/`. After\nchanging `deploy/.env` (via `tune-inference.sh`) or host tuning (via\n`check-bottlenecks.sh`'s recommendations), re-run the same script and diff\nagainst the previous result:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/tuning/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/tuning/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/tuning/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/tuning/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n## Manual Step-by-Step Setup (Advanced)\n\n`deploy.sh` covers the steps below automatically. Run them individually\nonly if you want to re-run a single step in isolation (e.g. re-validate\nafter a hardware change) or need to debug a specific stage.\n\n<details>\n<summary>Expand manual steps</summary>\n\n**Step 1 — Install prerequisites**\n\n```bash\nsudo bash scripts/prereqs/install-prereqs.sh\n```\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit,\nand system tools. Fully idempotent — safe to re-run.\n\nAnything already installed (driver, Docker, toolkit, NVIDIA runtime\nregistration in `daemon.json`) is left untouched — never updated, upgraded,\nor reconfigured. For anything missing, it prompts `[y/N]` before installing\neach component. Pass `-y`/`--yes` to auto-confirm every prompt for\nunattended/automated runs (e.g. `deploy.sh` still runs it in the\nforeground, so prompts surface normally unless you pass `-y`).\n\n**Step 2 — Validate system hardware**\n\n```bash\nbash scripts/deploy/validate-system.sh\n```\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and\ndisplay server VRAM impact on GPU 0.\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\nIf a display server is detected on GPU 0, switch to headless mode before\ndeploying (saves ~600–1500 MiB on GPU 0):\n\n```bash\nsudo systemctl isolate multi-user.target\n```\n\n**Step 3 — Generate tuned configuration**\n\n```bash\nbash scripts/tuning/tune-inference.sh\n```\n\nQueries your GPU topology dynamically and writes a hardware-appropriate\n`deploy/.env`. If `deploy/.env` doesn't exist yet, it's created from\nscratch. If it already exists, only the five hardware-tuned keys below are\nupdated in place — any change is printed as an `old -> new` diff before\nbeing applied. Everything else in the file (`MODEL`, `BIND_HOST`, `PORT`,\n`HF_CACHE_DIR`, `HF_TOKEN`, optional feature flags, comments) is left\nexactly as it is on disk.\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `deploy/.env` before continuing — you may manually adjust any\nvalue.\n\n**Step 4 — Start the server and monitor initialization**\n\n```bash\nbash scripts/deploy/validate-vram.sh\n```\n\nLaunches the container (`docker compose up -d` against the plain\n`docker-compose.yml`, no override file) and monitors VRAM allocation in\nreal time during the KV cache loading phase, every 5s for up to 150s.\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\nOr bypass the monitor entirely:\n\n```bash\ndocker compose -f deploy/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\nFrom here, continue with [Verify performance](#3-verify-performance) above, or [Editor Integration: VS Code Continue Extension](#editor-integration-vs-code-continue-extension) at the end of this document.\n\n> ⚠ Going this manual route skips the boot-persistence check `deploy.sh`\n> does automatically. `docker-compose.yml` sets `restart: unless-stopped`,\n> so the container itself restarts once the Docker daemon is up — but only\n> if `docker.service` is enabled to start at boot:\n> ```bash\n> sudo systemctl enable docker\n> ```\n\n</details>\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for\nTensor Parallelism goes over PCIe. A secondary slot running at x4 instead\nof x16 will reduce NCCL all-reduce bandwidth and may increase latency by\n10–25% on large token batches.\n\nTo diagnose: run `bash scripts/deploy/validate-system.sh` and review the\nPCIe table (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/tuning/check-bottlenecks.sh` for the more directly relevant\ncheck — it computes actual effective GB/s per link and flags anything\nbelow the Gen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce\nneeds, which a card can fail even while running at its own full rated spec\n(e.g. a Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on\n24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is\ncompeting with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy/docker-compose.yml down\n# or: bash scripts/deploy/stop.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/tuning/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy/docker-compose.yml logs -f\n\n# Restart after a config change\ndocker compose -f deploy/docker-compose.yml down\nsudo bash scripts/deploy/deploy.sh # Re-tunes and regenerates the override file\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n## Security Notes\n\n- The server binds to `0.0.0.0` on all interfaces by default. If running on\n a network-accessible machine, set `BIND_HOST=127.0.0.1` in `deploy/.env`\n to restrict it to localhost, or add a firewall rule.\n- vLLM does not enforce API key authentication by default. Add\n `--api-key <secret>` to the command in `docker-compose.yml` (or via\n `docker-compose.override.yml`) to enable it.\n- The HuggingFace cache is mounted from the host via `HF_CACHE_DIR`. Ensure\n the model cache directory has appropriate permissions.\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate\ncode validation and enforce security best practices before any changes are\ncommitted.\n\n### Installed Hooks\n\n**Syntax linting & formatting**\n\n- `trailing-whitespace` — trims trailing whitespace from files\n- `end-of-file-fixer` — ensures files end with a newline\n- `check-yaml` — validates YAML syntax (e.g. `deploy/docker-compose.yml`, `.pre-commit-config.yaml`)\n- `check-json` — validates JSON syntax\n- `check-added-large-files` — blocks accidentally committing large files (e.g. model weights, cached tensors)\n- `shellcheck` — runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/`\n\n**Security & secret detection**\n\n- `detect-private-key` — checks for the presence of private keys\n- `gitleaks` — scans staged changes for hardcoded secrets, API keys, or credentials using [Gitleaks](https://github.com/gitleaks/gitleaks)\n- **Custom commit message scanner** — `git-hooks/check-commit-msg-secrets.py` scans Git commit messages for secrets (e.g. AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` hook phase\n\n### Manual Verification\n\nRun every pre-commit check against all files at any time:\n\n```bash\npre-commit run --all-files\n```\n\n## Editor Integration: VS Code Continue Extension\n\nThis is a client-side setup step, separate from deploying, tuning, or\nbenchmarking the server. Run it on **whichever workstation has VS Code and\nthe [Continue](https://continue.dev) extension installed** — that machine\ndoes not need to be the one hosting vLLM. A common setup is a GPU host\nrunning the server on the LAN and one or more separate laptops/workstations\neach pointing their own VS Code at it.\n\n```bash\n# On the vLLM host itself (server reachable at localhost):\nbash scripts/deploy/setup-continue.sh\n\n# From any other workstation on the network — pass the host's BIND_HOST:PORT:\nbash scripts/deploy/setup-continue.sh 192.168.1.50:8000\n```\n\nThis injects the vLLM endpoint into `~/.continue/config.json` **on the\nmachine you run it on**, as both a chat model and a tab-autocomplete model\n— creating the file with full defaults if it doesn't exist, or patching it\nsafely (with backup) if it does. Reload VS Code afterward\n(`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://<host>:<port>/v1` (`localhost:8000` by default) |\n| Model | Resolved live from `GET /v1/models` if the server is reachable (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name`) — falls back to `MODEL=` from `deploy/.env` with a warning if it isn't (only relevant when run on the vLLM host itself, since that's the only place `deploy/.env` exists) |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nIf the server isn't reachable yet when you run this from a remote\nworkstation, it falls back to the default HuggingFace model ID — re-run it\nonce the server is up for an accurate config.\n\n## License\n\nMIT\n"}, "f2274fe422c7e974ad9773d811bd8d53b07eab7e": {"t": 1785766244.5453978, "n": 15277, "h": "932ef5c30a5c8d70", "c": "# vllm-containerized-deploy\n\n[](https://opensource.org/licenses/MIT)\n[](https://github.com/pre-commit/pre-commit)\n[](https://docs.docker.com/compose/)\n[](https://www.python.org/)\n[](https://github.com/koalaman/shellcheck)\n[](https://developer.nvidia.com/cuda-zone)\n[](https://github.com/vllm-project/vllm)\n\n> Production-grade automation for self-hosting **Qwen2.5-Coder-32B-Instruct-AWQ** on a dual-GPU RTX 3060 setup using vLLM with Tensor Parallelism.\n\n---\n\n## Hardware Target\n\n| Component | Specification |\n|-----------|--------------|\n| GPUs | 2× NVIDIA RTX 3060 12GB |\n| Total VRAM | 24 GB |\n| Parallelism | Tensor Parallel (size=2) |\n| Host OS | Ubuntu 22.04 LTS |\n| Model | `Qwen/Qwen2.5-Coder-32B-Instruct-AWQ` |\n| Quantization | AWQ (4-bit activation-aware) |\n| Context | 16,384 tokens (capped for KV cache budget) |\n\n---\n\n## Repository Structure\n\n```\nvllm-containerized-deploy/\n├── .gitignore # Git ignore patterns (ignores WORKLOG.md, deploy/.env, test results, environments)\n├── .pre-commit-config.yaml # Pre-commit framework configurations\n├── README.md # Comprehensive setup and operations documentation\n├── deploy/\n│ ├── docker-compose.yml # vLLM service definition with healthcheck\n│ └── .env.example # Annotated parameter reference\n└── scripts/\n ├── install-prereqs.sh # Idempotent dependency installer\n ├── validate-system.sh # Pre-flight hardware validation\n ├── tune-inference.sh # Hardware-sensing config generator\n ├── validate-vram.sh # Live startup telemetry monitor\n ├── setup-continue.sh # VS Code Continue extension hook\n ├── stop.sh # Graceful vLLM server teardown wrapper\n ├── snapshot-diagnostics.sh # Read-only GPU state + log capture (run before stop.sh when debugging)\n ├── check-bottlenecks.sh # Hardware & OS performance advisor (--json for machine-readable output)\n ├── check-commit-msg-secrets.py # Custom hook for commit-msg secrets validation\n ├── benchmark.sh # Single-stream token throughput evaluator\n ├── load-test.sh # Concurrent-request load tester (req/s, latency percentiles)\n └── compare-benchmarks.sh # Regression detector across benchmark-results/ history\n```\n\n`benchmark.sh` and `load-test.sh` each write a timestamped JSON record to\n`benchmark-results/` (created on first run) so tuning changes can be\ncompared over time with `compare-benchmarks.sh`.\n\n---\n\n## Quick Start\n\nRun steps **in order**. Each script validates prerequisites before proceeding.\n\n### Step 1 — Install Prerequisites\n\nInstalls NVIDIA drivers (if absent), Docker CE, nvidia-container-toolkit, and system tools. Fully idempotent — safe to re-run.\n\n```bash\nsudo bash scripts/install-prereqs.sh\n```\n\n> ⚠ If drivers were just installed, **reboot first**, then re-run the script.\n\n---\n\n### Step 2 — Validate System Hardware\n\nChecks Docker ↔ GPU connectivity, PCIe link quality (Gen/Width), and display server VRAM impact on GPU 0.\n\n```bash\nbash scripts/validate-system.sh\n```\n\n**What it checks:**\n\n| Check | Pass Condition |\n|-------|---------------|\n| Docker GPU access | Container sees all GPUs via `nvidia` runtime |\n| PCIe Gen | `current` == `max` for each GPU |\n| PCIe Width | `current` == `max` for each GPU |\n| Idle VRAM (GPU 0) | < 800 MiB (no display server consuming budget) |\n\n**If a display server is detected on GPU 0:**\n```bash\n# Switch to headless mode before deploying (saves ~600–1500 MiB on GPU 0)\nsudo systemctl isolate multi-user.target\n```\n\n---\n\n### Step 3 — Generate Tuned Configuration\n\nQueries your GPU topology dynamically and writes a hardware-appropriate `deploy/.env`.\n\n```bash\nbash scripts/tune-inference.sh\n```\n\n**Parameters auto-calculated:**\n\n| Parameter | Value (24 GiB setup) | Rationale |\n|-----------|---------------------|-----------|\n| `TENSOR_PARALLEL_SIZE` | 2 | One shard per GPU |\n| `GPU_MEMORY_UTILIZATION` | 0.90 | ~1.2 GiB headroom per card |\n| `MAX_MODEL_LEN` | 16384 | KV cache stays within VRAM budget |\n| `SWAP_SPACE` | 4 GiB | CPU offload buffer for burst traffic |\n\nReview `deploy/.env` before continuing. You may manually adjust any value.\n\n---\n\n### Step 4 — Start Server & Monitor Initialization\n\nLaunches the container and monitors VRAM allocation in real-time during the KV cache loading phase.\n\n```bash\nbash scripts/validate-vram.sh\n```\n\n**What it monitors (every 5s for up to 150s):**\n- Per-GPU VRAM usage with ASCII bar visualization\n- Container logs parsed for startup signals\n\n| Signal | Action |\n|--------|--------|\n| `Uvicorn running on...` | ✅ Exit 0 — server ready |\n| `CUDA out of memory` | ❌ Exit 1 — prints recovery instructions |\n| Timeout (150s) | ⚠ Model still downloading — check `docker logs vllm-coder-server --follow` |\n\n**Manual start (bypassing the monitor):**\n```bash\ndocker compose -f deploy/docker-compose.yml up -d\ndocker logs vllm-coder-server --follow\n```\n\n---\n\n### Step 5 — Configure VS Code Continue Extension\n\nInjects the local vLLM endpoint into `~/.continue/config.json` as both a chat model and tab-autocomplete model. Creates the file with full defaults if it doesn't exist; patches safely if it does (with backup).\n\n```bash\nbash scripts/setup-continue.sh\n```\n\nAfter running, reload VS Code (`Ctrl+Shift+P` → **Developer: Reload Window**).\n\n**Configured endpoints:**\n\n| Setting | Value |\n|---------|-------|\n| Provider | `openai` (OpenAI-compatible) |\n| API Base | `http://localhost:8000/v1` |\n| Model | Resolved live from `GET /v1/models` if the server is up (typically `qwen2.5-coder-32b-awq`, matching `--served-model-name` in `docker-compose.yml`) — falls back to `MODEL=` from `deploy/.env` with a warning if the server isn't reachable yet |\n| Autocomplete | Same resolved model, `max_tokens=512`, `temperature=0.05` |\n\nFollowing Step 4 before Step 5 (as ordered above) means the server is\nalready up, so the live-resolved value is what actually gets written —\nnot the raw HuggingFace model ID.\n\n---\n\n### Step 6 — Benchmark\n\nValidates throughput by sending a multi-step Rust/Tokio programming prompt and reporting tokens/second.\n\n```bash\nbash scripts/benchmark.sh\n```\n\n**Sample output:**\n```\n Run Elapsed (s) Prompt tok Comp tok Tok/s Finish\n ────────────────────────────────────────────────────────────────\n 1 42.31 387 1024 24.20 length\n 2 41.87 387 1024 24.46 length\n 3 43.02 387 1024 23.80 length\n\n ════════════════════════════════════════════════════════════════\n SUMMARY (n=3 runs)\n ════════════════════════════════════════════════════════════════\n Avg prompt tokens 387\n Avg completion tokens 1024\n Avg generation time 42.40 s\n Avg throughput 24.15 tok/s\n Peak throughput 24.46 tok/s\n Min throughput 23.80 tok/s\n ════════════════════════════════════════════════════════════════\n Performance tier: ✓ Good (15–30 tok/s)\n\n Results saved to: benchmark-results/benchmark_20260702T044144Z.json\n```\n\nEach run is also saved to `benchmark-results/benchmark_<timestamp>.json` for\nlater comparison (see Step 7).\n\n---\n\n### Step 7 — Load Test & Track Tuning Changes Over Time (optional)\n\n`benchmark.sh` measures single-stream throughput — one request at a time.\nReal usage (multiple editor sessions, multiple users) is concurrent.\n`load-test.sh` fires many small requests from several simultaneous workers\nand reports aggregate requests/sec, aggregate tokens/sec, and latency\npercentiles (p50/p95/p99):\n\n```bash\nbash scripts/load-test.sh [concurrency] [duration_seconds] # defaults: 4, 30\nbash scripts/load-test.sh 8 60 # 8 concurrent clients, 60s\n```\n\nIt also samples `nvidia-smi`'s per-GPU throttle-reason flags (power cap,\nHW/SW thermal slowdown, power brake) once per second for the duration of the\nrun and reports any that go active — distinct from `check-bottlenecks.sh`'s\npoint-in-time power-cap check, this catches *actual* throttling as it\nhappens under sustained concurrent load (e.g. a card that's fine at idle but\nthermal-throttles a minute into real traffic).\n\nEvery `benchmark.sh` and `load-test.sh` run is saved as a timestamped JSON\nrecord in `benchmark-results/`. After changing `deploy/.env` (via\n`tune-inference.sh`) or host tuning (via `check-bottlenecks.sh`'s\nrecommendations), re-run the same script and diff against the previous\nresult:\n\n```bash\nbash scripts/compare-benchmarks.sh # latest 2 benchmark.sh runs\nbash scripts/compare-benchmarks.sh --load-test # latest 2 load-test.sh runs\n```\n\nIt prints a per-metric delta table and exits non-zero if any metric\nregressed beyond its threshold (throughput: 10%, latency: 15–20% depending\non percentile) — safe to drop into a personal tuning script or CI job for\nGPU tuning iterations.\n\n`benchmark-results/` grows one JSON file per run and is never pruned\nautomatically. Once it has real history, trim it with:\n\n```bash\nbash scripts/compare-benchmarks.sh --prune # dry run — lists what would be deleted (keeps last 20 of each type)\nbash scripts/compare-benchmarks.sh --prune --keep 10 # dry run with a custom retention count\nbash scripts/compare-benchmarks.sh --prune --force # actually delete\n```\n\nDry run is the default since deleting benchmark history is irreversible —\nnothing is removed until you pass `--force`.\n\n---\n\n## API Usage\n\nOnce the server is running, it exposes a fully OpenAI-compatible API:\n\n```bash\n# Chat Completion\ncurl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-coder-32b-awq\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Write a Python async HTTP client\"}],\n \"max_tokens\": 512,\n \"temperature\": 0.1\n }'\n\n# List loaded models\ncurl http://localhost:8000/v1/models\n\n# Health check\ncurl http://localhost:8000/health\n```\n\n**OpenAI Python SDK:**\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"http://localhost:8000/v1\",\n api_key=\"dummy\", # vLLM does not enforce API keys by default\n)\n\nresponse = client.chat.completions.create(\n model=\"qwen2.5-coder-32b-awq\",\n messages=[{\"role\": \"user\", \"content\": \"Implement a binary search tree in Go\"}],\n max_tokens=1024,\n temperature=0.1,\n)\nprint(response.choices[0].message.content)\n```\n\n---\n\n## Tuning Reference\n\n### OOM Recovery\n\nIf vLLM exits with a CUDA OOM error during initialization:\n\n```bash\n# Edit deploy/.env\nMAX_MODEL_LEN=8192 # Halve the context window\nGPU_MEMORY_UTILIZATION=0.85 # Increase headroom\n\n# Restart\ndocker compose -f deploy/docker-compose.yml down\nbash scripts/validate-vram.sh\n```\n\n### PCIe Bandwidth Notes\n\nThe RTX 3060 does not support NVLink. All inter-GPU communication for Tensor Parallelism goes over PCIe. A secondary slot running at x4 instead of x16 will reduce NCCL all-reduce bandwidth and may increase latency by 10–25% on large token batches.\n\nTo diagnose: run `bash scripts/validate-system.sh` and review the PCIe\ntable (flags a GPU running below its *own* rated Gen/Width spec), or\n`bash scripts/check-bottlenecks.sh` for the more directly relevant check —\nit computes actual effective GB/s per link and flags anything below the\nGen3×8 / Gen4×4 floor that Tensor Parallelism's NCCL all-reduce needs,\nwhich a card can fail even while running at its own full rated spec (e.g.\na Gen2×16 slot).\n\n### Display Server Impact\n\nRunning a desktop environment while serving a 32B model is a tight fit on 24 GiB. If GPU 0 shows >800 MiB idle usage, the display server is competing with the KV cache:\n\n```bash\n# Free GPU 0 before deployment (non-destructive, re-enable with graphical.target)\nsudo systemctl isolate multi-user.target\n\n# Re-enable desktop when done\nsudo systemctl isolate graphical.target\n```\n\n---\n\n## Server Management\n\n```bash\n# Stop the server\ndocker compose -f deploy/docker-compose.yml down\n# or: bash scripts/stop.sh (same thing, plus a post-stop VRAM confirmation table)\n\n# Capture a diagnostic snapshot before stopping (GPU state + recent logs) —\n# useful when debugging a crash, OOM, or silent hang. Read-only, never\n# modifies GPU/container state.\nbash scripts/snapshot-diagnostics.sh [log_lines] # default: last 50 log lines\n# Saved to ~/.local/share/vllm-snapshots/snapshot_<timestamp>.txt\n\n# View live logs\ndocker compose -f deploy/docker-compose.yml logs -f\n\n# Restart after config change\ndocker compose -f deploy/docker-compose.yml down\nbash scripts/tune-inference.sh # Regenerate .env if needed\ndocker compose -f deploy/docker-compose.yml up -d\n\n# Check container health\ndocker inspect --format='{{.State.Health.Status}}' vllm-coder-server\n```\n\n---\n\n## Security Notes\n\n- The server binds to `0.0.0.0:8000` by default. If running on a network-accessible machine, use a firewall rule or change `HOST=127.0.0.1` in `deploy/.env`.\n- vLLM does not enforce API key authentication by default. Add `--api-key <secret>` to the command in `docker-compose.yml` to enable it.\n- The HuggingFace cache is mounted from the host. Ensure the model cache directory has appropriate permissions.\n\n---\n\n## Development & Code Quality\n\nThis repository uses [pre-commit](https://pre-commit.com/) to automate code validation and enforce security best practices before any changes are committed.\n\n### Installed Hooks\n\n1. **Syntax Linting & Formatting**:\n - `trailing-whitespace`: Trims trailing whitespace from files.\n - `end-of-file-fixer`: Ensures files end with a newline.\n - `check-yaml`: Validates YAML syntax (e.g. for `deploy/docker-compose.yml` and `.pre-commit-config.yaml`).\n - `check-json`: Validates JSON syntax.\n - `check-added-large-files`: Blocks accidentally committing large files (e.g. model weights, cached tensors) that shouldn't be tracked in git.\n - `shellcheck`: Runs [ShellCheck](https://github.com/koalaman/shellcheck) on all shell scripts in `scripts/` to catch syntax, portability, and scripting errors.\n2. **Security & Secret Detection**:\n - `detect-private-key`: Checks for the presence of private keys.\n - `gitleaks`: Scans staged changes for hardcoded secrets, API keys, or credentials using [Gitleaks](https://github.com/gitleaks/gitleaks).\n - **Custom Commit Message Scanner**: A custom local python script (`scripts/check-commit-msg-secrets.py`) that scans Git commit messages for secrets (e.g., AWS keys, Slack tokens, high-entropy API keys) during the `commit-msg` git hook phase.\n\n### Manual Verification\n\nYou can manually run the pre-commit checks on all files in the repository at any time:\n```bash\npre-commit run --all-files\n```\n\n---\n\n## License\n\nMIT\n"}}}} |