1 line
20 KiB
JSON
1 line
20 KiB
JSON
{"repo": "chinmaygit8765/alfredopensource", "files": {"docs/MODELS.md": {"cc9ba6d3a4d9f033d4fefac4f780fb7351e90679": {"t": 1785746372.4838161, "n": 9944, "h": "4ead7d53003c066d", "c": "# Choosing and connecting a model\n\nALFRED's brain is swappable. Every model sits behind one port\n(`ModelPort`), every structured request goes through one validated call\n(`domain/structured.structured_call`), and the adapter is chosen in one\nplace (`llm.provider` in `config/alfred.yaml`). This guide covers what\nworks, what to pick for your hardware, and the exact config for every\ncommon way of running a model.\n\nThe one-line answer to \"does my model work\": if it is an instruct or chat\ntuned LLM that Ollama or any OpenAI-compatible server can run, yes. That\ncovers essentially every known downloadable free model and every major\nhosted provider.\n\n## Why almost any chat model works\n\nALFRED never trusts a model to freestyle. Every structured request:\n\n1. sends the JSON schema of the expected reply with the request, so\n backends that support constrained decoding (Ollama `format`, the\n OpenAI-dialect `response_format`) are steered to emit valid JSON;\n2. extracts and validates the reply against the pydantic schema\n (`extract_json` strips markdown fences and leading chatter first);\n3. on a validation failure, feeds the errors back and retries, up to 3\n attempts, so a model that fumbles once gets to correct itself.\n\nThis loop is why a 7B free model is genuinely usable: the schema\ndiscipline comes from the harness, not from model obedience. What the\nmodel contributes is judgment; bigger models plan better, but small ones\nstill produce valid, working output.\n\nThree real requirements:\n\n| Requirement | Why |\n|---|---|\n| Instruct/chat tuned | Base (non-instruct) checkpoints ramble and ignore the output contract. Pick anything tagged instruct/chat; every model on ollama.com/library qualifies. |\n| Text in, text out | Embedding, reranker, vision-only, TTS, and diffusion models are not chat models and do not apply. Multimodal chat models (e.g. vision-capable LLMs) work fine; ALFRED just uses their text side. |\n| Roughly 8k+ context | Agent briefs carry the prompt, user model, memories, peer plans, and tool specs. Almost everything current clears this easily. |\n\n## Path 1: local and free with Ollama (the default)\n\nThe default posture: the brain never leaves your machine, costs nothing,\nand needs no key. Install [Ollama](https://ollama.com), pull a model, done.\n\n```yaml\nllm:\n provider: ollama # the default; shown for clarity\n host: \"http://127.0.0.1:11434\"\n name: \"qwen3:8b\"\n fallbacks: [\"qwen2.5:7b\", \"llama3.1:8b\"]\n```\n\n`ensure_model` resolves the name against what is actually pulled and\nwalks `fallbacks` in order, so a missing pull fails loudly at startup\nwith a list of what is available, never mid-conversation.\n\n### Known free model families (all work)\n\nEverything in the Ollama library works; these are the families worth\nknowing. Sizes are the commonly pulled quantized variants.\n\n| Family | Sizes | Notes for ALFRED |\n|---|---|---|\n| Qwen 3 | 0.6B to 32B+ | The default (`qwen3:8b`). Strong JSON discipline and tool-style output for its size. |\n| Qwen 2.5 | 0.5B to 72B | First fallback (`qwen2.5:7b`). Reliable, widely benchmarked. |\n| Llama 3.1 / 3.2 / 3.3 | 1B to 70B | Second fallback (`llama3.1:8b`). 3.2 1B/3B run on very small machines. |\n| Mistral / Ministral / Mistral Small | 3B to 24B | `mistral:7b` is a classic light option; Small 24B is a strong mid-size. |\n| Gemma 3 | 1B to 27B | Good quality per GB; the 4B is a solid low-RAM pick. |\n| Phi-4 | 14B | Strong reasoning for its size; MIT licensed. |\n| DeepSeek-R1 distills | 1.5B to 70B | Reasoning models; see the note on thinking models below. |\n\nPull any of them the same way: `ollama pull <name>`, set `llm.name`, run\n`alfred doctor`.\n\n### Sizing to your hardware\n\nRule of thumb for the default 4-bit quantizations: a model needs roughly\n0.6 to 0.7 GB of RAM (or VRAM) per billion parameters, plus headroom.\n\n| Your machine | Pick | Expect |\n|---|---|---|\n| 8 GB RAM, no GPU | `llama3.2:3b`, `qwen3:4b`, `gemma3:4b` | Fast, decent plans; keep agent prompts tight. |\n| 16 GB RAM or 8 GB VRAM | `qwen3:8b` (default), `llama3.1:8b` | The sweet spot; everything in this repo is tuned around this class. |\n| 32 GB RAM or 16 GB VRAM | `qwen3:14b`, `phi4:14b`, `mistral-small` | Noticeably better judgment in plans and reflections. |\n| 24 GB+ VRAM | `qwen3:32b`, `llama3.3:70b` (48 GB+) | Diminishing returns for ALFRED's workloads, but the reflections get sharp. |\n\nSmaller than 3B works mechanically (the schema loop holds) but plan\nquality drops: expect blunt plans and occasional retry churn.\n\n### A note on thinking/reasoning models\n\nDeepSeek-R1 style models emit a reasoning block before the answer.\nConstrained decoding plus `extract_json`'s chatter-stripping absorb this,\nbut they respond slower and burn tokens on thinking, which a weekly\nplanning run does not need. Prefer a plain instruct model unless you\nspecifically want the reasoning quality.\n\n## Path 2: local and free without Ollama\n\nAnything that serves the OpenAI dialect locally is one config block away.\nSame downloadable models, different runner:\n\n```yaml\nllm:\n provider: openai\n host: \"http://127.0.0.1:1234/v1\" # your server's base URL\n name: \"the-model-id-the-server-shows\"\n```\n\nNo key is needed for local servers; `ALFRED_LLM_API_KEY` can stay unset.\n\n| Server | Default base URL | Notes |\n|---|---|---|\n| LM Studio | `http://127.0.0.1:1234/v1` | Point-and-click model downloads; enable the local server in settings. |\n| llama.cpp (`llama-server`) | `http://127.0.0.1:8080/v1` | Runs raw GGUF files; grammar-enforced JSON supported. |\n| vLLM | `http://127.0.0.1:8000/v1` | Production-grade throughput on a GPU box. |\n| Jan | `http://127.0.0.1:1337/v1` | Desktop app with a bundled server. |\n| KoboldCpp | `http://127.0.0.1:5001/v1` | GGUF runner with an OpenAI-compatible endpoint. |\n| text-generation-webui | `http://127.0.0.1:5000/v1` | Enable the openai extension. |\n| Ollama on another machine | `http://<host>:11434/v1` | Ollama speaks the OpenAI dialect at `/v1`; use this to borrow a beefier box on your LAN. |\n\nServers that reject `response_format` with the usual HTTP 400 are handled\nautomatically: ALFRED retries the request unconstrained and the\nvalidation loop takes over.\n\n## Path 3: hosted APIs (opt-in, needs a key)\n\nWhen you want a bigger brain than your hardware carries. Set the key in\nthe environment, never in config:\n\n```yaml\nllm:\n provider: openai\n host: \"https://openrouter.ai/api/v1\"\n name: \"the-provider's-model-id\"\n```\n\n```powershell\n$env:ALFRED_LLM_API_KEY = \"your-key\"\nalfred doctor\n```\n\n| Provider | Base URL | Notes |\n|---|---|---|\n| OpenRouter | `https://openrouter.ai/api/v1` | One key, one URL, hundreds of models from every lab (including free-tier routes). The easiest single answer. |\n| OpenAI | `https://api.openai.com/v1` | GPT models. |\n| Anthropic | `https://api.anthropic.com/v1` | Claude models via Anthropic's OpenAI-compatible endpoint. |\n| Groq | `https://api.groq.com/openai/v1` | Open models (Llama, Qwen...) at very high speed; generous free tier. |\n| Together | `https://api.together.xyz/v1` | Large open-model catalog. |\n| DeepSeek | `https://api.deepseek.com/v1` | DeepSeek chat and reasoner. |\n| Mistral | `https://api.mistral.ai/v1` | Mistral's hosted models; free tier available. |\n| Google | `https://generativelanguage.googleapis.com/v1beta/openai` | Gemini via Google's OpenAI-compatible endpoint. |\n| xAI | `https://api.x.ai/v1` | Grok models. |\n\nModel ids change as providers ship; check the provider's model list and\nput the exact id in `llm.name`. `alfred doctor` probes the endpoint; a\nname the provider does not list logs a warning, and a truly wrong id\nfails with a readable 404 on first use.\n\nKnown exception: Azure OpenAI authenticates with different headers and a\ndeployment-based URL scheme, so it does not work out of the box; front it\nwith a compatibility proxy (e.g. LiteLLM) if you need it.\n\n### The privacy trade, stated plainly\n\nWith `provider: openai` pointed at a hosted URL, your prompts (which\ninclude your plans, outcomes, memories, and profile summary) leave your\nmachine for that provider. Everything else in ALFRED stays local: the\ndatabase, the audit log, the agents. Governance, allowlists, and tiers\nbehave identically whichever brain answers. Local-first remains the\ndefault and the recommendation; a hosted brain is a trade you make with\neyes open.\n\n## Per-agent model overrides\n\nAny agent can pin its own generation settings in `manifest.yaml`; the\nexecutor passes them through on every call for that agent:\n\n```yaml\nmodel:\n model: \"qwen3:14b\" # must be available on the SAME configured backend\n temperature: 0.2\n max_tokens: 2048\n```\n\nAll agents share the one configured backend (one `ModelPort`), so the\noverride picks a different model on that backend, not a different\nprovider. A practical split: meta agents (qa, scout) on a small fast\nmodel, planning agents on the biggest one your machine holds.\n\n## Troubleshooting\n\n- `alfred doctor` first, always: it probes the backend, resolves the\n model name, and checks the key env var when the provider is `openai`.\n- \"could not reach\": the server is not up, or `llm.host` is wrong. For\n OpenAI-dialect servers the base URL almost always ends in `/v1`.\n- HTTP 401/403: the key in `llm.api_key_env` (default\n `ALFRED_LLM_API_KEY`) is missing, wrong, or lacks access to the model.\n The error names the env var; the key itself is never printed.\n- HTTP 404 on a hosted provider: the model id in `llm.name` is not one\n the provider offers; copy the exact id from their catalog.\n- Garbage or empty plans on a tiny model: move up a size tier, or lower\n `llm.temperature` (0.2 to 0.4 suits planning work).\n- Retry churn in the logs (StructuredCallError after 3 attempts): the\n model is too weak for the schema or the temperature is too high; the\n 8B-class defaults do not have this problem.\n- Everything offline, no model at all: `alfred chat --fake` exercises the\n whole pipeline with a canned brain.\n"}, "9fab63efe9f4c1009e7155ee5fbf9c27db623dd9": {"t": 1785763012.5686202, "n": 9944, "h": "4ead7d53003c066d", "c": "# Choosing and connecting a model\n\nALFRED's brain is swappable. Every model sits behind one port\n(`ModelPort`), every structured request goes through one validated call\n(`domain/structured.structured_call`), and the adapter is chosen in one\nplace (`llm.provider` in `config/alfred.yaml`). This guide covers what\nworks, what to pick for your hardware, and the exact config for every\ncommon way of running a model.\n\nThe one-line answer to \"does my model work\": if it is an instruct or chat\ntuned LLM that Ollama or any OpenAI-compatible server can run, yes. That\ncovers essentially every known downloadable free model and every major\nhosted provider.\n\n## Why almost any chat model works\n\nALFRED never trusts a model to freestyle. Every structured request:\n\n1. sends the JSON schema of the expected reply with the request, so\n backends that support constrained decoding (Ollama `format`, the\n OpenAI-dialect `response_format`) are steered to emit valid JSON;\n2. extracts and validates the reply against the pydantic schema\n (`extract_json` strips markdown fences and leading chatter first);\n3. on a validation failure, feeds the errors back and retries, up to 3\n attempts, so a model that fumbles once gets to correct itself.\n\nThis loop is why a 7B free model is genuinely usable: the schema\ndiscipline comes from the harness, not from model obedience. What the\nmodel contributes is judgment; bigger models plan better, but small ones\nstill produce valid, working output.\n\nThree real requirements:\n\n| Requirement | Why |\n|---|---|\n| Instruct/chat tuned | Base (non-instruct) checkpoints ramble and ignore the output contract. Pick anything tagged instruct/chat; every model on ollama.com/library qualifies. |\n| Text in, text out | Embedding, reranker, vision-only, TTS, and diffusion models are not chat models and do not apply. Multimodal chat models (e.g. vision-capable LLMs) work fine; ALFRED just uses their text side. |\n| Roughly 8k+ context | Agent briefs carry the prompt, user model, memories, peer plans, and tool specs. Almost everything current clears this easily. |\n\n## Path 1: local and free with Ollama (the default)\n\nThe default posture: the brain never leaves your machine, costs nothing,\nand needs no key. Install [Ollama](https://ollama.com), pull a model, done.\n\n```yaml\nllm:\n provider: ollama # the default; shown for clarity\n host: \"http://127.0.0.1:11434\"\n name: \"qwen3:8b\"\n fallbacks: [\"qwen2.5:7b\", \"llama3.1:8b\"]\n```\n\n`ensure_model` resolves the name against what is actually pulled and\nwalks `fallbacks` in order, so a missing pull fails loudly at startup\nwith a list of what is available, never mid-conversation.\n\n### Known free model families (all work)\n\nEverything in the Ollama library works; these are the families worth\nknowing. Sizes are the commonly pulled quantized variants.\n\n| Family | Sizes | Notes for ALFRED |\n|---|---|---|\n| Qwen 3 | 0.6B to 32B+ | The default (`qwen3:8b`). Strong JSON discipline and tool-style output for its size. |\n| Qwen 2.5 | 0.5B to 72B | First fallback (`qwen2.5:7b`). Reliable, widely benchmarked. |\n| Llama 3.1 / 3.2 / 3.3 | 1B to 70B | Second fallback (`llama3.1:8b`). 3.2 1B/3B run on very small machines. |\n| Mistral / Ministral / Mistral Small | 3B to 24B | `mistral:7b` is a classic light option; Small 24B is a strong mid-size. |\n| Gemma 3 | 1B to 27B | Good quality per GB; the 4B is a solid low-RAM pick. |\n| Phi-4 | 14B | Strong reasoning for its size; MIT licensed. |\n| DeepSeek-R1 distills | 1.5B to 70B | Reasoning models; see the note on thinking models below. |\n\nPull any of them the same way: `ollama pull <name>`, set `llm.name`, run\n`alfred doctor`.\n\n### Sizing to your hardware\n\nRule of thumb for the default 4-bit quantizations: a model needs roughly\n0.6 to 0.7 GB of RAM (or VRAM) per billion parameters, plus headroom.\n\n| Your machine | Pick | Expect |\n|---|---|---|\n| 8 GB RAM, no GPU | `llama3.2:3b`, `qwen3:4b`, `gemma3:4b` | Fast, decent plans; keep agent prompts tight. |\n| 16 GB RAM or 8 GB VRAM | `qwen3:8b` (default), `llama3.1:8b` | The sweet spot; everything in this repo is tuned around this class. |\n| 32 GB RAM or 16 GB VRAM | `qwen3:14b`, `phi4:14b`, `mistral-small` | Noticeably better judgment in plans and reflections. |\n| 24 GB+ VRAM | `qwen3:32b`, `llama3.3:70b` (48 GB+) | Diminishing returns for ALFRED's workloads, but the reflections get sharp. |\n\nSmaller than 3B works mechanically (the schema loop holds) but plan\nquality drops: expect blunt plans and occasional retry churn.\n\n### A note on thinking/reasoning models\n\nDeepSeek-R1 style models emit a reasoning block before the answer.\nConstrained decoding plus `extract_json`'s chatter-stripping absorb this,\nbut they respond slower and burn tokens on thinking, which a weekly\nplanning run does not need. Prefer a plain instruct model unless you\nspecifically want the reasoning quality.\n\n## Path 2: local and free without Ollama\n\nAnything that serves the OpenAI dialect locally is one config block away.\nSame downloadable models, different runner:\n\n```yaml\nllm:\n provider: openai\n host: \"http://127.0.0.1:1234/v1\" # your server's base URL\n name: \"the-model-id-the-server-shows\"\n```\n\nNo key is needed for local servers; `ALFRED_LLM_API_KEY` can stay unset.\n\n| Server | Default base URL | Notes |\n|---|---|---|\n| LM Studio | `http://127.0.0.1:1234/v1` | Point-and-click model downloads; enable the local server in settings. |\n| llama.cpp (`llama-server`) | `http://127.0.0.1:8080/v1` | Runs raw GGUF files; grammar-enforced JSON supported. |\n| vLLM | `http://127.0.0.1:8000/v1` | Production-grade throughput on a GPU box. |\n| Jan | `http://127.0.0.1:1337/v1` | Desktop app with a bundled server. |\n| KoboldCpp | `http://127.0.0.1:5001/v1` | GGUF runner with an OpenAI-compatible endpoint. |\n| text-generation-webui | `http://127.0.0.1:5000/v1` | Enable the openai extension. |\n| Ollama on another machine | `http://<host>:11434/v1` | Ollama speaks the OpenAI dialect at `/v1`; use this to borrow a beefier box on your LAN. |\n\nServers that reject `response_format` with the usual HTTP 400 are handled\nautomatically: ALFRED retries the request unconstrained and the\nvalidation loop takes over.\n\n## Path 3: hosted APIs (opt-in, needs a key)\n\nWhen you want a bigger brain than your hardware carries. Set the key in\nthe environment, never in config:\n\n```yaml\nllm:\n provider: openai\n host: \"https://openrouter.ai/api/v1\"\n name: \"the-provider's-model-id\"\n```\n\n```powershell\n$env:ALFRED_LLM_API_KEY = \"your-key\"\nalfred doctor\n```\n\n| Provider | Base URL | Notes |\n|---|---|---|\n| OpenRouter | `https://openrouter.ai/api/v1` | One key, one URL, hundreds of models from every lab (including free-tier routes). The easiest single answer. |\n| OpenAI | `https://api.openai.com/v1` | GPT models. |\n| Anthropic | `https://api.anthropic.com/v1` | Claude models via Anthropic's OpenAI-compatible endpoint. |\n| Groq | `https://api.groq.com/openai/v1` | Open models (Llama, Qwen...) at very high speed; generous free tier. |\n| Together | `https://api.together.xyz/v1` | Large open-model catalog. |\n| DeepSeek | `https://api.deepseek.com/v1` | DeepSeek chat and reasoner. |\n| Mistral | `https://api.mistral.ai/v1` | Mistral's hosted models; free tier available. |\n| Google | `https://generativelanguage.googleapis.com/v1beta/openai` | Gemini via Google's OpenAI-compatible endpoint. |\n| xAI | `https://api.x.ai/v1` | Grok models. |\n\nModel ids change as providers ship; check the provider's model list and\nput the exact id in `llm.name`. `alfred doctor` probes the endpoint; a\nname the provider does not list logs a warning, and a truly wrong id\nfails with a readable 404 on first use.\n\nKnown exception: Azure OpenAI authenticates with different headers and a\ndeployment-based URL scheme, so it does not work out of the box; front it\nwith a compatibility proxy (e.g. LiteLLM) if you need it.\n\n### The privacy trade, stated plainly\n\nWith `provider: openai` pointed at a hosted URL, your prompts (which\ninclude your plans, outcomes, memories, and profile summary) leave your\nmachine for that provider. Everything else in ALFRED stays local: the\ndatabase, the audit log, the agents. Governance, allowlists, and tiers\nbehave identically whichever brain answers. Local-first remains the\ndefault and the recommendation; a hosted brain is a trade you make with\neyes open.\n\n## Per-agent model overrides\n\nAny agent can pin its own generation settings in `manifest.yaml`; the\nexecutor passes them through on every call for that agent:\n\n```yaml\nmodel:\n model: \"qwen3:14b\" # must be available on the SAME configured backend\n temperature: 0.2\n max_tokens: 2048\n```\n\nAll agents share the one configured backend (one `ModelPort`), so the\noverride picks a different model on that backend, not a different\nprovider. A practical split: meta agents (qa, scout) on a small fast\nmodel, planning agents on the biggest one your machine holds.\n\n## Troubleshooting\n\n- `alfred doctor` first, always: it probes the backend, resolves the\n model name, and checks the key env var when the provider is `openai`.\n- \"could not reach\": the server is not up, or `llm.host` is wrong. For\n OpenAI-dialect servers the base URL almost always ends in `/v1`.\n- HTTP 401/403: the key in `llm.api_key_env` (default\n `ALFRED_LLM_API_KEY`) is missing, wrong, or lacks access to the model.\n The error names the env var; the key itself is never printed.\n- HTTP 404 on a hosted provider: the model id in `llm.name` is not one\n the provider offers; copy the exact id from their catalog.\n- Garbage or empty plans on a tiny model: move up a size tier, or lower\n `llm.temperature` (0.2 to 0.4 suits planning work).\n- Retry churn in the logs (StructuredCallError after 3 attempts): the\n model is too weak for the schema or the temperature is too high; the\n 8B-class defaults do not have this problem.\n- Everything offline, no model at all: `alfred chat --fake` exercises the\n whole pipeline with a canned brain.\n"}}}} |