{"repo": "alpha-swarm/alpha-swarm2", "files": {"docs/INFRASTRUCTURE.md": {"38978958c56a1008227899412d40ed6f46127b45": {"t": 1785747177.278746, "n": 23282, "h": "dbf605c04d8c9b0d", "c": "# alpha-swarm Infrastructure\n\n## System Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│                          CLIENT BOUNDARY                                │\n│                                                                         │\n│   CLI ─────┐                                                            │\n│   Dashboard ┼──► gateway ──► swarm/control ──► project/manager          │\n│   MCP Server┘       │                              │                    │\n│                     │                              │                    │\n│              events/bus ◄──────────────────────────┘                    │\n└──────────────────────┬──────────────────────────────────────────────────┘\n                       │\n┌──────────────────────┼──────────────────────────────────────────────────┐\n│                 ORCHESTRATION BOUNDARY                                   │\n│                      │                                                  │\n│              ┌───────▼────────┐                                         │\n│              │  orchestrator  │                                         │\n│              │  (goal → tasks)│                                         │\n│              └──┬──┬──┬──┬───┘                                         │\n│                 │  │  │  │                                              │\n│    ┌────────────┘  │  │  └────────────┐                                │\n│    ▼               ▼  ▼               ▼                                │\n│ ┌──────┐  ┌──────┐ ┌──────┐  ┌──────┐                                 │\n│ │agent │  │agent │ │agent │  │agent │   (N one-shot workers)           │\n│ │  #1  │  │  #2  │ │  #3  │  │  #N  │                                 │\n│ └──┬───┘  └──┬───┘ └──┬───┘  └──┬───┘                                 │\n└────┼─────────┼────────┼─────────┼──────────────────────────────────────┘\n     │         │        │         │\n┌────┼─────────┼────────┼─────────┼──────────────────────────────────────┐\n│    │    INFERENCE BOUNDARY      │                                      │\n│    │         │        │         │                                      │\n│    └─────────┴────┬───┴─────────┘                                      │\n│                   ▼                                                    │\n│         ┌──────────────────┐                                           │\n│         │ inference-router │   routes by complexity + availability     │\n│         └────┬─────────┬───┘                                           │\n│              │         │                                               │\n│         ┌────▼───┐ ┌───▼────┐                                          │\n│         │ claude │ │ ollama │   ... future: openai, local llama.cpp    │\n│         │ prov.  │ │ prov.  │                                          │\n│         └────┬───┘ └───┬────┘                                          │\n│              │         │                                               │\n└──────────────┼─────────┼──────────────────────────────────────────────┘\n               │         │\n┌──────────────┼─────────┼──────────────────────────────────────────────┐\n│         PROVIDER BOUNDARY    (native Rust, host OS access)             │\n│              │         │                                               │\n│         ┌────▼───┐ ┌───▼────┐ ┌──────────┐ ┌──────────┐              │\n│         │ virtfs │ │quality │ │knowledge │ │ events   │              │\n│         │ prov.  │ │gate p. │ │base prov.│ │ bus prov.│              │\n│         └────┬───┘ └───┬────┘ └────┬─────┘ └────┬─────┘              │\n└──────────────┼─────────┼──────────┼────────────┼──────────────────────┘\n               │         │          │            │\n┌──────────────┼─────────┼──────────┼────────────┼──────────────────────┐\n│         EXTERNAL SERVICE BOUNDARY │            │                       │\n│              ▼         ▼          ▼            ▼                       │\n│         ┌──────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐              │\n│         │ git  │ │ cargo/   │ │ SurrealDB│ │   NATS   │              │\n│         │ repos│ │ npm/go   │ │ :8000    │ │  :4222   │              │\n│         └──────┘ └──────────┘ └──────────┘ └──────────┘              │\n│                                                                        │\n│    ┌──────────┐  ┌──────────┐                                          │\n│    │ Anthropic│  │  Ollama  │                                          │\n│    │ API      │  │  :11434  │                                          │\n│    └──────────┘  └──────────┘                                          │\n└────────────────────────────────────────────────────────────────────────┘\n```\n\n## Physical Deployment (Current Setup)\n\n```\n┌──────────────────────────┐          ┌──────────────────────────┐\n│  LOCAL MACHINE            │          │  CSATAPACI                │\n│                           │          │  M2 Max · 96GB · 3.6TB   │\n│  Tailscale: 100.79.38.122│◄────────►│  Tailscale: 100.81.10.8  │\n│                           │   NATS   │                           │\n│  ┌─────────────────────┐  │  cluster │  ┌─────────────────────┐  │\n│  │ NATS node 1         │  │◄────────►│  │ NATS node 2         │  │\n│  │ :4222 :6222 :8222   │  │          │  │ :4222 :6222 :8222   │  │\n│  └─────────────────────┘  │          │  └─────────────────────┘  │\n│                           │          │                           │\n│  ┌─────────────────────┐  │          │  ┌─────────────────────┐  │\n│  │ SurrealDB :8000     │  │          │  │ Ollama :11434       │  │\n│  │ (primary data store)│  │          │  │ · qwen2.5-coder:7b  │  │\n│  └─────────────────────┘  │          │  │ · deepseek-coder:33b│  │\n│                           │          │  │ · codellama:34b     │  │\n│  ┌─────────────────────┐  │          │  └─────────────────────┘  │\n│  │ WasmCloud Host      │  │          │                           │\n│  │ role: orchestrator   │  │          │  ┌─────────────────────┐  │\n│  │                      │  │          │  │ WasmCloud Host      │  │\n│  │ Components:          │  │          │  │ role: inference      │  │\n│  │  · gateway           │  │          │  │ ram: 96gb            │  │\n│  │  · orchestrator      │  │          │  │                      │  │\n│  │  · inference-router  │  │          │  │ Components:          │  │\n│  │                      │  │          │  │  · agent-worker (N)  │  │\n│  │ Providers:           │  │          │  │                      │  │\n│  │  · claude-provider   │  │          │  │ Providers:           │  │\n│  │  · knowledge-base    │  │          │  │  · ollama-provider   │  │\n│  │  · virtfs            │  │          │  │  · virtfs            │  │\n│  │  · quality-gate      │  │          │  │  · quality-gate      │  │\n│  └─────────────────────┘  │          │  └─────────────────────┘  │\n└──────────────────────────┘          └──────────────────────────┘\n\n┌──────────────────────────┐\n│  malna (RPi)              │\n│  8GB RAM · 512GB SSD      │\n│  Tailscale: TBD           │\n│                           │\n│  ┌─────────────────────┐  │\n│  │ NATS node 3         │  │   ← 3rd quorum node\n│  │ :4222 :6222         │  │\n│  └─────────────────────┘  │\n│                           │\n│  ┌─────────────────────┐  │\n│  │ SurrealDB (replica) │  │   ← persistent store\n│  │ :8000               │  │\n│  └─────────────────────┘  │\n│                           │\n│  ┌─────────────────────┐  │\n│  │ WasmCloud Host      │  │\n│  │ role: infra+quality  │  │\n│  │                      │  │\n│  │ Components:          │  │\n│  │  · quality-gate      │  │   ← runs build/test\n│  └─────────────────────┘  │\n└──────────────────────────┘\n```\n\n## Error Boundaries\n\n### Boundary 1: External Services\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  EXTERNAL SERVICE BOUNDARY                                    │\n│                                                               │\n│  Ollama (:11434)                                              │\n│  ├─ DOWN  → inference-router marks backend unavailable,      │\n│  │          falls back to Claude API                          │\n│  ├─ SLOW  → per-request timeout (configurable, default 120s) │\n│  ├─ MODEL MISSING → swarm-error::not-found                   │\n│  └─ OOM   → inference-failed, router escalates to smaller    │\n│             model or different backend                        │\n│                                                               │\n│  Claude API                                                   │\n│  ├─ 401/403   → swarm-error::inference-failed (bad API key)  │\n│  ├─ 429       → rate limited, backoff + retry                │\n│  ├─ 500/529   → overloaded, fallback to Ollama if available  │\n│  └─ TIMEOUT   → inference-failed                             │\n│                                                               │\n│  SurrealDB (:8000)                                            │\n│  ├─ DOWN → knowledge-base-provider returns io-error           │\n│  │         system enters DEGRADED mode:                       │\n│  │         agents skip knowledge queries, still functional    │\n│  ├─ QUERY TIMEOUT → io-error with context                    │\n│  └─ NAMESPACE MISSING → not-found                            │\n│                                                               │\n│  NATS (:4222)                                                 │\n│  ├─ DOWN → lattice partitions, no new scheduling             │\n│  │         in-flight agents finish locally                     │\n│  ├─ PARTITION → 2-node cluster has no quorum                 │\n│  │   each side operates independently until reconnect         │\n│  └─ RECONNECT → automatic, NATS client handles               │\n│                                                               │\n│  Git repos                                                    │\n│  ├─ LOCK CONTENTION → virtfs retries with backoff            │\n│  ├─ CORRUPT REPO → io-error, project marked as error         │\n│  └─ DISK FULL → io-error, checked before worktree creation   │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 2: Inference Router\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  INFERENCE BOUNDARY                                           │\n│                                                               │\n│  The router is the single point where backend failures are    │\n│  absorbed. Agents never see backend-specific errors.          │\n│                                                               │\n│  Routing strategy:                                            │\n│    simple task  → prefer Ollama (qwen2.5-coder:7b)           │\n│    medium task  → prefer Ollama (deepseek-coder:33b)         │\n│    complex task → prefer Claude API                           │\n│                                                               │\n│  Fallback chain:                                              │\n│    preferred backend fails                                    │\n│         │                                                     │\n│         ▼                                                     │\n│    try next backend with equivalent model                     │\n│         │                                                     │\n│         ▼                                                     │\n│    try any backend with any model                             │\n│         │                                                     │\n│         ▼                                                     │\n│    return swarm-error::inference-failed                       │\n│                                                               │\n│  Circuit breaker per backend:                                 │\n│    5 consecutive failures → open circuit for 30s              │\n│    during open → fast-fail, skip to next backend              │\n│    after 30s → half-open, probe with one request              │\n│    probe succeeds → close circuit, resume normal              │\n│                                                               │\n│  Health events emitted:                                       │\n│    backend.claude.up / backend.claude.down                    │\n│    backend.ollama.up / backend.ollama.down                    │\n│    inference.fallback (when preferred backend unavailable)    │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 3: Providers\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  PROVIDER BOUNDARY                                            │\n│                                                               │\n│  Each provider:                                               │\n│  ├─ ISOLATES external service failures                        │\n│  │   Never panics — all errors become result<_, swarm-error> │\n│  ├─ RETRIES internally (3x with exponential backoff)         │\n│  │   before returning error to component                      │\n│  ├─ HEALTH CHECK via WasmCloud lifecycle                      │\n│  │   unhealthy provider → WasmCloud restarts it               │\n│  └─ Emits events on state changes                             │\n│                                                               │\n│  virtfs-provider:                                             │\n│  ├─ Git lock contention → retry with backoff                  │\n│  ├─ Worktree create fails → io-error                          │\n│  ├─ Disk full → io-error (checked before worktree creation)  │\n│  └─ Concurrent repo access → mutex per repo                   │\n│                                                               │\n│  quality-gate-provider:                                       │\n│  ├─ Build tool not found → quality-gate-failed                │\n│  ├─ Process timeout (300s) → kill + quality-gate-failed       │\n│  ├─ Non-zero exit → structured check-result with              │\n│  │   stdout/stderr captured (not a swarm-error)               │\n│  └─ OOM during build → quality-gate-failed                    │\n│                                                               │\n│  knowledge-base-provider:                                     │\n│  ├─ SurrealDB down → reconnect loop (5s interval)            │\n│  ├─ After 30s unreachable → emit degraded event               │\n│  │   Components skip knowledge queries, still work            │\n│  └─ Query timeout (10s) → io-error                            │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 4: Orchestration\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  ORCHESTRATION BOUNDARY                                       │\n│                                                               │\n│  orchestrator:                                                │\n│  ├─ PLANNING FAILS (LLM error) →                             │\n│  │   goal-status = failed, event emitted                      │\n│  ├─ AGENT FAILS (task-status = failed) →                      │\n│  │   retry strategy:                                          │\n│  │   1. Retry same model (transient?)        max 1 retry     │\n│  │   2. Retry with larger model              max 1 retry     │\n│  │   3. Mark task failed, continue others                     │\n│  │   4. If >50% tasks fail → fail the goal                   │\n│  ├─ MERGE CONFLICT →                                          │\n│  │   re-run conflicting agent with updated base               │\n│  │   max 2 retries, then fail that task                       │\n│  └─ AGENT TIMEOUT (300s) →                                    │\n│     task unclaimed in SurrealDB after TTL                      │\n│     another agent can pick it up                               │\n│                                                               │\n│  agent-worker:                                                │\n│  ├─ WASM TRAP (panic, OOM) →                                  │\n│  │   WasmCloud catches it, task stays unclaimed                │\n│  │   orchestrator re-dispatches after timeout                  │\n│  ├─ INFERENCE BAD OUTPUT (unparseable) →                       │\n│  │   task-status = failed with error message                   │\n│  │   stored in knowledge-base for future avoidance             │\n│  └─ QUALITY GATE FAILS →                                      │\n│     diff discarded, error stored in knowledge-base             │\n│     task retried with error context fed to next attempt        │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 5: Client\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  CLIENT BOUNDARY                                              │\n│                                                               │\n│  gateway:                                                     │\n│  ├─ All swarm-errors → structured response to client          │\n│  ├─ Event bus down → client gets stale status                 │\n│  │   (poll-based, client handles staleness)                   │\n│  └─ Invalid request → swarm-error::internal                   │\n│                                                               │\n│  CLI:                                                         │\n│  ├─ NATS unreachable →                                        │\n│  │   \"Cannot reach swarm. Is NATS running?\"                   │\n│  ├─ Timeout waiting for goal → show partial status            │\n│  └─ Ctrl-C → goal keeps running (async by design)            │\n│     \"Goal abc123 still running. Check with:                   │\n│      alpha-swarm status abc123\"                                │\n└──────────────────────────────────────────────────────────────┘\n```\n\n## Error Propagation Flow\n\n```\n                    HAPPY PATH\n                    ──────────\nsubmit-goal ──► orchestrator.plan ──► agents[N] ──► merge ──► done\n                                         │\n                    ERROR PATHS           │\n                    ───────────           │\n                                         ▼\n                              ┌─────────────────────┐\n                              │ Agent fails?         │\n                              └──────┬──────────────┘\n                                     │\n                    ┌────────────────┬┴───────────────┐\n                    ▼                ▼                 ▼\n               inference         quality           wasm trap\n               failed            failed            (panic/OOM)\n                    │                │                 │\n                    ▼                ▼                 ▼\n               ┌────────────────────────────────────────┐\n               │ Store error in knowledge base          │\n               │ (embedding for future avoidance)       │\n               └───────────────┬────────────────────────┘\n                               │\n                      ┌────────▼────────┐\n                      │ Retry logic:    │\n                      │ 1. Same model   │\n                      │ 2. Bigger model │\n                      │ 3. Give up task │\n                      └────────┬────────┘\n                               │\n                  ┌────────────┼────────────┐\n                  ▼            ▼            ▼\n               retried      task failed   >50% failed\n               (loop back)  (continue     (fail entire\n                             others)       goal)\n```\n\n## Graceful Degradation Matrix\n\n```\n┌──────────────┬───────────┬──────────┬────────┬──────────────────┐\n│ What's Down  │ Ollama    │ Claude   │SurrealDB│ NATS             │\n│              │           │ API      │         │                  │\n├──────────────┼───────────┼──────────┼─────────┼──────────────────┤\n│ Can plan?    │ YES (1)   │ YES (2)  │ YES     │ NO               │\n│ Can run      │ YES (1)   │ YES (2)  │ YES (3) │ NO               │\n│  agents?     │           │          │         │                  │\n│ Can query    │ YES       │ YES      │ NO      │ YES              │\n│  knowledge?  │           │          │         │                  │\n│ Can emit     │ YES       │ YES      │ YES     │ NO               │\n│  events?     │           │          │         │                  │\n│ Can run      │ YES       │ YES      │ YES     │ NO               │\n│  quality     │           │          │         │                  │\n│  checks?     │           │          │         │                  │\n├──────────────┼───────────┼──────────┼─────────┼──────────────────┤\n│ System state │ DEGRADED  │ DEGRADED │DEGRADED │ PARTITIONED      │\n│              │ fallback  │ fallback │ no      │ in-flight agents │\n│              │ to Claude │ to Ollama│ knowledge│ finish locally   │\n└──────────────┴───────────┴──────────┴─────────┴──────────────────┘\n\n(1) Falls back to Claude API via inference-router\n(2) Falls back to Ollama via inference-router\n(3) Agents work but skip knowledge queries (no past context, no dedup)\n```\n\n## Startup & Shutdown\n\n### Startup Order\n\n```\n1. NATS             ← must be first (lattice backbone)\n   ├─ verify: nats server check\n   │\n2. SurrealDB        ← can start in parallel with step 1\n   ├─ verify: surreal isready\n   │\n3. Ollama           ← can start in parallel with step 1\n   ├─ verify: curl http://localhost:11434/\n   │\n4. WasmCloud Host   ← requires NATS\n   ├─ verify: wash get hosts\n   │\n5. wash app deploy wadm.yaml\n   ├─ verify: wash get inventory\n   └─ verify: wash get links\n```\n\n### Shutdown Order (Reverse)\n\n```\n1. wash app undeploy alpha-swarm     ← drain agents (wait 60s max)\n2. Stop WasmCloud Host\n3. Stop Ollama                       ← or leave running\n4. Stop SurrealDB\n5. Stop NATS                         ← last\n```\n\n### Scripts\n\n```bash\n# Local machine\n./scripts/infra-up-local.sh          # Start NATS + SurrealDB + WasmCloud\n./scripts/infra-down-local.sh        # Graceful shutdown\n\n# csatapaci (runs commands via SSH)\n./scripts/infra-up-csatapaci.sh      # Start NATS + SurrealDB + Ollama + WasmCloud\n./scripts/infra-down-csatapaci.sh    # Graceful shutdown\n\n# Both machines\n./scripts/infra-status.sh            # Check all services + lattice\n```\n\n## Configuration\n\n### Environment Variables\n\n```bash\n# NATS\nALPHA_SWARM_NATS_URL=nats://127.0.0.1:4222\n\n# SurrealDB\nALPHA_SWARM_SURREALDB_URL=ws://127.0.0.1:8000\nALPHA_SWARM_SURREALDB_NS=alpha_swarm\nALPHA_SWARM_SURREALDB_DB=swarm\nALPHA_SWARM_SURREALDB_USER=root\nALPHA_SWARM_SURREALDB_PASS=root\n\n# Ollama\nALPHA_SWARM_OLLAMA_URL=http://csatapaci:11434\nALPHA_SWARM_OLLAMA_TIMEOUT_SECS=120\n\n# Claude API\nALPHA_SWARM_CLAUDE_API_KEY=sk-ant-...\nALPHA_SWARM_CLAUDE_MODEL=claude-sonnet-4-20250514\n\n# Provider behavior\nALPHA_SWARM_RETRY_MAX=3\nALPHA_SWARM_RETRY_BACKOFF_MS=1000\nALPHA_SWARM_CIRCUIT_BREAKER_THRESHOLD=5\nALPHA_SWARM_CIRCUIT_BREAKER_RESET_SECS=30\n\n# Agent behavior\nALPHA_SWARM_TASK_TIMEOUT_SECS=300\nALPHA_SWARM_GOAL_FAIL_THRESHOLD=0.5\n```\n\n### WasmCloud Host Labels\n\n```toml\n# Local machine\n[labels]\nalpha-swarm-role = \"orchestrator\"\nalpha-swarm-gpu = \"false\"\n\n# csatapaci\n[labels]\nalpha-swarm-role = \"inference\"\nalpha-swarm-gpu = \"false\"\nalpha-swarm-ram-gb = \"96\"\nalpha-swarm-models = \"deepseek-coder:33b,codellama:34b,qwen2.5-coder:7b\"\n```\n\n### NATS Cluster\n\n```\nNode 1 (local):     100.79.38.122:4222 (client) / :6222 (cluster)\nNode 2 (csatapaci): 100.81.10.8:4222   (client) / :6222 (cluster)\nCluster name:       alpha-swarm\nJetStream:          enabled on both nodes\n```\n"}, "7632ac600ea376a752457eb7df2e703404f00903": {"t": 1785771009.960667, "n": 23282, "h": "dbf605c04d8c9b0d", "c": "# alpha-swarm Infrastructure\n\n## System Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│                          CLIENT BOUNDARY                                │\n│                                                                         │\n│   CLI ─────┐                                                            │\n│   Dashboard ┼──► gateway ──► swarm/control ──► project/manager          │\n│   MCP Server┘       │                              │                    │\n│                     │                              │                    │\n│              events/bus ◄──────────────────────────┘                    │\n└──────────────────────┬──────────────────────────────────────────────────┘\n                       │\n┌──────────────────────┼──────────────────────────────────────────────────┐\n│                 ORCHESTRATION BOUNDARY                                   │\n│                      │                                                  │\n│              ┌───────▼────────┐                                         │\n│              │  orchestrator  │                                         │\n│              │  (goal → tasks)│                                         │\n│              └──┬──┬──┬──┬───┘                                         │\n│                 │  │  │  │                                              │\n│    ┌────────────┘  │  │  └────────────┐                                │\n│    ▼               ▼  ▼               ▼                                │\n│ ┌──────┐  ┌──────┐ ┌──────┐  ┌──────┐                                 │\n│ │agent │  │agent │ │agent │  │agent │   (N one-shot workers)           │\n│ │  #1  │  │  #2  │ │  #3  │  │  #N  │                                 │\n│ └──┬───┘  └──┬───┘ └──┬───┘  └──┬───┘                                 │\n└────┼─────────┼────────┼─────────┼──────────────────────────────────────┘\n     │         │        │         │\n┌────┼─────────┼────────┼─────────┼──────────────────────────────────────┐\n│    │    INFERENCE BOUNDARY      │                                      │\n│    │         │        │         │                                      │\n│    └─────────┴────┬───┴─────────┘                                      │\n│                   ▼                                                    │\n│         ┌──────────────────┐                                           │\n│         │ inference-router │   routes by complexity + availability     │\n│         └────┬─────────┬───┘                                           │\n│              │         │                                               │\n│         ┌────▼───┐ ┌───▼────┐                                          │\n│         │ claude │ │ ollama │   ... future: openai, local llama.cpp    │\n│         │ prov.  │ │ prov.  │                                          │\n│         └────┬───┘ └───┬────┘                                          │\n│              │         │                                               │\n└──────────────┼─────────┼──────────────────────────────────────────────┘\n               │         │\n┌──────────────┼─────────┼──────────────────────────────────────────────┐\n│         PROVIDER BOUNDARY    (native Rust, host OS access)             │\n│              │         │                                               │\n│         ┌────▼───┐ ┌───▼────┐ ┌──────────┐ ┌──────────┐              │\n│         │ virtfs │ │quality │ │knowledge │ │ events   │              │\n│         │ prov.  │ │gate p. │ │base prov.│ │ bus prov.│              │\n│         └────┬───┘ └───┬────┘ └────┬─────┘ └────┬─────┘              │\n└──────────────┼─────────┼──────────┼────────────┼──────────────────────┘\n               │         │          │            │\n┌──────────────┼─────────┼──────────┼────────────┼──────────────────────┐\n│         EXTERNAL SERVICE BOUNDARY │            │                       │\n│              ▼         ▼          ▼            ▼                       │\n│         ┌──────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐              │\n│         │ git  │ │ cargo/   │ │ SurrealDB│ │   NATS   │              │\n│         │ repos│ │ npm/go   │ │ :8000    │ │  :4222   │              │\n│         └──────┘ └──────────┘ └──────────┘ └──────────┘              │\n│                                                                        │\n│    ┌──────────┐  ┌──────────┐                                          │\n│    │ Anthropic│  │  Ollama  │                                          │\n│    │ API      │  │  :11434  │                                          │\n│    └──────────┘  └──────────┘                                          │\n└────────────────────────────────────────────────────────────────────────┘\n```\n\n## Physical Deployment (Current Setup)\n\n```\n┌──────────────────────────┐          ┌──────────────────────────┐\n│  LOCAL MACHINE            │          │  CSATAPACI                │\n│                           │          │  M2 Max · 96GB · 3.6TB   │\n│  Tailscale: 100.79.38.122│◄────────►│  Tailscale: 100.81.10.8  │\n│                           │   NATS   │                           │\n│  ┌─────────────────────┐  │  cluster │  ┌─────────────────────┐  │\n│  │ NATS node 1         │  │◄────────►│  │ NATS node 2         │  │\n│  │ :4222 :6222 :8222   │  │          │  │ :4222 :6222 :8222   │  │\n│  └─────────────────────┘  │          │  └─────────────────────┘  │\n│                           │          │                           │\n│  ┌─────────────────────┐  │          │  ┌─────────────────────┐  │\n│  │ SurrealDB :8000     │  │          │  │ Ollama :11434       │  │\n│  │ (primary data store)│  │          │  │ · qwen2.5-coder:7b  │  │\n│  └─────────────────────┘  │          │  │ · deepseek-coder:33b│  │\n│                           │          │  │ · codellama:34b     │  │\n│  ┌─────────────────────┐  │          │  └─────────────────────┘  │\n│  │ WasmCloud Host      │  │          │                           │\n│  │ role: orchestrator   │  │          │  ┌─────────────────────┐  │\n│  │                      │  │          │  │ WasmCloud Host      │  │\n│  │ Components:          │  │          │  │ role: inference      │  │\n│  │  · gateway           │  │          │  │ ram: 96gb            │  │\n│  │  · orchestrator      │  │          │  │                      │  │\n│  │  · inference-router  │  │          │  │ Components:          │  │\n│  │                      │  │          │  │  · agent-worker (N)  │  │\n│  │ Providers:           │  │          │  │                      │  │\n│  │  · claude-provider   │  │          │  │ Providers:           │  │\n│  │  · knowledge-base    │  │          │  │  · ollama-provider   │  │\n│  │  · virtfs            │  │          │  │  · virtfs            │  │\n│  │  · quality-gate      │  │          │  │  · quality-gate      │  │\n│  └─────────────────────┘  │          │  └─────────────────────┘  │\n└──────────────────────────┘          └──────────────────────────┘\n\n┌──────────────────────────┐\n│  malna (RPi)              │\n│  8GB RAM · 512GB SSD      │\n│  Tailscale: TBD           │\n│                           │\n│  ┌─────────────────────┐  │\n│  │ NATS node 3         │  │   ← 3rd quorum node\n│  │ :4222 :6222         │  │\n│  └─────────────────────┘  │\n│                           │\n│  ┌─────────────────────┐  │\n│  │ SurrealDB (replica) │  │   ← persistent store\n│  │ :8000               │  │\n│  └─────────────────────┘  │\n│                           │\n│  ┌─────────────────────┐  │\n│  │ WasmCloud Host      │  │\n│  │ role: infra+quality  │  │\n│  │                      │  │\n│  │ Components:          │  │\n│  │  · quality-gate      │  │   ← runs build/test\n│  └─────────────────────┘  │\n└──────────────────────────┘\n```\n\n## Error Boundaries\n\n### Boundary 1: External Services\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  EXTERNAL SERVICE BOUNDARY                                    │\n│                                                               │\n│  Ollama (:11434)                                              │\n│  ├─ DOWN  → inference-router marks backend unavailable,      │\n│  │          falls back to Claude API                          │\n│  ├─ SLOW  → per-request timeout (configurable, default 120s) │\n│  ├─ MODEL MISSING → swarm-error::not-found                   │\n│  └─ OOM   → inference-failed, router escalates to smaller    │\n│             model or different backend                        │\n│                                                               │\n│  Claude API                                                   │\n│  ├─ 401/403   → swarm-error::inference-failed (bad API key)  │\n│  ├─ 429       → rate limited, backoff + retry                │\n│  ├─ 500/529   → overloaded, fallback to Ollama if available  │\n│  └─ TIMEOUT   → inference-failed                             │\n│                                                               │\n│  SurrealDB (:8000)                                            │\n│  ├─ DOWN → knowledge-base-provider returns io-error           │\n│  │         system enters DEGRADED mode:                       │\n│  │         agents skip knowledge queries, still functional    │\n│  ├─ QUERY TIMEOUT → io-error with context                    │\n│  └─ NAMESPACE MISSING → not-found                            │\n│                                                               │\n│  NATS (:4222)                                                 │\n│  ├─ DOWN → lattice partitions, no new scheduling             │\n│  │         in-flight agents finish locally                     │\n│  ├─ PARTITION → 2-node cluster has no quorum                 │\n│  │   each side operates independently until reconnect         │\n│  └─ RECONNECT → automatic, NATS client handles               │\n│                                                               │\n│  Git repos                                                    │\n│  ├─ LOCK CONTENTION → virtfs retries with backoff            │\n│  ├─ CORRUPT REPO → io-error, project marked as error         │\n│  └─ DISK FULL → io-error, checked before worktree creation   │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 2: Inference Router\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  INFERENCE BOUNDARY                                           │\n│                                                               │\n│  The router is the single point where backend failures are    │\n│  absorbed. Agents never see backend-specific errors.          │\n│                                                               │\n│  Routing strategy:                                            │\n│    simple task  → prefer Ollama (qwen2.5-coder:7b)           │\n│    medium task  → prefer Ollama (deepseek-coder:33b)         │\n│    complex task → prefer Claude API                           │\n│                                                               │\n│  Fallback chain:                                              │\n│    preferred backend fails                                    │\n│         │                                                     │\n│         ▼                                                     │\n│    try next backend with equivalent model                     │\n│         │                                                     │\n│         ▼                                                     │\n│    try any backend with any model                             │\n│         │                                                     │\n│         ▼                                                     │\n│    return swarm-error::inference-failed                       │\n│                                                               │\n│  Circuit breaker per backend:                                 │\n│    5 consecutive failures → open circuit for 30s              │\n│    during open → fast-fail, skip to next backend              │\n│    after 30s → half-open, probe with one request              │\n│    probe succeeds → close circuit, resume normal              │\n│                                                               │\n│  Health events emitted:                                       │\n│    backend.claude.up / backend.claude.down                    │\n│    backend.ollama.up / backend.ollama.down                    │\n│    inference.fallback (when preferred backend unavailable)    │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 3: Providers\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  PROVIDER BOUNDARY                                            │\n│                                                               │\n│  Each provider:                                               │\n│  ├─ ISOLATES external service failures                        │\n│  │   Never panics — all errors become result<_, swarm-error> │\n│  ├─ RETRIES internally (3x with exponential backoff)         │\n│  │   before returning error to component                      │\n│  ├─ HEALTH CHECK via WasmCloud lifecycle                      │\n│  │   unhealthy provider → WasmCloud restarts it               │\n│  └─ Emits events on state changes                             │\n│                                                               │\n│  virtfs-provider:                                             │\n│  ├─ Git lock contention → retry with backoff                  │\n│  ├─ Worktree create fails → io-error                          │\n│  ├─ Disk full → io-error (checked before worktree creation)  │\n│  └─ Concurrent repo access → mutex per repo                   │\n│                                                               │\n│  quality-gate-provider:                                       │\n│  ├─ Build tool not found → quality-gate-failed                │\n│  ├─ Process timeout (300s) → kill + quality-gate-failed       │\n│  ├─ Non-zero exit → structured check-result with              │\n│  │   stdout/stderr captured (not a swarm-error)               │\n│  └─ OOM during build → quality-gate-failed                    │\n│                                                               │\n│  knowledge-base-provider:                                     │\n│  ├─ SurrealDB down → reconnect loop (5s interval)            │\n│  ├─ After 30s unreachable → emit degraded event               │\n│  │   Components skip knowledge queries, still work            │\n│  └─ Query timeout (10s) → io-error                            │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 4: Orchestration\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  ORCHESTRATION BOUNDARY                                       │\n│                                                               │\n│  orchestrator:                                                │\n│  ├─ PLANNING FAILS (LLM error) →                             │\n│  │   goal-status = failed, event emitted                      │\n│  ├─ AGENT FAILS (task-status = failed) →                      │\n│  │   retry strategy:                                          │\n│  │   1. Retry same model (transient?)        max 1 retry     │\n│  │   2. Retry with larger model              max 1 retry     │\n│  │   3. Mark task failed, continue others                     │\n│  │   4. If >50% tasks fail → fail the goal                   │\n│  ├─ MERGE CONFLICT →                                          │\n│  │   re-run conflicting agent with updated base               │\n│  │   max 2 retries, then fail that task                       │\n│  └─ AGENT TIMEOUT (300s) →                                    │\n│     task unclaimed in SurrealDB after TTL                      │\n│     another agent can pick it up                               │\n│                                                               │\n│  agent-worker:                                                │\n│  ├─ WASM TRAP (panic, OOM) →                                  │\n│  │   WasmCloud catches it, task stays unclaimed                │\n│  │   orchestrator re-dispatches after timeout                  │\n│  ├─ INFERENCE BAD OUTPUT (unparseable) →                       │\n│  │   task-status = failed with error message                   │\n│  │   stored in knowledge-base for future avoidance             │\n│  └─ QUALITY GATE FAILS →                                      │\n│     diff discarded, error stored in knowledge-base             │\n│     task retried with error context fed to next attempt        │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 5: Client\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  CLIENT BOUNDARY                                              │\n│                                                               │\n│  gateway:                                                     │\n│  ├─ All swarm-errors → structured response to client          │\n│  ├─ Event bus down → client gets stale status                 │\n│  │   (poll-based, client handles staleness)                   │\n│  └─ Invalid request → swarm-error::internal                   │\n│                                                               │\n│  CLI:                                                         │\n│  ├─ NATS unreachable →                                        │\n│  │   \"Cannot reach swarm. Is NATS running?\"                   │\n│  ├─ Timeout waiting for goal → show partial status            │\n│  └─ Ctrl-C → goal keeps running (async by design)            │\n│     \"Goal abc123 still running. Check with:                   │\n│      alpha-swarm status abc123\"                                │\n└──────────────────────────────────────────────────────────────┘\n```\n\n## Error Propagation Flow\n\n```\n                    HAPPY PATH\n                    ──────────\nsubmit-goal ──► orchestrator.plan ──► agents[N] ──► merge ──► done\n                                         │\n                    ERROR PATHS           │\n                    ───────────           │\n                                         ▼\n                              ┌─────────────────────┐\n                              │ Agent fails?         │\n                              └──────┬──────────────┘\n                                     │\n                    ┌────────────────┬┴───────────────┐\n                    ▼                ▼                 ▼\n               inference         quality           wasm trap\n               failed            failed            (panic/OOM)\n                    │                │                 │\n                    ▼                ▼                 ▼\n               ┌────────────────────────────────────────┐\n               │ Store error in knowledge base          │\n               │ (embedding for future avoidance)       │\n               └───────────────┬────────────────────────┘\n                               │\n                      ┌────────▼────────┐\n                      │ Retry logic:    │\n                      │ 1. Same model   │\n                      │ 2. Bigger model │\n                      │ 3. Give up task │\n                      └────────┬────────┘\n                               │\n                  ┌────────────┼────────────┐\n                  ▼            ▼            ▼\n               retried      task failed   >50% failed\n               (loop back)  (continue     (fail entire\n                             others)       goal)\n```\n\n## Graceful Degradation Matrix\n\n```\n┌──────────────┬───────────┬──────────┬────────┬──────────────────┐\n│ What's Down  │ Ollama    │ Claude   │SurrealDB│ NATS             │\n│              │           │ API      │         │                  │\n├──────────────┼───────────┼──────────┼─────────┼──────────────────┤\n│ Can plan?    │ YES (1)   │ YES (2)  │ YES     │ NO               │\n│ Can run      │ YES (1)   │ YES (2)  │ YES (3) │ NO               │\n│  agents?     │           │          │         │                  │\n│ Can query    │ YES       │ YES      │ NO      │ YES              │\n│  knowledge?  │           │          │         │                  │\n│ Can emit     │ YES       │ YES      │ YES     │ NO               │\n│  events?     │           │          │         │                  │\n│ Can run      │ YES       │ YES      │ YES     │ NO               │\n│  quality     │           │          │         │                  │\n│  checks?     │           │          │         │                  │\n├──────────────┼───────────┼──────────┼─────────┼──────────────────┤\n│ System state │ DEGRADED  │ DEGRADED │DEGRADED │ PARTITIONED      │\n│              │ fallback  │ fallback │ no      │ in-flight agents │\n│              │ to Claude │ to Ollama│ knowledge│ finish locally   │\n└──────────────┴───────────┴──────────┴─────────┴──────────────────┘\n\n(1) Falls back to Claude API via inference-router\n(2) Falls back to Ollama via inference-router\n(3) Agents work but skip knowledge queries (no past context, no dedup)\n```\n\n## Startup & Shutdown\n\n### Startup Order\n\n```\n1. NATS             ← must be first (lattice backbone)\n   ├─ verify: nats server check\n   │\n2. SurrealDB        ← can start in parallel with step 1\n   ├─ verify: surreal isready\n   │\n3. Ollama           ← can start in parallel with step 1\n   ├─ verify: curl http://localhost:11434/\n   │\n4. WasmCloud Host   ← requires NATS\n   ├─ verify: wash get hosts\n   │\n5. wash app deploy wadm.yaml\n   ├─ verify: wash get inventory\n   └─ verify: wash get links\n```\n\n### Shutdown Order (Reverse)\n\n```\n1. wash app undeploy alpha-swarm     ← drain agents (wait 60s max)\n2. Stop WasmCloud Host\n3. Stop Ollama                       ← or leave running\n4. Stop SurrealDB\n5. Stop NATS                         ← last\n```\n\n### Scripts\n\n```bash\n# Local machine\n./scripts/infra-up-local.sh          # Start NATS + SurrealDB + WasmCloud\n./scripts/infra-down-local.sh        # Graceful shutdown\n\n# csatapaci (runs commands via SSH)\n./scripts/infra-up-csatapaci.sh      # Start NATS + SurrealDB + Ollama + WasmCloud\n./scripts/infra-down-csatapaci.sh    # Graceful shutdown\n\n# Both machines\n./scripts/infra-status.sh            # Check all services + lattice\n```\n\n## Configuration\n\n### Environment Variables\n\n```bash\n# NATS\nALPHA_SWARM_NATS_URL=nats://127.0.0.1:4222\n\n# SurrealDB\nALPHA_SWARM_SURREALDB_URL=ws://127.0.0.1:8000\nALPHA_SWARM_SURREALDB_NS=alpha_swarm\nALPHA_SWARM_SURREALDB_DB=swarm\nALPHA_SWARM_SURREALDB_USER=root\nALPHA_SWARM_SURREALDB_PASS=root\n\n# Ollama\nALPHA_SWARM_OLLAMA_URL=http://csatapaci:11434\nALPHA_SWARM_OLLAMA_TIMEOUT_SECS=120\n\n# Claude API\nALPHA_SWARM_CLAUDE_API_KEY=sk-ant-...\nALPHA_SWARM_CLAUDE_MODEL=claude-sonnet-4-20250514\n\n# Provider behavior\nALPHA_SWARM_RETRY_MAX=3\nALPHA_SWARM_RETRY_BACKOFF_MS=1000\nALPHA_SWARM_CIRCUIT_BREAKER_THRESHOLD=5\nALPHA_SWARM_CIRCUIT_BREAKER_RESET_SECS=30\n\n# Agent behavior\nALPHA_SWARM_TASK_TIMEOUT_SECS=300\nALPHA_SWARM_GOAL_FAIL_THRESHOLD=0.5\n```\n\n### WasmCloud Host Labels\n\n```toml\n# Local machine\n[labels]\nalpha-swarm-role = \"orchestrator\"\nalpha-swarm-gpu = \"false\"\n\n# csatapaci\n[labels]\nalpha-swarm-role = \"inference\"\nalpha-swarm-gpu = \"false\"\nalpha-swarm-ram-gb = \"96\"\nalpha-swarm-models = \"deepseek-coder:33b,codellama:34b,qwen2.5-coder:7b\"\n```\n\n### NATS Cluster\n\n```\nNode 1 (local):     100.79.38.122:4222 (client) / :6222 (cluster)\nNode 2 (csatapaci): 100.81.10.8:4222   (client) / :6222 (cluster)\nCluster name:       alpha-swarm\nJetStream:          enabled on both nodes\n```\n"}, "ca0e1ce9183ff23073ced9519bbaddc7e5094089": {"t": 1785771011.8380315, "n": 22528, "h": "04029a2b3c6b785c", "c": "# alpha-swarm Infrastructure\n\n## System Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│                          CLIENT BOUNDARY                                │\n│                                                                         │\n│   CLI ─────┐                                                            │\n│   Dashboard ┼──► gateway ──► swarm/control ──► project/manager          │\n│   MCP Server┘       │                              │                    │\n│                     │                              │                    │\n│              events/bus ◄──────────────────────────┘                    │\n└──────────────────────┬──────────────────────────────────────────────────┘\n                       │\n┌──────────────────────┼──────────────────────────────────────────────────┐\n│                 ORCHESTRATION BOUNDARY                                   │\n│                      │                                                  │\n│              ┌───────▼────────┐                                         │\n│              │  orchestrator  │                                         │\n│              │  (goal → tasks)│                                         │\n│              └──┬──┬──┬──┬───┘                                         │\n│                 │  │  │  │                                              │\n│    ┌────────────┘  │  │  └────────────┐                                │\n│    ▼               ▼  ▼               ▼                                │\n│ ┌──────┐  ┌──────┐ ┌──────┐  ┌──────┐                                 │\n│ │agent │  │agent │ │agent │  │agent │   (N one-shot workers)           │\n│ │  #1  │  │  #2  │ │  #3  │  │  #N  │                                 │\n│ └──┬───┘  └──┬───┘ └──┬───┘  └──┬───┘                                 │\n└────┼─────────┼────────┼─────────┼──────────────────────────────────────┘\n     │         │        │         │\n┌────┼─────────┼────────┼─────────┼──────────────────────────────────────┐\n│    │    INFERENCE BOUNDARY      │                                      │\n│    │         │        │         │                                      │\n│    └─────────┴────┬───┴─────────┘                                      │\n│                   ▼                                                    │\n│         ┌──────────────────┐                                           │\n│         │ inference-router │   routes by complexity + availability     │\n│         └────┬─────────┬───┘                                           │\n│              │         │                                               │\n│         ┌────▼───┐ ┌───▼────┐                                          │\n│         │ claude │ │ ollama │   ... future: openai, local llama.cpp    │\n│         │ prov.  │ │ prov.  │                                          │\n│         └────┬───┘ └───┬────┘                                          │\n│              │         │                                               │\n└──────────────┼─────────┼──────────────────────────────────────────────┘\n               │         │\n┌──────────────┼─────────┼──────────────────────────────────────────────┐\n│         PROVIDER BOUNDARY    (native Rust, host OS access)             │\n│              │         │                                               │\n│         ┌────▼───┐ ┌───▼────┐ ┌──────────┐ ┌──────────┐              │\n│         │ virtfs │ │quality │ │knowledge │ │ events   │              │\n│         │ prov.  │ │gate p. │ │base prov.│ │ bus prov.│              │\n│         └────┬───┘ └───┬────┘ └────┬─────┘ └────┬─────┘              │\n└──────────────┼─────────┼──────────┼────────────┼──────────────────────┘\n               │         │          │            │\n┌──────────────┼─────────┼──────────┼────────────┼──────────────────────┐\n│         EXTERNAL SERVICE BOUNDARY │            │                       │\n│              ▼         ▼          ▼            ▼                       │\n│         ┌──────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐              │\n│         │ git  │ │ cargo/   │ │ SurrealDB│ │   NATS   │              │\n│         │ repos│ │ npm/go   │ │ :8000    │ │  :4222   │              │\n│         └──────┘ └──────────┘ └──────────┘ └──────────┘              │\n│                                                                        │\n│    ┌──────────┐  ┌──────────┐                                          │\n│    │ Anthropic│  │  Ollama  │                                          │\n│    │ API      │  │  :11434  │                                          │\n│    └──────────┘  └──────────┘                                          │\n└────────────────────────────────────────────────────────────────────────┘\n```\n\n## Physical Deployment (Current Setup)\n\n```\n┌──────────────────────────┐          ┌──────────────────────────┐\n│  LOCAL MACHINE            │          │  CSATAPACI                │\n│                           │          │  M2 Max · 96GB · 3.6TB   │\n│  Tailscale: 100.79.38.122│◄────────►│  Tailscale: 100.81.10.8  │\n│                           │   NATS   │                           │\n│  ┌─────────────────────┐  │  cluster │  ┌─────────────────────┐  │\n│  │ NATS node 1         │  │◄────────►│  │ NATS node 2         │  │\n│  │ :4222 :6222 :8222   │  │          │  │ :4222 :6222 :8222   │  │\n│  └─────────────────────┘  │          │  └─────────────────────┘  │\n│                           │          │                           │\n│  ┌─────────────────────┐  │          │  ┌─────────────────────┐  │\n│  │ SurrealDB :8000     │  │          │  │ Ollama :11434       │  │\n│  │ (primary data store)│  │          │  │ · qwen2.5-coder:7b  │  │\n│  └─────────────────────┘  │          │  │ · deepseek-coder:33b│  │\n│                           │          │  │ · codellama:34b     │  │\n│  ┌─────────────────────┐  │          │  └─────────────────────┘  │\n│  │ WasmCloud Host      │  │          │                           │\n│  │ role: orchestrator   │  │          │  ┌─────────────────────┐  │\n│  │                      │  │          │  │ WasmCloud Host      │  │\n│  │ Components:          │  │          │  │ role: inference      │  │\n│  │  · gateway           │  │          │  │ ram: 96gb            │  │\n│  │  · orchestrator      │  │          │  │                      │  │\n│  │  · inference-router  │  │          │  │ Components:          │  │\n│  │                      │  │          │  │  · agent-worker (N)  │  │\n│  │ Providers:           │  │          │  │                      │  │\n│  │  · claude-provider   │  │          │  │ Providers:           │  │\n│  │  · knowledge-base    │  │          │  │  · ollama-provider   │  │\n│  │  · virtfs            │  │          │  │  · virtfs            │  │\n│  │  · quality-gate      │  │          │  │  · quality-gate      │  │\n│  └─────────────────────┘  │          │  └─────────────────────┘  │\n└──────────────────────────┘          └──────────────────────────┘\n```\n\n## Error Boundaries\n\n### Boundary 1: External Services\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  EXTERNAL SERVICE BOUNDARY                                    │\n│                                                               │\n│  Ollama (:11434)                                              │\n│  ├─ DOWN  → inference-router marks backend unavailable,      │\n│  │          falls back to Claude API                          │\n│  ├─ SLOW  → per-request timeout (configurable, default 120s) │\n│  ├─ MODEL MISSING → swarm-error::not-found                   │\n│  └─ OOM   → inference-failed, router escalates to smaller    │\n│             model or different backend                        │\n│                                                               │\n│  Claude API                                                   │\n│  ├─ 401/403   → swarm-error::inference-failed (bad API key)  │\n│  ├─ 429       → rate limited, backoff + retry                │\n│  ├─ 500/529   → overloaded, fallback to Ollama if available  │\n│  └─ TIMEOUT   → inference-failed                             │\n│                                                               │\n│  SurrealDB (:8000)                                            │\n│  ├─ DOWN → knowledge-base-provider returns io-error           │\n│  │         system enters DEGRADED mode:                       │\n│  │         agents skip knowledge queries, still functional    │\n│  ├─ QUERY TIMEOUT → io-error with context                    │\n│  └─ NAMESPACE MISSING → not-found                            │\n│                                                               │\n│  NATS (:4222)                                                 │\n│  ├─ DOWN → lattice partitions, no new scheduling             │\n│  │         in-flight agents finish locally                     │\n│  ├─ PARTITION → 2-node cluster has no quorum                 │\n│  │   each side operates independently until reconnect         │\n│  └─ RECONNECT → automatic, NATS client handles               │\n│                                                               │\n│  Git repos                                                    │\n│  ├─ LOCK CONTENTION → virtfs retries with backoff            │\n│  ├─ CORRUPT REPO → io-error, project marked as error         │\n│  └─ DISK FULL → io-error, checked before worktree creation   │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 2: Inference Router\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  INFERENCE BOUNDARY                                           │\n│                                                               │\n│  The router is the single point where backend failures are    │\n│  absorbed. Agents never see backend-specific errors.          │\n│                                                               │\n│  Routing strategy:                                            │\n│    simple task  → prefer Ollama (qwen2.5-coder:7b)           │\n│    medium task  → prefer Ollama (deepseek-coder:33b)         │\n│    complex task → prefer Claude API                           │\n│                                                               │\n│  Fallback chain:                                              │\n│    preferred backend fails                                    │\n│         │                                                     │\n│         ▼                                                     │\n│    try next backend with equivalent model                     │\n│         │                                                     │\n│         ▼                                                     │\n│    try any backend with any model                             │\n│         │                                                     │\n│         ▼                                                     │\n│    return swarm-error::inference-failed                       │\n│                                                               │\n│  Circuit breaker per backend:                                 │\n│    5 consecutive failures → open circuit for 30s              │\n│    during open → fast-fail, skip to next backend              │\n│    after 30s → half-open, probe with one request              │\n│    probe succeeds → close circuit, resume normal              │\n│                                                               │\n│  Health events emitted:                                       │\n│    backend.claude.up / backend.claude.down                    │\n│    backend.ollama.up / backend.ollama.down                    │\n│    inference.fallback (when preferred backend unavailable)    │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 3: Providers\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  PROVIDER BOUNDARY                                            │\n│                                                               │\n│  Each provider:                                               │\n│  ├─ ISOLATES external service failures                        │\n│  │   Never panics — all errors become result<_, swarm-error> │\n│  ├─ RETRIES internally (3x with exponential backoff)         │\n│  │   before returning error to component                      │\n│  ├─ HEALTH CHECK via WasmCloud lifecycle                      │\n│  │   unhealthy provider → WasmCloud restarts it               │\n│  └─ Emits events on state changes                             │\n│                                                               │\n│  virtfs-provider:                                             │\n│  ├─ Git lock contention → retry with backoff                  │\n│  ├─ Worktree create fails → io-error                          │\n│  ├─ Disk full → io-error (checked before worktree creation)  │\n│  └─ Concurrent repo access → mutex per repo                   │\n│                                                               │\n│  quality-gate-provider:                                       │\n│  ├─ Build tool not found → quality-gate-failed                │\n│  ├─ Process timeout (300s) → kill + quality-gate-failed       │\n│  ├─ Non-zero exit → structured check-result with              │\n│  │   stdout/stderr captured (not a swarm-error)               │\n│  └─ OOM during build → quality-gate-failed                    │\n│                                                               │\n│  knowledge-base-provider:                                     │\n│  ├─ SurrealDB down → reconnect loop (5s interval)            │\n│  ├─ After 30s unreachable → emit degraded event               │\n│  │   Components skip knowledge queries, still work            │\n│  └─ Query timeout (10s) → io-error                            │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 4: Orchestration\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  ORCHESTRATION BOUNDARY                                       │\n│                                                               │\n│  orchestrator:                                                │\n│  ├─ PLANNING FAILS (LLM error) →                             │\n│  │   goal-status = failed, event emitted                      │\n│  ├─ AGENT FAILS (task-status = failed) →                      │\n│  │   retry strategy:                                          │\n│  │   1. Retry same model (transient?)        max 1 retry     │\n│  │   2. Retry with larger model              max 1 retry     │\n│  │   3. Mark task failed, continue others                     │\n│  │   4. If >50% tasks fail → fail the goal                   │\n│  ├─ MERGE CONFLICT →                                          │\n│  │   re-run conflicting agent with updated base               │\n│  │   max 2 retries, then fail that task                       │\n│  └─ AGENT TIMEOUT (300s) →                                    │\n│     task unclaimed in SurrealDB after TTL                      │\n│     another agent can pick it up                               │\n│                                                               │\n│  agent-worker:                                                │\n│  ├─ WASM TRAP (panic, OOM) →                                  │\n│  │   WasmCloud catches it, task stays unclaimed                │\n│  │   orchestrator re-dispatches after timeout                  │\n│  ├─ INFERENCE BAD OUTPUT (unparseable) →                       │\n│  │   task-status = failed with error message                   │\n│  │   stored in knowledge-base for future avoidance             │\n│  └─ QUALITY GATE FAILS →                                      │\n│     diff discarded, error stored in knowledge-base             │\n│     task retried with error context fed to next attempt        │\n└──────────────────────────────────────────────────────────────┘\n```\n\n### Boundary 5: Client\n\n```\n┌──────────────────────────────────────────────────────────────┐\n│  CLIENT BOUNDARY                                              │\n│                                                               │\n│  gateway:                                                     │\n│  ├─ All swarm-errors → structured response to client          │\n│  ├─ Event bus down → client gets stale status                 │\n│  │   (poll-based, client handles staleness)                   │\n│  └─ Invalid request → swarm-error::internal                   │\n│                                                               │\n│  CLI:                                                         │\n│  ├─ NATS unreachable →                                        │\n│  │   \"Cannot reach swarm. Is NATS running?\"                   │\n│  ├─ Timeout waiting for goal → show partial status            │\n│  └─ Ctrl-C → goal keeps running (async by design)            │\n│     \"Goal abc123 still running. Check with:                   │\n│      alpha-swarm status abc123\"                                │\n└──────────────────────────────────────────────────────────────┘\n```\n\n## Error Propagation Flow\n\n```\n                    HAPPY PATH\n                    ──────────\nsubmit-goal ──► orchestrator.plan ──► agents[N] ──► merge ──► done\n                                         │\n                    ERROR PATHS           │\n                    ───────────           │\n                                         ▼\n                              ┌─────────────────────┐\n                              │ Agent fails?         │\n                              └──────┬──────────────┘\n                                     │\n                    ┌────────────────┬┴───────────────┐\n                    ▼                ▼                 ▼\n               inference         quality           wasm trap\n               failed            failed            (panic/OOM)\n                    │                │                 │\n                    ▼                ▼                 ▼\n               ┌────────────────────────────────────────┐\n               │ Store error in knowledge base          │\n               │ (embedding for future avoidance)       │\n               └───────────────┬────────────────────────┘\n                               │\n                      ┌────────▼────────┐\n                      │ Retry logic:    │\n                      │ 1. Same model   │\n                      │ 2. Bigger model │\n                      │ 3. Give up task │\n                      └────────┬────────┘\n                               │\n                  ┌────────────┼────────────┐\n                  ▼            ▼            ▼\n               retried      task failed   >50% failed\n               (loop back)  (continue     (fail entire\n                             others)       goal)\n```\n\n## Graceful Degradation Matrix\n\n```\n┌──────────────┬───────────┬──────────┬────────┬──────────────────┐\n│ What's Down  │ Ollama    │ Claude   │SurrealDB│ NATS             │\n│              │           │ API      │         │                  │\n├──────────────┼───────────┼──────────┼─────────┼──────────────────┤\n│ Can plan?    │ YES (1)   │ YES (2)  │ YES     │ NO               │\n│ Can run      │ YES (1)   │ YES (2)  │ YES (3) │ NO               │\n│  agents?     │           │          │         │                  │\n│ Can query    │ YES       │ YES      │ NO      │ YES              │\n│  knowledge?  │           │          │         │                  │\n│ Can emit     │ YES       │ YES      │ YES     │ NO               │\n│  events?     │           │          │         │                  │\n│ Can run      │ YES       │ YES      │ YES     │ NO               │\n│  quality     │           │          │         │                  │\n│  checks?     │           │          │         │                  │\n├──────────────┼───────────┼──────────┼─────────┼──────────────────┤\n│ System state │ DEGRADED  │ DEGRADED │DEGRADED │ PARTITIONED      │\n│              │ fallback  │ fallback │ no      │ in-flight agents │\n│              │ to Claude │ to Ollama│ knowledge│ finish locally   │\n└──────────────┴───────────┴──────────┴─────────┴──────────────────┘\n\n(1) Falls back to Claude API via inference-router\n(2) Falls back to Ollama via inference-router\n(3) Agents work but skip knowledge queries (no past context, no dedup)\n```\n\n## Startup & Shutdown\n\n### Startup Order\n\n```\n1. NATS             ← must be first (lattice backbone)\n   ├─ verify: nats server check\n   │\n2. SurrealDB        ← can start in parallel with step 1\n   ├─ verify: surreal isready\n   │\n3. Ollama           ← can start in parallel with step 1\n   ├─ verify: curl http://localhost:11434/\n   │\n4. WasmCloud Host   ← requires NATS\n   ├─ verify: wash get hosts\n   │\n5. wash app deploy wadm.yaml\n   ├─ verify: wash get inventory\n   └─ verify: wash get links\n```\n\n### Shutdown Order (Reverse)\n\n```\n1. wash app undeploy alpha-swarm     ← drain agents (wait 60s max)\n2. Stop WasmCloud Host\n3. Stop Ollama                       ← or leave running\n4. Stop SurrealDB\n5. Stop NATS                         ← last\n```\n\n### Scripts\n\n```bash\n# Local machine\n./scripts/infra-up-local.sh          # Start NATS + SurrealDB + WasmCloud\n./scripts/infra-down-local.sh        # Graceful shutdown\n\n# csatapaci (runs commands via SSH)\n./scripts/infra-up-csatapaci.sh      # Start NATS + SurrealDB + Ollama + WasmCloud\n./scripts/infra-down-csatapaci.sh    # Graceful shutdown\n\n# Both machines\n./scripts/infra-status.sh            # Check all services + lattice\n```\n\n## Configuration\n\n### Environment Variables\n\n```bash\n# NATS\nALPHA_SWARM_NATS_URL=nats://127.0.0.1:4222\n\n# SurrealDB\nALPHA_SWARM_SURREALDB_URL=ws://127.0.0.1:8000\nALPHA_SWARM_SURREALDB_NS=alpha_swarm\nALPHA_SWARM_SURREALDB_DB=swarm\nALPHA_SWARM_SURREALDB_USER=root\nALPHA_SWARM_SURREALDB_PASS=root\n\n# Ollama\nALPHA_SWARM_OLLAMA_URL=http://csatapaci:11434\nALPHA_SWARM_OLLAMA_TIMEOUT_SECS=120\n\n# Claude API\nALPHA_SWARM_CLAUDE_API_KEY=sk-ant-...\nALPHA_SWARM_CLAUDE_MODEL=claude-sonnet-4-20250514\n\n# Provider behavior\nALPHA_SWARM_RETRY_MAX=3\nALPHA_SWARM_RETRY_BACKOFF_MS=1000\nALPHA_SWARM_CIRCUIT_BREAKER_THRESHOLD=5\nALPHA_SWARM_CIRCUIT_BREAKER_RESET_SECS=30\n\n# Agent behavior\nALPHA_SWARM_TASK_TIMEOUT_SECS=300\nALPHA_SWARM_GOAL_FAIL_THRESHOLD=0.5\n```\n\n### WasmCloud Host Labels\n\n```toml\n# Local machine\n[labels]\nalpha-swarm-role = \"orchestrator\"\nalpha-swarm-gpu = \"false\"\n\n# csatapaci\n[labels]\nalpha-swarm-role = \"inference\"\nalpha-swarm-gpu = \"false\"\nalpha-swarm-ram-gb = \"96\"\nalpha-swarm-models = \"deepseek-coder:33b,codellama:34b,qwen2.5-coder:7b\"\n```\n\n### NATS Cluster\n\n```\nNode 1 (local):     100.79.38.122:4222 (client) / :6222 (cluster)\nNode 2 (csatapaci): 100.81.10.8:4222   (client) / :6222 (cluster)\nCluster name:       alpha-swarm\nJetStream:          enabled on both nodes\n```\n"}}}}