hack/tools/scripts/llm-key-hunter/results/_cache/content/anuragid_qualitative-research-tool.json

{"repo": "anuragid/qualitative-research-tool", "files": {"docs/superpowers/plans/2026-04-06-methodex-hobby-scaling-plan.md": {"6760f774b17e4ad5f6e628622f921d5337703fd6": {"t": 1785746532.28104, "n": 123265, "h": "21bddfe0db1cc14e", "c": "# Methodex Hobby-Plan Scaling — Implementation Plan\n\n> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.\n\n**Goal:** Refactor Methodex's Celery pipeline and Railway topology to serve ~100 users / ~25 concurrent analyses peak on the Hobby plan without single-points-of-failure, by chaining the analysis pipeline into per-step tasks, extracting Celery Beat into its own service, scaling backend+worker to 2 replicas each, and trimming idle memory footprint.\n\n**Architecture:** Five workstreams. WS1 (T6 — CI polish), WS2 (T4 — memory + cleanup), and WS3 (T1 — Celery chain refactor) run in **parallel worktrees**. WS4 (T2+T5 — infrastructure + Redis hardening) runs in **Wave 2** after WS1/WS2/WS3 have soaked on single-replica production for ≥24h. WS5 (validation) runs after WS4 lands. WS2 rebases on WS3 before merging because both edit the tasks/ directory.\n\n**Tech Stack:** FastAPI, Celery 5.4 (threads pool), SQLAlchemy 2.0 (sync psycopg2), PostgreSQL (Railway plugin), Redis (Railway plugin), uvicorn, GitHub Actions, Railway GraphQL API, pytest.\n\n**Design spec:** `docs/superpowers/specs/2026-04-06-methodex-hobby-scaling-design.md`\n\n---\n\n## File Map\n\n| Workstream | Modified | Created | Deleted |\n|---|---|---|---|\n| WS1 (T6 CI polish) | `.github/workflows/ci.yml` | `scripts/ci/wait-for-railway-deploy.py` | — |\n| WS2 (T4 memory + cleanup) | `backend/requirements.txt`, `backend/app/services/llm_service.py`, `backend/app/services/s3_service.py`, `backend/app/services/encryption_service.py`, `backend/app/auth.py` | `backend/tests/test_memory_baseline.py` | — |\n| WS3 (T1 chain refactor) | `backend/app/tasks/analysis_steps.py`, `backend/app/tasks/celery_app.py`, `backend/app/routes/videos.py`, `backend/app/routes/projects.py`, `backend/tests/test_analysis_retry.py`, `backend/tests/test_watchdog_race.py` | `backend/app/tasks/project_analysis_steps.py`, `backend/app/tasks/pipeline_errors.py`, `backend/app/tasks/_pipeline_utils.py`, `backend/tests/test_analysis_chain.py`, `backend/tests/test_project_analysis_chain.py`, `backend/tests/test_pipeline_errors.py` | `backend/app/tasks/analysis_tasks.py` |\n| WS4 (T2+T5 infra + Redis) | `backend/scripts/startup.sh`, `backend/app/database.py`, `backend/app/main.py`, Railway service config (via API) | `scripts/railway-service-config.py`, `backend/tests/test_healthcheck_ready.py`, `backend/tests/test_database_pool_config.py` | — |\n| WS5 (validation) | `memory/project_hobby_scaling_progress.md` (update status) | `scripts/production-smoke-test.py`, `scripts/burst-load-test.py` | — |\n\n---\n\n## Dependency & Execution Order\n\n```\nWAVE 1 (parallel worktrees, independent subagents):\n  ┌────────────┐  ┌──────────────┐  ┌────────────────┐\n  │ WS1 (T6)   │  │ WS2 (T4)     │  │ WS3 (T1)       │\n  │ CI polish  │  │ memory+rename│  │ chain refactor │\n  └─────┬──────┘  └──────┬───────┘  └───────┬────────┘\n        │                │                  │\n        │        rebase on WS3 ─────────────┤\n        │                │                  │\n        └────────────────┴──────────────────┘\n                         │\n                  Merge Wave 1 to main\n                         │\n                  Soak ≥24h on single-replica production\n                         │\nWAVE 2:           ┌──────▼───────┐\n                  │ WS4 (T2+T5)  │\n                  │ infra+Redis  │\n                  └──────┬───────┘\n                         │\n                  Merge Wave 2 to main\n                         │\nWAVE 3:           ┌──────▼───────┐\n                  │ WS5          │\n                  │ validation   │\n                  └──────────────┘\n```\n\n**Hard rules:**\n1. WS2 must rebase on WS3 before merging (both touch `backend/app/tasks/` imports).\n2. WS4 must not begin until WS1, WS2, WS3 are merged and have soaked.\n3. No `git push --force` to main ever. Use `git push` or merge via PR.\n4. Every task ends with a commit. Do not batch commits across tasks.\n5. Run `cd backend && pytest tests/ -v` before committing any backend code change; the test suite must stay green throughout.\n\n---\n\n## Task 0: Pre-flight (run once before launching workstreams)\n\n**Files:**\n- Read-only verification\n\n- [ ] **Step 1: Verify we're on main with a clean working tree**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngit status\ngit log --oneline origin/main..HEAD\n```\n\nExpected: current branch `main`, clean working tree, spec commit `5576daf` visible in `git log origin/main..HEAD`.\n\n- [ ] **Step 2: Verify backend test suite is green before any changes**\n\n```bash\ncd backend && pip install -r requirements.txt -r requirements-dev.txt && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: 0 failing tests, 0 errors.\n\nIf any tests fail on clean main before changes, STOP and investigate — do not start the refactor on a broken baseline.\n\n- [ ] **Step 3: Record current per-service memory footprint baseline**\n\nQuery Railway for current memory usage so we can measure improvement post-T4:\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin).get(\"user\",{}).get(\"token\",\"\"))')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ estimatedUsage(workspaceId: \\\"37eb4e96-0873-4033-a392-3c6593a68802\\\", projectId: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\", measurements: [MEMORY_USAGE_GB, CPU_USAGE, NETWORK_TX_GB, DISK_USAGE_GB]) { measurement estimatedValue } }\"}' \\\n  | python3 -m json.tool\n```\n\nSave the output to a scratch file — it will be compared against post-T4 numbers in WS5.\n\n- [ ] **Step 4: Create worktree directories for parallel execution**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngit worktree add -b ci/railway-deploy-wait ../5d-worktrees/ws1-ci origin/main\ngit worktree add -b cleanup/memory-and-llm-naming ../5d-worktrees/ws2-cleanup origin/main\ngit worktree add -b refactor/celery-chain ../5d-worktrees/ws3-chain origin/main\ngit worktree list\n```\n\nExpected: three worktree entries alongside the main checkout. Each subagent that takes on WS1/WS2/WS3 works inside the matching directory.\n\n**Note:** `origin/main` is the base so worktrees don't include local unpushed commits. If `origin/main` is behind and contains bugs the unpushed commits fix, rebase worktrees onto local main instead — but prefer the clean origin/main base when possible.\n\n---\n\n# WORKSTREAM 1 (WS1) — T6 CI Deploy-Wait\n\n**Worktree:** `../5d-worktrees/ws1-ci` on branch `ci/railway-deploy-wait`\n\n**Goal:** GitHub Actions CI reports green only after Railway has successfully deployed the backend for the current commit SHA, not merely after the test suite passes.\n\n**Blocking?** No — runs fully in parallel with WS2 and WS3.\n\n## Task 1.1: Add the Railway deploy-wait script\n\n**Files:**\n- Create: `scripts/ci/wait-for-railway-deploy.py`\n\n- [ ] **Step 1: Create the script**\n\nCreate `scripts/ci/wait-for-railway-deploy.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"Wait for a Railway backend deployment matching a given commit SHA to reach SUCCESS.\n\nUsed in GitHub Actions to block CI from reporting green until the backend is\nactually live. Polls the Railway GraphQL API every 15 seconds for up to 10 minutes.\n\nEnvironment variables:\n- RAILWAY_API_TOKEN: workspace-scoped Railway API token (GitHub secret)\n- RAILWAY_PROJECT_ID: Railway project id (default: methodex project id)\n- RAILWAY_BACKEND_SERVICE_ID: backend service id\n- COMMIT_SHA: the commit sha we're waiting for (usually $GITHUB_SHA)\n\"\"\"\n\nimport os\nimport sys\nimport time\nimport json\nimport urllib.request\nimport urllib.error\n\nRAILWAY_API = \"https://backboard.railway.app/graphql/v2\"\nPOLL_INTERVAL_SECONDS = 15\nMAX_WAIT_SECONDS = 600\n\nQUERY = \"\"\"\nquery DeploymentsForCommit($projectId: String!, $serviceId: String!) {\n  deployments(\n    first: 20\n    input: { projectId: $projectId, serviceId: $serviceId }\n  ) {\n    edges {\n      node {\n        id\n        status\n        meta\n        createdAt\n      }\n    }\n  }\n}\n\"\"\"\n\n\ndef gql(query: str, variables: dict, token: str) -> dict:\n    body = json.dumps({\"query\": query, \"variables\": variables}).encode(\"utf-8\")\n    req = urllib.request.Request(\n        RAILWAY_API,\n        data=body,\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {token}\",\n        },\n        method=\"POST\",\n    )\n    try:\n        with urllib.request.urlopen(req, timeout=30) as resp:\n            return json.loads(resp.read().decode(\"utf-8\"))\n    except urllib.error.HTTPError as e:\n        print(f\"HTTP {e.code}: {e.read().decode('utf-8', errors='replace')}\", file=sys.stderr)\n        raise\n\n\ndef find_deployment_for_sha(token: str, project_id: str, service_id: str, sha: str):\n    \"\"\"Return (deployment_id, status) for the deployment matching the SHA, or (None, None).\"\"\"\n    data = gql(QUERY, {\"projectId\": project_id, \"serviceId\": service_id}, token)\n    if \"errors\" in data:\n        print(f\"GraphQL errors: {data['errors']}\", file=sys.stderr)\n        return None, None\n    edges = (data.get(\"data\") or {}).get(\"deployments\", {}).get(\"edges\", [])\n    for edge in edges:\n        node = edge.get(\"node\", {})\n        meta = node.get(\"meta\") or {}\n        commit = None\n        if isinstance(meta, dict):\n            commit = meta.get(\"commitHash\") or meta.get(\"commit\") or meta.get(\"sha\")\n        if commit and commit.startswith(sha[:7]):\n            return node.get(\"id\"), node.get(\"status\")\n    return None, None\n\n\ndef main():\n    token = os.environ.get(\"RAILWAY_API_TOKEN\")\n    project_id = os.environ.get(\"RAILWAY_PROJECT_ID\", \"154d302f-8609-4897-a10c-1f0d5bfc4f06\")\n    service_id = os.environ.get(\"RAILWAY_BACKEND_SERVICE_ID\", \"2b70a900-042c-4083-b00b-0d01f3ece5dc\")\n    sha = os.environ.get(\"COMMIT_SHA\") or os.environ.get(\"GITHUB_SHA\")\n\n    if not token:\n        print(\"RAILWAY_API_TOKEN env var is required\", file=sys.stderr)\n        return 2\n    if not sha:\n        print(\"COMMIT_SHA (or GITHUB_SHA) env var is required\", file=sys.stderr)\n        return 2\n\n    print(f\"Waiting for Railway backend deployment of {sha[:7]} to reach SUCCESS...\")\n\n    start = time.monotonic()\n    while time.monotonic() - start < MAX_WAIT_SECONDS:\n        dep_id, status = find_deployment_for_sha(token, project_id, service_id, sha)\n        if status is None:\n            print(f\"  [{int(time.monotonic() - start)}s] No deployment found yet for {sha[:7]}\")\n        elif status == \"SUCCESS\":\n            print(f\"  [{int(time.monotonic() - start)}s] Deployment {dep_id} SUCCESS\")\n            return 0\n        elif status in (\"FAILED\", \"CRASHED\", \"REMOVED\"):\n            print(f\"  [{int(time.monotonic() - start)}s] Deployment {dep_id} {status} — aborting\")\n            return 1\n        else:\n            print(f\"  [{int(time.monotonic() - start)}s] Deployment {dep_id} {status}, still waiting\")\n        time.sleep(POLL_INTERVAL_SECONDS)\n\n    print(f\"Timed out after {MAX_WAIT_SECONDS}s waiting for deployment\", file=sys.stderr)\n    return 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make it executable**\n\n```bash\nchmod +x scripts/ci/wait-for-railway-deploy.py\n```\n\n- [ ] **Step 3: Smoke-test the script locally against the existing production deployment**\n\n```bash\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin).get(\"user\",{}).get(\"token\",\"\"))')\nexport COMMIT_SHA=$(git -C /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool rev-parse origin/main)\npython3 scripts/ci/wait-for-railway-deploy.py\n```\n\nExpected: script prints \"Deployment <id> SUCCESS\" within 30 seconds (the existing production deployment for origin/main already succeeded).\n\nIf the script cannot locate a deployment for the SHA, iterate on the `find_deployment_for_sha` query — Railway's `deployments` edge may expose the commit in a different meta key. Adjust the commit-matching logic until the smoke test passes.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add scripts/ci/wait-for-railway-deploy.py\ngit commit -m \"ci: add Railway deploy-wait script\n\nPolls Railway GraphQL for the backend deployment matching the current\ncommit SHA, blocks until SUCCESS (10 min timeout). Used by a new CI job\nso merges to main only report green when the backend is actually live.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 1.2: Add the deploy-wait job to CI\n\n**Files:**\n- Modify: `.github/workflows/ci.yml`\n\n- [ ] **Step 1: Read the current CI file**\n\n```bash\ncat .github/workflows/ci.yml\n```\n\nIdentify the `backend-ci` job and confirm the `deploy-frontend` job shape to follow conventions.\n\n- [ ] **Step 2: Append a new `wait-backend-deploy` job**\n\nEdit `.github/workflows/ci.yml`, adding a new job block at the bottom (after `deploy-frontend`):\n\n```yaml\n  wait-backend-deploy:\n    name: Wait for Railway Backend Deploy\n    needs: backend-ci\n    if: github.ref == 'refs/heads/main' && github.event_name == 'push'\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v5\n\n      - name: Set up Python 3.11\n        uses: actions/setup-python@v6\n        with:\n          python-version: \"3.11\"\n\n      - name: Wait for Railway deployment to reach SUCCESS\n        env:\n          RAILWAY_API_TOKEN: ${{ secrets.RAILWAY_API_TOKEN }}\n          RAILWAY_PROJECT_ID: \"154d302f-8609-4897-a10c-1f0d5bfc4f06\"\n          RAILWAY_BACKEND_SERVICE_ID: \"2b70a900-042c-4083-b00b-0d01f3ece5dc\"\n          COMMIT_SHA: ${{ github.sha }}\n        run: python3 scripts/ci/wait-for-railway-deploy.py\n```\n\n- [ ] **Step 3: Verify the YAML is valid**\n\n```bash\npython3 -c \"import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))\" && echo OK\n```\n\nExpected: `OK`.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add .github/workflows/ci.yml\ngit commit -m \"ci: add wait-backend-deploy job\n\nBlocks CI from reporting green until Railway confirms the backend\ndeployment reached SUCCESS for the pushed commit. Uses the new\nscripts/ci/wait-for-railway-deploy.py helper. Requires a new\nRAILWAY_API_TOKEN secret to be added in GitHub repo settings.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 1.3: Document the new secret requirement\n\n**Files:**\n- Modify: `README.md` (or create one-liner ops doc)\n\n- [ ] **Step 1: Check README for a secrets section**\n\n```bash\ngrep -in \"secret\\|RAILWAY_API_TOKEN\" README.md 2>/dev/null | head -10\n```\n\n- [ ] **Step 2: If a secrets section exists, add the new secret there; otherwise create `docs/ops/ci-secrets.md`**\n\nCreate `docs/ops/ci-secrets.md` (if README has no secrets section):\n\n```markdown\n# CI Secrets\n\nGitHub Actions secrets required by `.github/workflows/ci.yml`:\n\n| Secret | Purpose | How to obtain |\n|---|---|---|\n| `CLOUDFLARE_API_TOKEN` | Deploy frontend to Cloudflare Pages | Cloudflare dashboard → My Profile → API Tokens |\n| `CLOUDFLARE_ACCOUNT_ID` | Cloudflare account selection | Cloudflare dashboard, right sidebar |\n| `VITE_CLERK_PUBLISHABLE_KEY` | Production Clerk key (pk_live_…) | Clerk dashboard → API Keys |\n| `VITE_API_URL` | `https://api.methodex.ai` | Static |\n| `VITE_CLERK_PROXY_URL` | `/__clerk` (reverse-proxy path) | Static |\n| `VITE_SENTRY_DSN` | Frontend Sentry DSN | Sentry project settings |\n| `SENTRY_AUTH_TOKEN` | Source-map upload | Sentry account → auth tokens |\n| `RAILWAY_API_TOKEN` | Railway GraphQL token for deploy-wait job | Railway dashboard → Account → Tokens (workspace-scoped) |\n\nTo rotate a secret, create a new one in the dashboard, update the GitHub\nActions secret, then revoke the old one.\n```\n\n- [ ] **Step 3: Commit**\n\n```bash\ngit add docs/ops/ci-secrets.md\ngit commit -m \"docs: add CI secrets inventory\n\nLists all GitHub Actions secrets consumed by .github/workflows/ci.yml\nand how to obtain them. Adds the new RAILWAY_API_TOKEN entry required\nby the wait-backend-deploy job.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 1.4: WS1 handoff\n\n- [ ] **Step 1: Final verification in the WS1 worktree**\n\n```bash\ngit log --oneline origin/main..HEAD\n```\n\nExpected: 3 commits (add script, add CI job, add docs).\n\n- [ ] **Step 2: Note manual action required**\n\nThe `RAILWAY_API_TOKEN` GitHub secret must be added before this PR merges to main, or the new CI job will fail. Flag this in the PR description:\n\n> **⚠️ Pre-merge action:** Add `RAILWAY_API_TOKEN` to GitHub Actions secrets (workspace-scoped Railway token). Otherwise the new `wait-backend-deploy` job will fail on the first push to main.\n\nWS1 is complete. The branch is ready for PR review and merge.\n\n---\n\n# WORKSTREAM 2 (WS2) — T4 Memory + llm_service Cleanup\n\n**Worktree:** `../5d-worktrees/ws2-cleanup` on branch `cleanup/memory-and-llm-naming`\n\n**Goal:** Remove unused `langgraph` dependency, lazy-import heavy modules (boto3, openai, cryptography), rename `FREE_MODEL_FALLBACKS → STANDARD_MODEL_FALLBACKS`, fix stale docstrings that describe the app as using free-tier OpenRouter models (it does not).\n\n**Blocking?** Must **rebase on WS3** before merging because both WS2 and WS3 touch the `backend/app/tasks/` directory and imports. Start in parallel; rebase at merge time.\n\n## Task 2.1: Verify langgraph is unused before removing it\n\n**Files:**\n- Read-only verification\n\n- [ ] **Step 1: Grep for any remaining langgraph imports**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngrep -rn \"from langgraph\\|import langgraph\" backend/app backend/tests 2>&1\n```\n\nExpected: zero hits. If any code still imports `langgraph`, STOP — fix those imports to use direct node function calls (see `backend/app/tasks/analysis_tasks.py:1-12` docstring for the pattern) BEFORE removing the dependency. File a finding in the commit message and reopen Task 2.1 if needed.\n\n- [ ] **Step 2: Confirm requirements.txt has the line**\n\n```bash\ngrep -n \"^langgraph\" backend/requirements.txt\n```\n\nExpected: one hit, line ~14.\n\n## Task 2.2: Remove langgraph from requirements.txt\n\n**Files:**\n- Modify: `backend/requirements.txt`\n\n- [ ] **Step 1: Remove the langgraph line**\n\nEdit `backend/requirements.txt`, delete the line `langgraph==1.0.10`.\n\n- [ ] **Step 2: Reinstall to verify the app still imports**\n\n```bash\ncd backend && pip uninstall -y langgraph && pip install -r requirements.txt\npython -c \"import app.main; print('OK')\"\n```\n\nExpected: `OK` with no ImportError.\n\n- [ ] **Step 3: Run the full backend test suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -20\n```\n\nExpected: all tests pass (same count as the pre-flight baseline).\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/requirements.txt\ngit commit -m \"chore: drop unused langgraph dependency\n\nThe codebase moved to direct node function calls in 2026-03 (see\nbackend/app/tasks/analysis_tasks.py docstring) but langgraph stayed\nin requirements.txt. Removing it shrinks per-container idle memory\nby ~50 MB per Python process.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.3: Trim sentry-sdk extras\n\n**Files:**\n- Modify: `backend/requirements.txt`\n\n- [ ] **Step 1: Change sentry-sdk extras**\n\nEdit `backend/requirements.txt`, change:\n\n```\nsentry-sdk[fastapi,celery,openai]>=2.41.0\n```\n\nto:\n\n```\nsentry-sdk[fastapi,celery]>=2.41.0\n```\n\n- [ ] **Step 2: Reinstall and verify**\n\n```bash\ncd backend && pip install -r requirements.txt && python -c \"import sentry_sdk; print(sentry_sdk.VERSION)\"\n```\n\nExpected: version string printed, no import errors.\n\n- [ ] **Step 3: Run the test suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -20\n```\n\nExpected: all tests pass. If `test_sentry_config.py` or `test_llm_monitoring.py` tests fail because they reference the removed openai integration, update them to stop asserting on OpenAI Sentry spans — we deliberately removed that integration.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/requirements.txt\ngit commit -m \"chore: remove openai extra from sentry-sdk\n\nThe OpenAI Sentry integration auto-instruments every LLM call with\ndetailed traces, which is noisy for our workload and not what we use\nSentry for. Keeping [fastapi,celery] which we actually need.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.4: Lazy-import boto3 in s3_service.py\n\n**Files:**\n- Modify: `backend/app/services/s3_service.py`\n\n- [ ] **Step 1: Read the file to find the top-level boto3 import**\n\n```bash\ngrep -n \"^import boto3\\|^from boto3\\|^from botocore\" backend/app/services/s3_service.py\n```\n\n- [ ] **Step 2: Move the boto3 import into the function(s) that use it**\n\nEdit `backend/app/services/s3_service.py`. Find every top-level `import boto3` / `from botocore...` statement and move them into the first function or method that actually uses boto3. For example, if the module has:\n\n```python\nimport boto3\nfrom botocore.client import Config\n```\n\nAnd a class `S3Service` with a method `_get_client()` that uses them, move the imports into `_get_client`:\n\n```python\ndef _get_client(self):\n    import boto3\n    from botocore.client import Config\n    return boto3.client(...)\n```\n\nKeep the imports lazy in every function that touches boto3 (Python caches module imports so repeated calls have near-zero overhead).\n\n- [ ] **Step 3: Verify the backend still imports and the tests still pass**\n\n```bash\ncd backend && python -c \"from app.services.s3_service import s3_service; print('OK')\"\npytest tests/ -v 2>&1 | tail -20\n```\n\nExpected: `OK`, all tests pass. If `test_upload_validation.py` or related tests fail, the lazy import is probably inside a branch that the test never exercises — move it earlier in the function or to a helper that's called on init.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/services/s3_service.py\ngit commit -m \"perf: lazy-import boto3 in s3_service\n\nboto3 + botocore loads ~80 MB on import. The backend API process\nnever touches S3 (only the worker does), so deferring the import\nuntil first use sheds that memory from every backend replica.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.5: Lazy-import openai in llm_service.py\n\n**Files:**\n- Modify: `backend/app/services/llm_service.py`\n\n- [ ] **Step 1: Find the top-level openai import**\n\n```bash\ngrep -n \"^from openai\\|^import openai\" backend/app/services/llm_service.py\n```\n\n- [ ] **Step 2: Move it into the client factory**\n\nFind the method (probably `__init__` or `_get_client`) that constructs the OpenAI client. Move the `from openai import OpenAI` (or equivalent) into that method.\n\nExample pattern if the current code is:\n\n```python\nfrom openai import OpenAI\n\nclass LLMService:\n    def __init__(self):\n        self.client = OpenAI(base_url=..., api_key=...)\n```\n\nChange to:\n\n```python\nclass LLMService:\n    def __init__(self):\n        self._client = None\n\n    @property\n    def client(self):\n        if self._client is None:\n            from openai import OpenAI\n            self._client = OpenAI(base_url=..., api_key=...)\n        return self._client\n```\n\nPreserve every existing call site's behavior.\n\n- [ ] **Step 3: Run the llm_service tests**\n\n```bash\ncd backend && pytest tests/test_llm_service.py tests/test_llm_service_retry.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/services/llm_service.py\ngit commit -m \"perf: lazy-import openai in llm_service\n\nDefers the ~25 MB openai SDK import until the first LLM call.\nUses a cached @property so subsequent calls have no overhead.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.6: Lazy-import cryptography in encryption_service.py\n\n**Files:**\n- Modify: `backend/app/services/encryption_service.py`\n\n- [ ] **Step 1: Find the top-level cryptography import**\n\n```bash\ngrep -n \"cryptography\" backend/app/services/encryption_service.py\n```\n\n- [ ] **Step 2: Move `from cryptography.fernet import Fernet` into the functions that use it**\n\nSame pattern as Task 2.5 — defer the import to first use, cache if needed.\n\n- [ ] **Step 3: Run encryption tests**\n\n```bash\ncd backend && pytest tests/test_encryption.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/services/encryption_service.py\ngit commit -m \"perf: lazy-import cryptography in encryption_service\n\nDefers the ~15 MB cryptography module import until the first\nencrypt/decrypt call. Most requests don't touch BYOK keys so\nthis shaves idle memory from backend replicas.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.7: Rename FREE_MODEL_FALLBACKS → STANDARD_MODEL_FALLBACKS\n\n**Files:**\n- Modify: `backend/app/services/llm_service.py`\n\n- [ ] **Step 1: Find every reference to the symbol**\n\n```bash\ngrep -rn \"FREE_MODEL_FALLBACKS\" backend/\n```\n\n- [ ] **Step 2: Rename every occurrence**\n\nIn `backend/app/services/llm_service.py`, rename the constant and all references:\n\n```python\n# Before\nFREE_MODEL_FALLBACKS = [\n    \"meta-llama/llama-4-scout\",\n    \"nvidia/nemotron-3-super-120b-a12b\",\n    \"deepseek/deepseek-chat-v3-0324\",\n]\n\n# After\nSTANDARD_MODEL_FALLBACKS = [\n    \"meta-llama/llama-4-scout\",\n    \"nvidia/nemotron-3-super-120b-a12b\",\n    \"deepseek/deepseek-chat-v3-0324\",\n]\n```\n\nAnd update every reference in the file (grep output from Step 1).\n\n- [ ] **Step 3: Verify no lingering references**\n\n```bash\ngrep -rn \"FREE_MODEL_FALLBACKS\" backend/\n```\n\nExpected: zero hits.\n\n- [ ] **Step 4: Run the relevant tests**\n\n```bash\ncd backend && pytest tests/test_llm_service.py tests/test_llm_service_retry.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/services/llm_service.py\ngit commit -m \"refactor: rename FREE_MODEL_FALLBACKS to STANDARD_MODEL_FALLBACKS\n\nThe list contains paid open-source models on OpenRouter, not free-tier\nmodels. The 'FREE' prefix was a historical artifact from the original\nmigration plan that confused both humans and AI assistants.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.8: Fix stale free-tier docstrings in llm_service.py\n\n**Files:**\n- Modify: `backend/app/services/llm_service.py`\n\n- [ ] **Step 1: Find the stale comments**\n\n```bash\ngrep -n \"free.*[Oo]penRouter\\|[Ff]ree.*model\" backend/app/services/llm_service.py\n```\n\nExpected hits include:\n- Line ~199–200: \"When the primary model is persistently rate-limited (common with free OpenRouter models)\"\n- Line ~296–297: \"Free models may return null content (e.g. finish_reason: length\"\n\n- [ ] **Step 2: Rewrite those comments**\n\nChange:\n\n```python\n# When the primary model is persistently rate-limited (common with free\n# OpenRouter models), this method automatically tries fallback models.\n```\n\nto:\n\n```python\n# When the primary model is persistently rate-limited, this method\n# automatically tries fallback models from STANDARD_MODEL_FALLBACKS.\n```\n\nAnd:\n\n```python\n# Free models may return null content (e.g. finish_reason: length\n# with no output). Fall back to next model when using shared key.\n```\n\nto:\n\n```python\n# Open-source models may return null content (e.g. finish_reason:\n# length with no output). Fall back to next model when using shared key.\n```\n\n- [ ] **Step 3: Verify no \"free\" references remain in the LLM service**\n\n```bash\ngrep -in \"free\" backend/app/services/llm_service.py\n```\n\nExpected: zero hits related to model tier (hits on unrelated English words like \"free memory\" are fine).\n\n- [ ] **Step 4: Run tests**\n\n```bash\ncd backend && pytest tests/test_llm_service.py tests/test_llm_service_retry.py -v\n```\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/services/llm_service.py\ngit commit -m \"docs: correct stale 'free OpenRouter models' comments in llm_service\n\nThe app has never used free-tier OpenRouter models — DEFAULT_MODEL is\nmeta-llama/llama-4-scout (paid) and there is no ':free' suffix anywhere\nin production code paths. The comments were vestigial language from\nthe original migration plan.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.9: WS2 handoff and merge prep\n\n- [ ] **Step 1: Run full backend test suite**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass. Count should match the pre-flight baseline.\n\n- [ ] **Step 2: Rebase on WS3 (chain-refactor) before merging**\n\nWhen WS3 lands on main, rebase WS2 on top:\n\n```bash\ncd ../5d-worktrees/ws2-cleanup\ngit fetch origin\ngit rebase origin/main\n```\n\nResolve any conflicts in `backend/app/tasks/` imports — both branches touch that area. After rebase, re-run `pytest tests/ -v` and fix any test breakage before merging.\n\n- [ ] **Step 3: Verify WS2 commits**\n\n```bash\ngit log --oneline origin/main..HEAD\n```\n\nExpected: 7 commits from this workstream (drop langgraph, trim sentry, lazy-import ×3, rename, docs).\n\nWS2 is complete.\n\n---\n\n# WORKSTREAM 3 (WS3) — T1 Celery Chain Refactor\n\n**Worktree:** `../5d-worktrees/ws3-chain` on branch `refactor/celery-chain`\n\n**Goal:** Convert the monolithic `analyze_video_task` and `analyze_project_task` into Celery `chain`s of per-step tasks. Each step task runs independently, frees its worker thread between steps, and has its own retry lifecycle. Delete the monolithic task file entirely.\n\n**Blocking?** No — runs in parallel with WS1 and WS2. But this must merge BEFORE WS2 so WS2 can rebase on it cleanly.\n\n## Task 3.1: Create shared pipeline utilities module\n\n**Files:**\n- Create: `backend/app/tasks/_pipeline_utils.py`\n- Modify: `backend/app/tasks/celery_app.py` (later)\n\n- [ ] **Step 1: Extract sanitize_error and error-JSON helpers from analysis_tasks.py**\n\nCreate `backend/app/tasks/_pipeline_utils.py`:\n\n```python\n\"\"\"Shared helpers for the analysis pipeline tasks.\n\nExtracted from the pre-refactor analysis_tasks.py so that step tasks\nand the chain error handler can share the same sanitization and\nstructured-error logic.\n\"\"\"\n\nimport json\nimport re\nfrom typing import Optional\n\nfrom app.utils.error_classification import build_structured_error, is_retryable\n\n# Pattern matches common API key formats (OpenRouter sk-or-*, generic long tokens)\n_API_KEY_PATTERN = re.compile(\n    r\"(sk-or-v1-[A-Za-z0-9]{4})[A-Za-z0-9]{20,}\"  # OpenRouter keys\n    r\"|\"\n    r\"(sk-[A-Za-z0-9]{4})[A-Za-z0-9]{20,}\"  # OpenAI-style keys\n    r\"|\"\n    r\"(Bearer\\s+)[A-Za-z0-9_\\-]{20,}\"  # Bearer tokens in error messages\n    r\"|\"\n    r\"([a-f0-9]{4})[a-f0-9]{28,}\"  # AssemblyAI and other hex keys\n)\n\n\ndef sanitize_error(message: str) -> str:\n    \"\"\"Strip potential API key material from error messages before storage.\"\"\"\n    return _API_KEY_PATTERN.sub(\n        lambda m: (m.group(1) or m.group(2) or m.group(3) or m.group(4) or \"\") + \"***REDACTED***\",\n        message,\n    )\n\n\ndef build_error_json(step: str, exc: Exception, message: str) -> str:\n    \"\"\"Build a structured error JSON string from pipeline state info.\"\"\"\n    return json.dumps(build_structured_error(\n        step=step,\n        exc=exc,\n        message=sanitize_error(message),\n    ))\n\n\ndef build_pipeline_error_json(failed_step: str, error_str: str, error_type: Optional[str] = None) -> str:\n    \"\"\"Build a structured error JSON string for a pipeline node failure.\"\"\"\n    etype = error_type or \"unknown\"\n    return json.dumps({\n        \"step\": failed_step,\n        \"error_type\": etype,\n        \"retryable\": is_retryable(etype),\n        \"message\": f\"Analysis failed at step '{failed_step}': {error_str}\",\n        \"details\": error_str,\n    })\n\n\nclass PipelineError(Exception):\n    \"\"\"Internal exception carrying structured error JSON from the pipeline.\"\"\"\n\n    def __init__(self, message: str, structured_json: Optional[str] = None):\n        super().__init__(message)\n        self.structured_json = structured_json\n```\n\n- [ ] **Step 2: Commit**\n\n```bash\ngit add backend/app/tasks/_pipeline_utils.py\ngit commit -m \"refactor: extract pipeline utilities into _pipeline_utils module\n\nExtracts sanitize_error, build_error_json, build_pipeline_error_json,\nand PipelineError from the monolithic analysis_tasks.py so the chain\nrefactor can share them between step tasks and the error handler.\n\nNo functional change — helpers are byte-identical to the originals.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.2: Add cancellation precheck helper for step tasks\n\n**Files:**\n- Modify: `backend/app/tasks/analysis_steps.py`\n\n- [ ] **Step 1: Add a cancellation precheck helper near the top of the file**\n\nIn `backend/app/tasks/analysis_steps.py`, after the existing imports and before `class NonRetryableAnalysisError`, add:\n\n```python\nfrom app.tasks._pipeline_utils import sanitize_error, build_error_json\n\n\ndef _check_cancellation(db: Session, video_id: str) -> bool:\n    \"\"\"Return True if the analysis should stop (watchdog error, row gone).\n\n    Called at the start of every step task in the chain so a halted\n    pipeline doesn't do redundant work on subsequent links.\n    \"\"\"\n    try:\n        db.expire_all()\n        analysis = db.query(VideoAnalysis).filter(\n            VideoAnalysis.video_id == UUID(video_id)\n        ).first()\n        if analysis is None or analysis.status == \"error\":\n            return True\n        return False\n    except Exception:\n        logger.exception(\"_check_cancellation failed, proceeding\")\n        return False\n```\n\n- [ ] **Step 2: Verify tests still pass**\n\n```bash\ncd backend && pytest tests/test_analysis_retry.py tests/test_watchdog_race.py tests/test_analysis_step_non_retryable.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 3: Commit**\n\n```bash\ngit add backend/app/tasks/analysis_steps.py\ngit commit -m \"refactor: add _check_cancellation helper for chain halts\n\nEach step task in the upcoming Celery chain will call this at the top\nto exit early if the watchdog (or a prior failed step) already marked\nthe analysis as error. Replaces the in-function _is_cancelled polling\nfrom the monolithic analyze_video_task.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.3: Wire cancellation precheck into every existing video step task\n\n**Files:**\n- Modify: `backend/app/tasks/analysis_steps.py`\n\n- [ ] **Step 1: Add precheck call to each of the 5 step tasks**\n\nFor each of `analyze_chunk_step`, `analyze_infer_step`, `analyze_relate_step`, `analyze_explain_step`, `analyze_activate_step`, add a cancellation check at the very top of the `try:` block, right after the `logger.info(f\"Starting ... for video {video_id}\")` line.\n\nExample for `analyze_chunk_step`:\n\n```python\ndef analyze_chunk_step(self, video_id: str, user_id: str | None = None):\n    \"\"\"\n    Step 1: CHUNK - Break transcript into discrete pieces.\n    \"\"\"\n    try:\n        logger.info(f\"Starting CHUNK step for video {video_id}\")\n\n        # Cancellation precheck — watchdog or prior halted step\n        if _check_cancellation(self.db, video_id):\n            logger.info(f\"Skipping chunk for {video_id} — already in error state\")\n            return {\"video_id\": video_id, \"status\": \"skipped\"}\n\n        # ... existing code continues ...\n```\n\nRepeat for all 5 step tasks.\n\n- [ ] **Step 2: Write a test that verifies the cancellation precheck returns \"skipped\"**\n\nCreate or update `backend/tests/test_analysis_chain.py`:\n\n```python\n\"\"\"Tests for the Celery chain-based analysis pipeline.\"\"\"\n\nfrom unittest.mock import patch\nfrom uuid import uuid4\n\nimport pytest\nfrom sqlalchemy.orm import Session\n\nfrom app.models.database_models import Project, Transcript, Video, VideoAnalysis\nfrom app.tasks.analysis_steps import analyze_chunk_step, _check_cancellation\n\n\ndef _seed_project_video_transcript(db: Session, user_id: str = \"dev_user_local\"):\n    project = Project(name=\"test\", user_id=user_id)\n    db.add(project)\n    db.flush()\n    video = Video(\n        project_id=project.id,\n        filename=\"v.mp4\",\n        s3_key=f\"videos/{project.id}/{uuid4()}/v.mp4\",\n        s3_url=\"https://example/v.mp4\",\n        file_size_bytes=100,\n        status=\"transcribed\",\n    )\n    db.add(video)\n    db.flush()\n    transcript = Transcript(\n        video_id=video.id,\n        status=\"completed\",\n        processed_transcript={\"duration_seconds\": 60},\n        raw_transcript={\"utterances\": []},\n    )\n    db.add(transcript)\n    db.commit()\n    return project, video\n\n\ndef test_cancellation_precheck_returns_skipped_when_analysis_already_in_error(db_session):\n    \"\"\"If watchdog marked analysis as error, step task should return skipped.\"\"\"\n    _, video = _seed_project_video_transcript(db_session)\n    analysis = VideoAnalysis(\n        video_id=video.id,\n        status=\"error\",  # watchdog already marked it\n        step_status={\"chunk\": \"error\"},\n    )\n    db_session.add(analysis)\n    db_session.commit()\n\n    assert _check_cancellation(db_session, str(video.id)) is True\n\n\ndef test_cancellation_precheck_returns_false_when_analysis_is_processing(db_session):\n    \"\"\"If analysis is actively processing, step task should continue.\"\"\"\n    _, video = _seed_project_video_transcript(db_session)\n    analysis = VideoAnalysis(\n        video_id=video.id,\n        status=\"processing\",\n        step_status={\"chunk\": \"processing\"},\n    )\n    db_session.add(analysis)\n    db_session.commit()\n\n    assert _check_cancellation(db_session, str(video.id)) is False\n```\n\n- [ ] **Step 3: Run the new test**\n\n```bash\ncd backend && pytest tests/test_analysis_chain.py -v\n```\n\nExpected: both tests pass. If `db_session` fixture doesn't exist in `conftest.py`, adapt the test to use whatever fixture the existing suite uses for DB sessions (grep `conftest.py` for `db_session` or similar).\n\n- [ ] **Step 4: Run the full analysis-step test suite**\n\n```bash\npytest tests/test_analysis_retry.py tests/test_analysis_step_non_retryable.py tests/test_watchdog_race.py tests/test_analysis_chain.py -v\n```\n\nExpected: all pass. If `test_watchdog_race.py` breaks because it relied on the old `_is_cancelled` behavior, update it to assert the new `_check_cancellation` flow.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/tasks/analysis_steps.py backend/tests/test_analysis_chain.py\ngit commit -m \"refactor: add cancellation precheck to all video step tasks\n\nEvery chain link now exits early if watchdog (or a prior step) marked\nthe analysis as error. Returns {status: skipped} so downstream chain\nlinks see a clean non-error return and also short-circuit naturally.\n\nAdds test_analysis_chain.py with initial coverage for the precheck.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.4: Move initial-state setup from route into analyze_chunk_step\n\n**Files:**\n- Modify: `backend/app/tasks/analysis_steps.py`\n\n- [ ] **Step 1: Update analyze_chunk_step to create/reset VideoAnalysis if needed**\n\nIn `backend/app/tasks/analysis_steps.py`, update `analyze_chunk_step` so it handles the initial-state transitions that were previously done in the route's `trigger_video_analysis` handler:\n\n```python\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_chunk_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_chunk_step(self, video_id: str, user_id: str | None = None):\n    \"\"\"Step 1: CHUNK — break transcript into discrete pieces.\n\n    Also responsible for chain-start state transitions: create or reset\n    the VideoAnalysis row, mark all steps pending, set status=processing.\n    \"\"\"\n    try:\n        logger.info(f\"Starting CHUNK step for video {video_id}\")\n\n        if _check_cancellation(self.db, video_id):\n            logger.info(f\"Skipping chunk for {video_id} — already in error state\")\n            return {\"video_id\": video_id, \"status\": \"skipped\"}\n\n        state = get_video_analysis_state(self.db, UUID(video_id))\n        analysis = state[\"analysis\"]\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        # Chain-start transitions — previously done in the route\n        analysis.status = \"processing\"\n        analysis.current_step = \"chunk\"\n        analysis.step_status = {\n            \"chunk\": \"processing\",\n            \"infer\": \"pending\",\n            \"relate\": \"pending\",\n            \"explain\": \"pending\",\n            \"activate\": \"pending\",\n        }\n        analysis.started_at = datetime.now(timezone.utc)\n\n        # Mark the video as analyzing (may already be set by the route)\n        video = self.db.query(Video).filter(Video.id == UUID(video_id)).first()\n        if video:\n            video.status = \"analyzing\"\n            video.error_message = None\n        self.db.commit()\n\n        result = chunk_node({\n            \"video_id\": video_id,\n            \"transcript\": state[\"transcript\"],\n            \"speaker_labels\": state[\"speaker_labels\"],\n            \"speaker_roles\": state[\"speaker_roles\"],\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"chunks\") is None:\n            _raise_for_node_error(\"chunk\", result)\n\n        analysis.chunks = result.get(\"chunks\")\n        analysis.chunk_completed_at = datetime.now(timezone.utc)\n        analysis.step_status = {**(analysis.step_status or {}), \"chunk\": \"completed\"}\n        self.db.commit()\n\n        logger.info(f\"CHUNK step completed for video {video_id}\")\n        return {\n            \"video_id\": video_id,\n            \"status\": \"success\",\n            \"chunks_count\": len(result.get(\"chunks\", [])),\n        }\n\n    except Exception as e:\n        logger.error(f\"CHUNK step failed for video {video_id}: {e}\")\n        _update_analysis_error(self.db, video_id, \"chunk\")\n        raise\n```\n\n- [ ] **Step 2: Add a test for the chain-start transitions**\n\nAppend to `backend/tests/test_analysis_chain.py`:\n\n```python\ndef test_chunk_step_initializes_step_status_and_sets_processing(db_session, monkeypatch):\n    \"\"\"analyze_chunk_step should init step_status dict and mark analysis processing.\"\"\"\n    _, video = _seed_project_video_transcript(db_session)\n\n    # Stub the chunk_node so we don't hit the LLM\n    from app.tasks import analysis_steps\n    def fake_chunk_node(state):\n        return {\"chunks\": [{\"id\": \"C001\", \"text\": \"hello\"}], \"current_step\": \"chunk\"}\n    monkeypatch.setattr(analysis_steps, \"chunk_node\", fake_chunk_node)\n\n    # Stub BYOK resolver\n    monkeypatch.setattr(\n        analysis_steps, \"_resolve_byok\",\n        lambda db, user_id: (None, None),\n    )\n\n    # Invoke the task function directly (bypassing Celery's queue)\n    task_self = type(\"T\", (), {\"db\": db_session, \"request\": type(\"R\", (), {\"retries\": 0})()})\n    try:\n        analysis_steps.analyze_chunk_step.run.__wrapped__(task_self, str(video.id), \"dev_user_local\")\n    except AttributeError:\n        # Older Celery versions expose the function differently\n        analysis_steps.analyze_chunk_step(str(video.id), \"dev_user_local\")\n\n    analysis = db_session.query(VideoAnalysis).filter_by(video_id=video.id).first()\n    assert analysis is not None\n    assert analysis.status == \"processing\"\n    assert analysis.step_status[\"chunk\"] == \"completed\"\n    assert analysis.step_status[\"infer\"] == \"pending\"\n    assert analysis.step_status[\"relate\"] == \"pending\"\n    assert analysis.step_status[\"explain\"] == \"pending\"\n    assert analysis.step_status[\"activate\"] == \"pending\"\n```\n\n- [ ] **Step 3: Run the test**\n\n```bash\ncd backend && pytest tests/test_analysis_chain.py -v\n```\n\nExpected: all tests pass. If the task-invocation pattern (`run.__wrapped__`) doesn't match this Celery version, adapt to however the existing tests call step tasks directly — grep `backend/tests/test_analysis_retry.py` for the pattern that works.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/tasks/analysis_steps.py backend/tests/test_analysis_chain.py\ngit commit -m \"refactor: move initial-state setup into analyze_chunk_step\n\nChain-start state transitions (create VideoAnalysis row if missing,\ninit step_status dict, set status=processing, clear error_message)\nmove from the route handler into the chunk step task. The route\nbecomes a pure dispatcher.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.5: Create pipeline_errors.py with the chain error handler\n\n**Files:**\n- Create: `backend/app/tasks/pipeline_errors.py`\n- Create: `backend/tests/test_pipeline_errors.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_pipeline_errors.py`:\n\n```python\n\"\"\"Tests for the chain error handler.\"\"\"\n\nfrom uuid import uuid4\nfrom unittest.mock import MagicMock\n\nimport pytest\n\nfrom app.models.database_models import Project, Video, VideoAnalysis\n\n\ndef _seed_video_in_processing(db_session, user_id=\"dev_user_local\"):\n    project = Project(name=\"test\", user_id=user_id)\n    db_session.add(project)\n    db_session.flush()\n    video = Video(\n        project_id=project.id,\n        filename=\"v.mp4\",\n        s3_key=f\"videos/{project.id}/{uuid4()}/v.mp4\",\n        s3_url=\"https://example/v.mp4\",\n        file_size_bytes=100,\n        status=\"analyzing\",\n    )\n    db_session.add(video)\n    db_session.flush()\n    analysis = VideoAnalysis(\n        video_id=video.id,\n        status=\"processing\",\n        step_status={\"chunk\": \"completed\", \"infer\": \"processing\"},\n    )\n    db_session.add(analysis)\n    db_session.commit()\n    return video, analysis\n\n\ndef test_handle_pipeline_error_marks_video_and_analysis_error(db_session):\n    \"\"\"When a chain link fails, the error handler should mark both records.\"\"\"\n    from app.tasks.pipeline_errors import handle_pipeline_error\n\n    video, analysis = _seed_video_in_processing(db_session)\n    fake_exc = RuntimeError(\"LLM timeout\")\n    fake_request = MagicMock()\n    fake_request.task = \"analyze_infer_step\"\n\n    task_self = type(\"T\", (), {\"db\": db_session})()\n    handle_pipeline_error.run.__func__(\n        task_self, fake_request, fake_exc, \"fake_traceback\", str(video.id)\n    )\n\n    db_session.refresh(video)\n    db_session.refresh(analysis)\n    assert video.status == \"error\"\n    assert video.error_message  # JSON string\n    assert analysis.status == \"error\"\n    assert analysis.completed_at is not None\n\n\ndef test_handle_pipeline_error_is_idempotent(db_session):\n    \"\"\"Running the error handler twice should not change state after the first.\"\"\"\n    from app.tasks.pipeline_errors import handle_pipeline_error\n\n    video, analysis = _seed_video_in_processing(db_session)\n    fake_exc = RuntimeError(\"boom\")\n    fake_request = MagicMock()\n    fake_request.task = \"analyze_infer_step\"\n    task_self = type(\"T\", (), {\"db\": db_session})()\n\n    handle_pipeline_error.run.__func__(task_self, fake_request, fake_exc, \"tb\", str(video.id))\n    first_error_msg = video.error_message\n\n    handle_pipeline_error.run.__func__(task_self, fake_request, fake_exc, \"tb\", str(video.id))\n    db_session.refresh(video)\n    # Should be unchanged on the second call\n    assert video.error_message == first_error_msg\n    assert video.status == \"error\"\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_pipeline_errors.py -v\n```\n\nExpected: FAIL with `ModuleNotFoundError: No module named 'app.tasks.pipeline_errors'`.\n\n- [ ] **Step 3: Create the pipeline_errors.py module**\n\nCreate `backend/app/tasks/pipeline_errors.py`:\n\n```python\n\"\"\"Chain error handler for the Celery analysis pipeline.\n\nWhen a link in the analysis chain fails after retries are exhausted,\nCelery invokes this task via .on_error() with the task request, the\nexception, a traceback string, and our explicit video_id argument.\n\nThe handler is idempotent — if an individual step task's except block\nalready marked the analysis as error (the common case), this handler\nis a no-op.\n\"\"\"\n\nimport logging\nfrom datetime import datetime, timezone\nfrom uuid import UUID\n\nfrom app.models.database_models import Video, VideoAnalysis\nfrom app.tasks._pipeline_utils import build_error_json, sanitize_error\nfrom app.tasks.base import DatabaseTask\nfrom app.tasks.celery_app import celery_app\n\nlogger = logging.getLogger(__name__)\n\n\n@celery_app.task(base=DatabaseTask, bind=True, name=\"handle_pipeline_error\")\ndef handle_pipeline_error(self, request, exc, traceback, video_id: str):\n    \"\"\"Chain error handler — marks video + analysis as error, idempotent.\"\"\"\n    try:\n        self.db.rollback()\n\n        video = self.db.query(Video).filter(Video.id == UUID(video_id)).first()\n        analysis = self.db.query(VideoAnalysis).filter(\n            VideoAnalysis.video_id == UUID(video_id)\n        ).first()\n\n        # Determine which step failed from the request context\n        failed_step = \"unknown\"\n        if request is not None:\n            task_name = getattr(request, \"task\", None)\n            if task_name and isinstance(task_name, str):\n                # e.g. \"analyze_infer_step\" → \"infer\"\n                failed_step = task_name.replace(\"analyze_\", \"\").replace(\"_step\", \"\")\n\n        error_json = build_error_json(\n            step=failed_step,\n            exc=exc if isinstance(exc, Exception) else Exception(str(exc)),\n            message=str(exc),\n        )\n\n        # Idempotent update: only set error state if not already set\n        if video and video.status not in (\"error\", \"analyzed\"):\n            video.status = \"error\"\n            video.error_message = error_json\n            logger.info(f\"handle_pipeline_error: marked video {video_id} as error (step={failed_step})\")\n        elif video:\n            logger.info(f\"handle_pipeline_error: video {video_id} already in {video.status}, no-op\")\n\n        if analysis and analysis.status != \"error\":\n            analysis.status = \"error\"\n            analysis.completed_at = datetime.now(timezone.utc)\n\n        self.db.commit()\n\n    except Exception as e:\n        logger.error(f\"handle_pipeline_error itself failed: {sanitize_error(str(e))}\", exc_info=True)\n        try:\n            self.db.rollback()\n        except Exception:\n            pass\n```\n\n- [ ] **Step 4: Run the test to verify it passes**\n\n```bash\ncd backend && pytest tests/test_pipeline_errors.py -v\n```\n\nExpected: PASS.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/tasks/pipeline_errors.py backend/tests/test_pipeline_errors.py\ngit commit -m \"feat: add handle_pipeline_error chain error handler\n\nNew task invoked by Celery via chain.on_error() when any step task\nfails after retries. Marks the video + VideoAnalysis rows as error\n(idempotent — no-op if already marked). Extracts the failing step\nname from request.task for structured logging.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.6: Create per-step tasks for the 3-node project analysis\n\n**Files:**\n- Create: `backend/app/tasks/project_analysis_steps.py`\n- Create: `backend/tests/test_project_analysis_chain.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_project_analysis_chain.py`:\n\n```python\n\"\"\"Tests for the Celery chain-based project analysis pipeline.\"\"\"\n\nfrom uuid import uuid4\n\nimport pytest\n\nfrom app.models.database_models import (\n    Project,\n    ProjectAnalysis,\n    Video,\n    VideoAnalysis,\n)\n\n\ndef _seed_project_with_completed_videos(db_session, user_id=\"dev_user_local\"):\n    project = Project(name=\"test\", user_id=user_id, description=\"Research goal\")\n    db_session.add(project)\n    db_session.flush()\n    for i in range(2):\n        video = Video(\n            project_id=project.id,\n            filename=f\"v{i}.mp4\",\n            s3_key=f\"videos/{project.id}/{uuid4()}/v{i}.mp4\",\n            s3_url=f\"https://example/v{i}.mp4\",\n            file_size_bytes=100,\n            status=\"analyzed\",\n        )\n        db_session.add(video)\n        db_session.flush()\n        analysis = VideoAnalysis(\n            video_id=video.id,\n            status=\"completed\",\n            patterns=[{\"id\": f\"P{i}\", \"text\": \"pattern\"}],\n            insights=[{\"id\": f\"I{i}\", \"text\": \"insight\"}],\n            design_principles=[{\"id\": f\"DP{i}\", \"text\": \"principle\"}],\n        )\n        db_session.add(analysis)\n    db_session.commit()\n    return project\n\n\ndef test_cross_relate_step_aggregates_patterns_and_runs_node(db_session, monkeypatch):\n    \"\"\"cross_relate step should read completed video analyses and run cross_relate_node.\"\"\"\n    from app.tasks import project_analysis_steps\n\n    project = _seed_project_with_completed_videos(db_session)\n\n    def fake_cross_relate(state):\n        assert len(state[\"video_patterns\"]) == 2\n        return {\"cross_video_patterns\": [{\"id\": \"CP1\", \"text\": \"cross pattern\"}]}\n\n    monkeypatch.setattr(project_analysis_steps, \"cross_relate_node\", fake_cross_relate)\n    monkeypatch.setattr(project_analysis_steps, \"_resolve_byok\", lambda db, user_id: (None, None))\n\n    task_self = type(\"T\", (), {\"db\": db_session, \"request\": type(\"R\", (), {\"retries\": 0})()})\n    project_analysis_steps.analyze_cross_relate_step.run.__func__(\n        task_self, str(project.id), \"dev_user_local\"\n    )\n\n    pa = db_session.query(ProjectAnalysis).filter_by(project_id=project.id).first()\n    assert pa is not None\n    assert pa.cross_video_patterns == [{\"id\": \"CP1\", \"text\": \"cross pattern\"}]\n    assert pa.status == \"processing\"\n\n\ndef test_cross_activate_step_marks_project_analysis_completed(db_session, monkeypatch):\n    \"\"\"cross_activate (terminal step) should mark status completed with completed_at.\"\"\"\n    from app.tasks import project_analysis_steps\n\n    project = _seed_project_with_completed_videos(db_session)\n    pa = ProjectAnalysis(\n        project_id=project.id,\n        status=\"processing\",\n        video_ids=[],\n        cross_video_patterns=[{\"id\": \"CP1\"}],\n        cross_video_insights=[{\"id\": \"CI1\"}],\n    )\n    db_session.add(pa)\n    db_session.commit()\n\n    def fake_cross_activate(state):\n        return {\"cross_video_principles\": [{\"id\": \"CDP1\", \"text\": \"principle\"}]}\n\n    monkeypatch.setattr(project_analysis_steps, \"cross_activate_node\", fake_cross_activate)\n    monkeypatch.setattr(project_analysis_steps, \"_resolve_byok\", lambda db, user_id: (None, None))\n\n    task_self = type(\"T\", (), {\"db\": db_session, \"request\": type(\"R\", (), {\"retries\": 0})()})\n    project_analysis_steps.analyze_cross_activate_step.run.__func__(\n        task_self, str(project.id), \"dev_user_local\"\n    )\n\n    db_session.refresh(pa)\n    assert pa.status == \"completed\"\n    assert pa.completed_at is not None\n    assert pa.cross_video_principles == [{\"id\": \"CDP1\", \"text\": \"principle\"}]\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_project_analysis_chain.py -v\n```\n\nExpected: FAIL with `ModuleNotFoundError: No module named 'app.tasks.project_analysis_steps'`.\n\n- [ ] **Step 3: Create the project_analysis_steps.py module**\n\nCreate `backend/app/tasks/project_analysis_steps.py`:\n\n```python\n\"\"\"Separate Celery tasks for step-by-step cross-video project analysis.\n\nMirrors backend/app/tasks/analysis_steps.py but for the 3-node project\nanalysis pipeline: cross_relate → cross_explain → cross_activate.\nEach step reads state from the ProjectAnalysis row, runs one node,\nand writes results back.\n\"\"\"\n\nimport logging\nfrom datetime import datetime, timezone\nfrom typing import Any, Dict\nfrom uuid import UUID\n\nfrom sqlalchemy.orm import Session\n\nfrom app.agents.nodes.cross_activate import cross_activate_node\nfrom app.agents.nodes.cross_explain import cross_explain_node\nfrom app.agents.nodes.cross_relate import cross_relate_node\nfrom app.models.database_models import Project, ProjectAnalysis, Video, VideoAnalysis\nfrom app.services.byok_service import resolve_byok as _resolve_byok\nfrom app.tasks.analysis_steps import NonRetryableAnalysisError, _raise_for_node_error\nfrom app.tasks.base import DatabaseTask\nfrom app.tasks.celery_app import celery_app\n\nlogger = logging.getLogger(__name__)\n\n\ndef _check_project_cancellation(db: Session, project_id: str) -> bool:\n    \"\"\"Return True if the project analysis should stop.\"\"\"\n    try:\n        db.expire_all()\n        pa = db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if pa is None or pa.status == \"error\":\n            return True\n        return False\n    except Exception:\n        logger.exception(\"_check_project_cancellation failed, proceeding\")\n        return False\n\n\ndef _get_or_create_project_analysis(db: Session, project_id: UUID) -> ProjectAnalysis:\n    \"\"\"Get existing ProjectAnalysis or create a new one with aggregated state.\"\"\"\n    pa = db.query(ProjectAnalysis).filter(ProjectAnalysis.project_id == project_id).first()\n    if pa is None:\n        # Aggregate from completed video analyses\n        video_analyses = db.query(VideoAnalysis).join(Video).filter(\n            Video.project_id == project_id,\n            VideoAnalysis.status == \"completed\",\n        ).all()\n        if not video_analyses:\n            raise Exception(\"At least one completed video analysis is required\")\n\n        video_ids = [va.video_id for va in video_analyses]\n        pa = ProjectAnalysis(\n            project_id=project_id,\n            video_ids=video_ids,\n            status=\"processing\",\n            started_at=datetime.now(timezone.utc),\n        )\n        db.add(pa)\n        db.commit()\n        db.refresh(pa)\n    return pa\n\n\ndef _update_project_analysis_error(db: Session, project_id: str, step_name: str):\n    \"\"\"Mark ProjectAnalysis as error, safe to call on dirty session.\"\"\"\n    try:\n        db.rollback()\n        pa = db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if pa:\n            pa.status = \"error\"\n            pa.completed_at = datetime.now(timezone.utc)\n        db.commit()\n    except Exception as commit_error:\n        logger.error(f\"Failed to update project error status for {step_name}: {commit_error}\")\n        try:\n            db.rollback()\n        except Exception:\n            pass\n\n\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_cross_relate_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_cross_relate_step(self, project_id: str, user_id: str | None = None):\n    \"\"\"Step 1 of 3: find meta-patterns across videos.\"\"\"\n    try:\n        logger.info(f\"Starting CROSS_RELATE step for project {project_id}\")\n\n        if _check_project_cancellation(self.db, project_id):\n            logger.info(f\"Skipping cross_relate for {project_id} — already in error state\")\n            return {\"project_id\": project_id, \"status\": \"skipped\"}\n\n        pa = _get_or_create_project_analysis(self.db, UUID(project_id))\n\n        # Aggregate patterns from completed videos\n        video_analyses = self.db.query(VideoAnalysis).join(Video).filter(\n            Video.project_id == UUID(project_id),\n            VideoAnalysis.status == \"completed\",\n        ).all()\n\n        all_patterns = []\n        all_insights = []\n        all_principles = []\n        video_ids = []\n        for va in video_analyses:\n            video_ids.append(str(va.video_id))\n            if va.patterns:\n                all_patterns.extend(va.patterns)\n            if va.insights:\n                all_insights.extend(va.insights)\n            if va.design_principles:\n                all_principles.extend(va.design_principles)\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        result = cross_relate_node({\n            \"project_id\": project_id,\n            \"video_ids\": video_ids,\n            \"video_patterns\": all_patterns,\n            \"video_insights\": all_insights,\n            \"video_principles\": all_principles,\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"cross_video_patterns\") is None:\n            _raise_for_node_error(\"cross_relate\", result)\n\n        pa.cross_video_patterns = result.get(\"cross_video_patterns\")\n        pa.status = \"processing\"\n        self.db.commit()\n\n        logger.info(f\"CROSS_RELATE step completed for project {project_id}\")\n        return {\"project_id\": project_id, \"status\": \"success\"}\n\n    except Exception as e:\n        logger.error(f\"CROSS_RELATE step failed for project {project_id}: {e}\")\n        _update_project_analysis_error(self.db, project_id, \"cross_relate\")\n        raise\n\n\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_cross_explain_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_cross_explain_step(self, project_id: str, user_id: str | None = None):\n    \"\"\"Step 2 of 3: synthesize cross-video insights from meta-patterns.\"\"\"\n    try:\n        logger.info(f\"Starting CROSS_EXPLAIN step for project {project_id}\")\n\n        if _check_project_cancellation(self.db, project_id):\n            return {\"project_id\": project_id, \"status\": \"skipped\"}\n\n        pa = self.db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if not pa or not pa.cross_video_patterns:\n            raise Exception(\"No cross-video patterns available for cross_explain\")\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        result = cross_explain_node({\n            \"project_id\": project_id,\n            \"cross_video_patterns\": pa.cross_video_patterns,\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"cross_video_insights\") is None:\n            _raise_for_node_error(\"cross_explain\", result)\n\n        pa.cross_video_insights = result.get(\"cross_video_insights\")\n        self.db.commit()\n\n        logger.info(f\"CROSS_EXPLAIN step completed for project {project_id}\")\n        return {\"project_id\": project_id, \"status\": \"success\"}\n\n    except Exception as e:\n        logger.error(f\"CROSS_EXPLAIN step failed for project {project_id}: {e}\")\n        _update_project_analysis_error(self.db, project_id, \"cross_explain\")\n        raise\n\n\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_cross_activate_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_cross_activate_step(self, project_id: str, user_id: str | None = None):\n    \"\"\"Step 3 of 3: derive system-level design principles. Terminal step.\"\"\"\n    try:\n        logger.info(f\"Starting CROSS_ACTIVATE step for project {project_id}\")\n\n        if _check_project_cancellation(self.db, project_id):\n            return {\"project_id\": project_id, \"status\": \"skipped\"}\n\n        pa = self.db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if not pa or not pa.cross_video_insights:\n            raise Exception(\"No cross-video insights available for cross_activate\")\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        result = cross_activate_node({\n            \"project_id\": project_id,\n            \"cross_video_insights\": pa.cross_video_insights,\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"cross_video_principles\") is None:\n            _raise_for_node_error(\"cross_activate\", result)\n\n        pa.cross_video_principles = result.get(\"cross_video_principles\")\n        pa.status = \"completed\"\n        pa.completed_at = datetime.now(timezone.utc)\n        self.db.commit()\n\n        logger.info(f\"CROSS_ACTIVATE step completed for project {project_id}\")\n        return {\"project_id\": project_id, \"status\": \"success\"}\n\n    except Exception as e:\n        logger.error(f\"CROSS_ACTIVATE step failed for project {project_id}: {e}\")\n        _update_project_analysis_error(self.db, project_id, \"cross_activate\")\n        raise\n```\n\n- [ ] **Step 4: Run the test to verify it passes**\n\n```bash\ncd backend && pytest tests/test_project_analysis_chain.py -v\n```\n\nExpected: PASS.\n\n- [ ] **Step 5: Run full suite to make sure nothing regressed**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/app/tasks/project_analysis_steps.py backend/tests/test_project_analysis_chain.py\ngit commit -m \"feat: add per-step project analysis tasks for chain refactor\n\nNew file project_analysis_steps.py mirrors the pattern of\nanalysis_steps.py but for the 3-node cross-video pipeline:\ncross_relate → cross_explain → cross_activate. Each step is an\nindependent Celery task with autoretry, cancellation precheck,\nand DB-backed state transitions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.7: Register new tasks in celery_app.py and add queue routing\n\n**Files:**\n- Modify: `backend/app/tasks/celery_app.py`\n\n- [ ] **Step 1: Update the Celery include list and add task_routes**\n\nEdit `backend/app/tasks/celery_app.py`. Update the `Celery(...)` constructor to include the new task modules, and add `task_routes` + `task_default_queue`:\n\n```python\ncelery_app = Celery(\n    \"qualitative_research_tasks\",\n    broker=settings.CELERY_BROKER_URL,\n    backend=settings.CELERY_RESULT_BACKEND,\n    include=[\n        \"app.tasks.transcription_tasks\",\n        \"app.tasks.analysis_steps\",\n        \"app.tasks.project_analysis_steps\",  # NEW\n        \"app.tasks.pipeline_errors\",          # NEW\n        \"app.tasks.watchdog_tasks\",\n        \"app.tasks.model_validation_tasks\",\n    ],\n)\n\ncelery_app.conf.update(\n    # ... existing config ...\n\n    # Queue routing — analyze tasks on their own queue so they can be\n    # scaled or isolated independently later.\n    task_default_queue=\"celery\",\n    task_routes={\n        \"analyze_chunk_step\":      {\"queue\": \"analyze\"},\n        \"analyze_infer_step\":      {\"queue\": \"analyze\"},\n        \"analyze_relate_step\":     {\"queue\": \"analyze\"},\n        \"analyze_explain_step\":    {\"queue\": \"analyze\"},\n        \"analyze_activate_step\":   {\"queue\": \"analyze\"},\n        \"analyze_cross_relate_step\":   {\"queue\": \"analyze\"},\n        \"analyze_cross_explain_step\":  {\"queue\": \"analyze\"},\n        \"analyze_cross_activate_step\": {\"queue\": \"analyze\"},\n        \"handle_pipeline_error\":       {\"queue\": \"analyze\"},\n        \"transcribe_video\":            {\"queue\": \"transcribe\"},\n        \"check_transcription\":         {\"queue\": \"transcribe\"},\n        # watchdog + model validation use the default \"celery\" queue\n    },\n)\n```\n\n- [ ] **Step 2: Verify Celery can still discover and register tasks**\n\n```bash\ncd backend && python -c \"from app.tasks.celery_app import celery_app; print(sorted(celery_app.tasks.keys()))\" | tr ',' '\\n'\n```\n\nExpected output includes the 5 video step tasks, the 3 cross project step tasks, handle_pipeline_error, transcribe_video, check_transcription, reset_stuck_analyses, validate_openrouter_models.\n\n- [ ] **Step 3: Run the full backend test suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/tasks/celery_app.py\ngit commit -m \"refactor: register new tasks + add queue routing\n\ncelery_app.py now includes project_analysis_steps and pipeline_errors\nmodules. task_routes routes all 8 analysis step tasks and the error\nhandler to the 'analyze' queue, transcription to 'transcribe', and\nwatchdog/model-validation to the default 'celery' queue.\n\nWorker will need to be started with -Q analyze,transcribe,celery\nto consume from all three — that change lands in WS4.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.8: Dispatch chain from the video analyze route\n\n**Files:**\n- Modify: `backend/app/routes/videos.py`\n\n- [ ] **Step 1: Find the existing analyze_video_task.delay dispatch**\n\n```bash\ngrep -n \"analyze_video_task\" backend/app/routes/videos.py\n```\n\n- [ ] **Step 2: Replace with chain dispatch**\n\nIn the `trigger_video_analysis` function (around line 572), replace the monolithic dispatch:\n\n```python\n# OLD:\nfrom app.tasks.analysis_tasks import analyze_video_task\ntask = analyze_video_task.delay(str(video_id), current_user_id)\n```\n\nwith the chain dispatch:\n\n```python\n# NEW:\nfrom celery import chain\nfrom app.tasks.analysis_steps import (\n    analyze_activate_step,\n    analyze_chunk_step,\n    analyze_explain_step,\n    analyze_infer_step,\n    analyze_relate_step,\n)\nfrom app.tasks.pipeline_errors import handle_pipeline_error\n\npipeline = chain(\n    analyze_chunk_step.si(str(video_id), current_user_id),\n    analyze_infer_step.si(str(video_id), current_user_id),\n    analyze_relate_step.si(str(video_id), current_user_id),\n    analyze_explain_step.si(str(video_id), current_user_id),\n    analyze_activate_step.si(str(video_id), current_user_id),\n).on_error(handle_pipeline_error.s(video_id=str(video_id)))\n\ntask = pipeline.apply_async()\n```\n\nKeep the response shape identical (`task_id`, `status`, etc.). Remove the now-unused setup code that was creating the `VideoAnalysis` row or initializing `step_status` — that logic moved into `analyze_chunk_step` in Task 3.4.\n\n- [ ] **Step 3: Run the route tests**\n\n```bash\ncd backend && pytest tests/ -v -k \"analyze or video or route\" 2>&1 | tail -30\n```\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/routes/videos.py\ngit commit -m \"refactor: dispatch Celery chain from video analyze route\n\nPOST /videos/{id}/analyze now dispatches a chain of the 5 per-step\ntasks (chunk → infer → relate → explain → activate) with immutable\nsignatures (.si) so each task receives its own explicit args. The\nerror handler is attached via .on_error().\n\nThe response shape is unchanged so the frontend polls the same\nstatus endpoint and sees the same transitions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.9: Dispatch chain from the project analyze route\n\n**Files:**\n- Modify: `backend/app/routes/projects.py`\n\n- [ ] **Step 1: Find the existing analyze_project_task.delay dispatch**\n\n```bash\ngrep -n \"analyze_project_task\" backend/app/routes/projects.py\n```\n\n- [ ] **Step 2: Replace with chain dispatch**\n\nSame pattern as Task 3.8:\n\n```python\nfrom celery import chain\nfrom app.tasks.project_analysis_steps import (\n    analyze_cross_activate_step,\n    analyze_cross_explain_step,\n    analyze_cross_relate_step,\n)\nfrom app.tasks.pipeline_errors import handle_pipeline_error\n\npipeline = chain(\n    analyze_cross_relate_step.si(str(project_id), current_user_id),\n    analyze_cross_explain_step.si(str(project_id), current_user_id),\n    analyze_cross_activate_step.si(str(project_id), current_user_id),\n).on_error(handle_pipeline_error.s(video_id=str(project_id)))\n# NOTE: the error handler's video_id parameter is a slight misnomer\n# for projects — leave it for now; it'll just tag the error log with\n# the project_id. Rename to record_id or similar as future cleanup.\n\ntask = pipeline.apply_async()\n```\n\n- [ ] **Step 3: Run tests**\n\n```bash\ncd backend && pytest tests/ -v -k \"project or analyze\" 2>&1 | tail -30\n```\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/routes/projects.py\ngit commit -m \"refactor: dispatch Celery chain from project analyze route\n\nPOST /projects/{id}/analyze now dispatches a chain of the 3 per-step\ncross-video tasks (cross_relate → cross_explain → cross_activate).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.10: Delete analysis_tasks.py\n\n**Files:**\n- Delete: `backend/app/tasks/analysis_tasks.py`\n- Modify: any file still importing from it\n\n- [ ] **Step 1: Find remaining imports of analysis_tasks**\n\n```bash\ngrep -rn \"from app.tasks.analysis_tasks\\|from app.tasks import analysis_tasks\" backend/\n```\n\nExpected: should be empty after Tasks 3.8 and 3.9 (routes no longer import it). If any test file still imports from it (likely `test_analysis_retry.py`), update those tests next — see Step 3.\n\n- [ ] **Step 2: Delete the file**\n\n```bash\ngit rm backend/app/tasks/analysis_tasks.py\n```\n\n- [ ] **Step 3: Update any test that still references the deleted module**\n\nFor each failing import from Step 1:\n\n- `test_analysis_retry.py`: if it asserts on `analyze_video_task` retry behavior, rewrite it to assert on `analyze_chunk_step` retry behavior instead (chunk step now owns the retry semantics for the first pipeline link).\n- `test_watchdog_race.py`: replace references to `_is_cancelled` with the new `_check_cancellation` helper.\n\n- [ ] **Step 4: Run the full test suite**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass. If tests still fail because of functional gaps (not mere naming), file a finding in the commit message and adjust — don't paper over with test skips.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add -A\ngit commit -m \"refactor: delete monolithic analysis_tasks.py\n\nThe monolithic analyze_video_task and analyze_project_task are\nreplaced by Celery chains of per-step tasks in analysis_steps.py\nand project_analysis_steps.py. The in-function retry/cancellation\nlogic is replaced by per-task autoretry + _check_cancellation\nprecheck. Helpers moved to _pipeline_utils.py.\n\n-673 / +0 lines net.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.11: Full integration smoke test on WS3\n\n**Files:**\n- Read-only\n\n- [ ] **Step 1: Run the complete test suite one more time**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -40\n```\n\nExpected: all tests pass, with the same count as the pre-flight baseline (minus any tests that were intentionally removed because they tested the deleted monolithic task).\n\n- [ ] **Step 2: Verify celery_app discovers all expected tasks**\n\n```bash\npython -c \"from app.tasks.celery_app import celery_app; import json; print(json.dumps(sorted(celery_app.tasks.keys()), indent=2))\"\n```\n\nExpected tasks in the output:\n- analyze_chunk_step, analyze_infer_step, analyze_relate_step, analyze_explain_step, analyze_activate_step\n- analyze_cross_relate_step, analyze_cross_explain_step, analyze_cross_activate_step\n- handle_pipeline_error\n- transcribe_video, check_transcription\n- reset_stuck_analyses, validate_openrouter_models\n\nAnd NOT in the output: analyze_video, analyze_project (the deleted monolithic tasks).\n\n- [ ] **Step 3: WS3 handoff**\n\n```bash\ngit log --oneline origin/main..HEAD\n```\n\nExpected: ~10 commits (one per task in this workstream).\n\nWS3 is complete. Ready for PR review and merge. **WS2 must rebase on WS3 after it merges.**\n\n---\n\n# WORKSTREAM 4 (WS4) — T2+T5 Infrastructure, Multi-Replica, Redis Hardening\n\n**Worktree:** a fresh worktree created AFTER Wave 1 merges — do not start until WS1/WS2/WS3 are all merged to origin/main and have soaked for ≥24h.\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngit fetch origin\ngit worktree add -b infra/multi-replica-topology ../5d-worktrees/ws4-infra origin/main\ncd ../5d-worktrees/ws4-infra\n```\n\n**Goal:** Extract Celery Beat into its own Railway service. Scale backend and worker to `numReplicas=2`. Add `/health/ready` readiness endpoint. Resize DB pool via environment-aware config. Apply Redis maxmemory + eviction policy. Remove worker's unused public domain.\n\n**Blocking?** Yes — must land after Wave 1 and before WS5 validation.\n\n## Task 4.1: Add service-type-aware DB pool config\n\n**Files:**\n- Modify: `backend/app/database.py`\n- Create: `backend/tests/test_database_pool_config.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_database_pool_config.py`:\n\n```python\n\"\"\"Tests for service-type-aware database pool sizing.\"\"\"\n\nimport importlib\nimport os\n\nimport pytest\n\n\n@pytest.mark.parametrize(\"service_type,expected_pool_size,expected_overflow\", [\n    (\"api\", 5, 5),\n    (\"worker\", 4, 4),\n    (\"beat\", 1, 1),\n])\ndef test_pool_size_by_service_type(service_type, expected_pool_size, expected_overflow, monkeypatch):\n    \"\"\"Pool size should vary by SERVICE_TYPE to avoid Postgres connection exhaustion.\"\"\"\n    monkeypatch.setenv(\"SERVICE_TYPE\", service_type)\n    # Re-import database module so it picks up the new SERVICE_TYPE\n    import app.database\n    importlib.reload(app.database)\n    # The engine's pool carries the size\n    assert app.database.engine.pool.size() == expected_pool_size or \\\n        app.database._pool[\"pool_size\"] == expected_pool_size\n    assert app.database._pool[\"max_overflow\"] == expected_overflow\n\n\ndef test_unknown_service_type_falls_back_to_api_pool(monkeypatch):\n    \"\"\"Unknown SERVICE_TYPE should not crash; fall back to api pool sizing.\"\"\"\n    monkeypatch.setenv(\"SERVICE_TYPE\", \"something-weird\")\n    import app.database\n    importlib.reload(app.database)\n    assert app.database._pool[\"pool_size\"] == 5\n    assert app.database._pool[\"max_overflow\"] == 5\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_database_pool_config.py -v\n```\n\nExpected: FAIL (because `_pool` doesn't exist yet in `database.py`).\n\n- [ ] **Step 3: Update database.py to read SERVICE_TYPE**\n\nEdit `backend/app/database.py`:\n\n```python\n\"\"\"Database connection and session management.\"\"\"\n\nimport os\nfrom typing import Generator\n\nfrom sqlalchemy import create_engine\nfrom sqlalchemy.orm import DeclarativeBase, Session, sessionmaker\n\nfrom app.config import settings\n\n# Build connect_args based on database backend\n_connect_args: dict = {}\nif \"postgresql\" in settings.DATABASE_URL:\n    _connect_args[\"options\"] = \"-c statement_timeout=30000\"  # 30s query timeout\n\n# Service-type-aware pool sizing. Picked so total connection count\n# across all replicas stays comfortably under Postgres max_connections=100.\n#\n# backend: 5 + 5 overflow per uvicorn worker × 2 workers × 2 replicas = 40 max\n# worker:  4 + 4 overflow per replica × 2 replicas = 16 max\n# beat:    1 + 1 overflow × 1 replica = 2 max\n# Total worst case: ~58 connections. Plenty of headroom under 100.\n_SERVICE_TYPE = os.environ.get(\"SERVICE_TYPE\", \"api\")\n_POOL_CONFIG = {\n    \"api\":    {\"pool_size\": 5, \"max_overflow\": 5},\n    \"worker\": {\"pool_size\": 4, \"max_overflow\": 4},\n    \"beat\":   {\"pool_size\": 1, \"max_overflow\": 1},\n}\n_pool = _POOL_CONFIG.get(_SERVICE_TYPE, _POOL_CONFIG[\"api\"])\n\n# Create database engine\nengine = create_engine(\n    settings.DATABASE_URL,\n    connect_args=_connect_args,\n    pool_pre_ping=True,  # Verify connections before using them\n    pool_size=_pool[\"pool_size\"],\n    max_overflow=_pool[\"max_overflow\"],\n    pool_recycle=1800,  # Recycle connections after 30 minutes\n    echo=settings.DEBUG,\n)\n\n# Create session factory\nSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)\n\n\n# Base class for ORM models (modern SQLAlchemy 2.0 style)\nclass Base(DeclarativeBase):\n    pass\n\n\ndef get_db() -> Generator[Session, None, None]:\n    \"\"\"Dependency for FastAPI routes to get database session.\"\"\"\n    db = SessionLocal()\n    try:\n        yield db\n    finally:\n        db.close()\n```\n\n- [ ] **Step 4: Run the test to verify it passes**\n\n```bash\ncd backend && pytest tests/test_database_pool_config.py -v\n```\n\nExpected: all tests pass.\n\n- [ ] **Step 5: Run the full suite to make sure nothing regressed**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass.\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/app/database.py backend/tests/test_database_pool_config.py\ngit commit -m \"feat: service-type-aware DB pool sizing\n\nReads SERVICE_TYPE env var and picks a pool size that keeps total\nconcurrent connections under Postgres max_connections=100 across\nthe new 2-replica backend + 2-replica worker + 1-replica beat\ntopology. Total worst-case usage: ~58 connections.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.2: Add /health/live and /health/ready endpoints\n\n**Files:**\n- Modify: `backend/app/main.py`\n- Create: `backend/tests/test_healthcheck_ready.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_healthcheck_ready.py`:\n\n```python\n\"\"\"Tests for the /health/live and /health/ready endpoints.\"\"\"\n\nfrom unittest.mock import patch\n\nimport pytest\nfrom fastapi.testclient import TestClient\n\n\n@pytest.fixture\ndef client():\n    from app.main import app\n    return TestClient(app)\n\n\ndef test_health_live_returns_200(client):\n    \"\"\"Liveness check is always 200 as long as the process is running.\"\"\"\n    resp = client.get(\"/health/live\")\n    assert resp.status_code == 200\n    assert resp.json() == {\"status\": \"alive\"}\n\n\ndef test_health_returns_200_backwards_compat(client):\n    \"\"\"The old /health endpoint must still work for existing monitoring.\"\"\"\n    resp = client.get(\"/health\")\n    assert resp.status_code == 200\n\n\ndef test_health_ready_returns_200_when_db_and_redis_ok(client):\n    \"\"\"Readiness check returns 200 when both DB and Redis are reachable.\"\"\"\n    resp = client.get(\"/health/ready\")\n    assert resp.status_code == 200\n    assert resp.json() == {\"status\": \"ready\"}\n\n\ndef test_health_ready_returns_503_when_db_fails(client):\n    \"\"\"Readiness check returns 503 when DB ping fails.\"\"\"\n    with patch(\"app.main._probe_db\", side_effect=Exception(\"db down\")):\n        resp = client.get(\"/health/ready\")\n        assert resp.status_code == 503\n        assert resp.json()[\"status\"] == \"db_down\"\n\n\ndef test_health_ready_returns_503_when_redis_fails(client):\n    \"\"\"Readiness check returns 503 when Redis ping fails.\"\"\"\n    with patch(\"app.main._probe_redis\", side_effect=Exception(\"redis down\")):\n        resp = client.get(\"/health/ready\")\n        assert resp.status_code == 503\n        assert resp.json()[\"status\"] == \"redis_down\"\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_healthcheck_ready.py -v\n```\n\nExpected: FAIL (endpoints don't exist yet).\n\n- [ ] **Step 3: Add the endpoints and probe helpers to main.py**\n\nIn `backend/app/main.py`, replace the existing `/health` endpoint with:\n\n```python\nfrom sqlalchemy import text\nfrom fastapi.responses import JSONResponse\n\n\ndef _probe_db() -> None:\n    \"\"\"Raise if the DB is not reachable. Used by /health/ready.\"\"\"\n    from app.database import SessionLocal\n    with SessionLocal() as db:\n        db.execute(text(\"SELECT 1\"))\n\n\ndef _probe_redis() -> None:\n    \"\"\"Raise if Redis (Celery broker) is not reachable. Used by /health/ready.\"\"\"\n    from app.tasks.celery_app import celery_app\n    with celery_app.broker_connection() as conn:\n        conn.ensure_connection(max_retries=1)\n\n\n@app.get(\"/\")\nasync def root():\n    \"\"\"Root endpoint.\"\"\"\n    return {\"status\": \"ok\"}\n\n\n@app.get(\"/health\")\nasync def health_check():\n    \"\"\"Backwards-compatible liveness alias.\"\"\"\n    return {\"status\": \"healthy\"}\n\n\n@app.get(\"/health/live\")\nasync def health_live():\n    \"\"\"Liveness — is the process running?\"\"\"\n    return {\"status\": \"alive\"}\n\n\n@app.get(\"/health/ready\")\nasync def health_ready():\n    \"\"\"Readiness — can this replica serve traffic (DB + Redis OK)?\"\"\"\n    try:\n        _probe_db()\n    except Exception:\n        return JSONResponse(status_code=503, content={\"status\": \"db_down\"})\n    try:\n        _probe_redis()\n    except Exception:\n        return JSONResponse(status_code=503, content={\"status\": \"redis_down\"})\n    return {\"status\": \"ready\"}\n```\n\n- [ ] **Step 4: Run the tests**\n\n```bash\ncd backend && pytest tests/test_healthcheck_ready.py -v\n```\n\nExpected: all tests pass. If the `_probe_redis` mock doesn't fire because the module re-imports it, use `monkeypatch` on the specific `app.main._probe_redis` attribute instead.\n\n- [ ] **Step 5: Run the full suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/app/main.py backend/tests/test_healthcheck_ready.py\ngit commit -m \"feat: add /health/live and /health/ready endpoints\n\nSplits the single /health endpoint into liveness (always 200 as long\nas the process runs) and readiness (verifies DB + Redis connectivity).\nRailway will point healthcheckPath at /health/ready so broken replicas\nget pulled from load balancer rotation automatically.\n\nThe old /health stays as an alias for backwards compat with existing\nmonitoring.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.3: Update startup.sh for beat service + multi-worker uvicorn + worker queues\n\n**Files:**\n- Modify: `backend/scripts/startup.sh`\n\n- [ ] **Step 1: Add beat service branch, --workers to uvicorn, -Q to worker**\n\nEdit `backend/scripts/startup.sh`. Replace the final `if/else` block that starts the service with:\n\n```sh\n# Start the application\nif [ \"$SERVICE\" = \"worker\" ]; then\n    echo \"🔨 Starting Celery worker...\"\n    # --pool=threads: I/O-bound (LLM, AssemblyAI, R2)\n    # --concurrency: CELERY_CONCURRENCY env var (default 16 post-scaling)\n    # -Q analyze,transcribe,celery: consume from all three queues\n    exec celery -A app.tasks.celery_app worker \\\n        --pool=threads \\\n        --concurrency=${CELERY_CONCURRENCY:-16} \\\n        --loglevel=info \\\n        --without-heartbeat \\\n        --without-mingle \\\n        --without-gossip \\\n        -Q analyze,transcribe,celery\nelif [ \"$SERVICE\" = \"beat\" ]; then\n    echo \"⏰ Starting Celery beat scheduler...\"\n    # Beat service runs ONLY the scheduler. Hard-pinned to 1 replica.\n    # Periodic tasks fire exactly once per schedule regardless of\n    # how many worker replicas are running.\n    exec celery -A app.tasks.celery_app beat --loglevel=info\nelse\n    echo \"🌐 Starting API server...\"\n    if [ \"$IS_PRODUCTION\" = true ]; then\n        # Production: 2 uvicorn workers per replica, no reload\n        exec uvicorn app.main:app \\\n            --host 0.0.0.0 \\\n            --port ${PORT:-8000} \\\n            --workers 2 \\\n            --proxy-headers \\\n            --forwarded-allow-ips='*'\n    else\n        # Development: with reload\n        exec uvicorn app.main:app --host 0.0.0.0 --port ${PORT:-8000} --reload\n    fi\nfi\n```\n\nNote that the worker branch no longer has `--beat`. That's intentional — Beat moves to its own dedicated service.\n\n- [ ] **Step 2: Update the Redis-wait block so it also runs for the beat service**\n\nEarlier in the same file, find the block that waits for Redis only when `SERVICE = worker`. Update it to also wait for Redis when `SERVICE = beat`:\n\n```sh\n# Wait for Redis to be ready (required for Celery broker, beat scheduler)\nif [ \"$SERVICE\" = \"worker\" ] || [ \"$SERVICE\" = \"beat\" ]; then\n    echo \"⏳ Waiting for Redis...\"\n    # ... existing Redis-wait code unchanged ...\nfi\n```\n\n- [ ] **Step 3: Shellcheck the script**\n\n```bash\nshellcheck backend/scripts/startup.sh 2>&1 | head -20\n```\n\nIf shellcheck isn't installed, skip — the script is a straightforward bash file.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/scripts/startup.sh\ngit commit -m \"feat: update startup.sh for beat service + multi-worker uvicorn\n\n- New 'beat' service branch runs celery beat scheduler only (no worker)\n- Worker branch loses --beat (moved to dedicated service) and gains\n  -Q analyze,transcribe,celery so it consumes from all queues\n- uvicorn gets --workers 2 in production for within-replica concurrency\n- Redis wait block now also runs for the beat service\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.4: Create Railway service configurator script\n\n**Files:**\n- Create: `scripts/railway-service-config.py`\n\n- [ ] **Step 1: Create the script**\n\nCreate `scripts/railway-service-config.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"Idempotent Railway service configuration for the methodex project.\n\nApplies the target topology (replicas, healthchecks, drain settings) to\neach service via the Railway GraphQL API. Safe to re-run — Railway's\nmutation is idempotent and will no-op on unchanged fields.\n\nAlso creates the new 'beat' service if it doesn't yet exist, and removes\nthe worker's unused public domain.\n\nUsage:\n    export RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\n    python3 scripts/railway-service-config.py --dry-run   # show what would change\n    python3 scripts/railway-service-config.py             # apply\n\nExit codes:\n    0 — success (or dry-run completed)\n    1 — any mutation or validation failure\n    2 — missing env vars\n\"\"\"\n\nimport argparse\nimport json\nimport os\nimport sys\nimport urllib.error\nimport urllib.request\n\nRAILWAY_API = \"https://backboard.railway.app/graphql/v2\"\n\nPROJECT_ID = \"154d302f-8609-4897-a10c-1f0d5bfc4f06\"\nWORKSPACE_ID = \"37eb4e96-0873-4033-a392-3c6593a68802\"\n\n# Target topology — edit here to change replica/health settings\nTARGET = {\n    \"backend\": {\n        \"id\": \"2b70a900-042c-4083-b00b-0d01f3ece5dc\",\n        \"numReplicas\": 2,\n        \"healthcheckPath\": \"/health/ready\",\n        \"healthcheckTimeout\": 10,\n        \"drainingSeconds\": 30,\n    },\n    \"worker\": {\n        \"id\": \"08097b12-1501-4dff-a990-edcd95c73ed4\",\n        \"numReplicas\": 2,\n        \"healthcheckPath\": None,   # workers have no HTTP endpoint\n        \"drainingSeconds\": 60,\n    },\n    \"beat\": {\n        # id populated at runtime if the service already exists,\n        # otherwise created during this script\n        \"id\": None,\n        \"numReplicas\": 1,\n        \"healthcheckPath\": None,\n        \"drainingSeconds\": 10,\n        \"createIfMissing\": True,\n    },\n}\n\n\ndef gql(query: str, variables: dict, token: str) -> dict:\n    body = json.dumps({\"query\": query, \"variables\": variables}).encode(\"utf-8\")\n    req = urllib.request.Request(\n        RAILWAY_API,\n        data=body,\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {token}\",\n        },\n        method=\"POST\",\n    )\n    with urllib.request.urlopen(req, timeout=30) as resp:\n        data = json.loads(resp.read().decode(\"utf-8\"))\n    if \"errors\" in data:\n        raise RuntimeError(f\"GraphQL errors: {data['errors']}\")\n    return data[\"data\"]\n\n\ndef list_services(token: str) -> list[dict]:\n    query = \"\"\"\n    query ListServices($projectId: String!) {\n        project(id: $projectId) {\n            services { edges { node { id name } } }\n        }\n    }\n    \"\"\"\n    data = gql(query, {\"projectId\": PROJECT_ID}, token)\n    return [e[\"node\"] for e in data[\"project\"][\"services\"][\"edges\"]]\n\n\ndef update_service_instance(token: str, service_id: str, env_id: str, updates: dict, dry_run: bool) -> None:\n    \"\"\"Mutate serviceInstanceUpdate with the given updates.\"\"\"\n    if dry_run:\n        print(f\"  [DRY] would update service {service_id}: {updates}\")\n        return\n\n    mutation = \"\"\"\n    mutation UpdateInstance($serviceId: String!, $environmentId: String!, $input: ServiceInstanceUpdateInput!) {\n        serviceInstanceUpdate(serviceId: $serviceId, environmentId: $environmentId, input: $input)\n    }\n    \"\"\"\n    data = gql(mutation, {\n        \"serviceId\": service_id,\n        \"environmentId\": env_id,\n        \"input\": updates,\n    }, token)\n    print(f\"  applied to {service_id}: {list(updates.keys())}\")\n\n\ndef get_production_environment_id(token: str) -> str:\n    \"\"\"Return the production environment id for our project.\"\"\"\n    query = \"\"\"\n    query Envs($projectId: String!) {\n        project(id: $projectId) {\n            environments { edges { node { id name } } }\n        }\n    }\n    \"\"\"\n    data = gql(query, {\"projectId\": PROJECT_ID}, token)\n    for e in data[\"project\"][\"environments\"][\"edges\"]:\n        if e[\"node\"][\"name\"] == \"production\":\n            return e[\"node\"][\"id\"]\n    raise RuntimeError(\"production environment not found\")\n\n\ndef ensure_beat_service_exists(token: str, existing_services: list[dict], env_id: str, dry_run: bool) -> str | None:\n    \"\"\"Create the 'beat' service if it doesn't exist. Return its id.\"\"\"\n    for svc in existing_services:\n        if svc[\"name\"] == \"beat\":\n            print(f\"  beat service already exists: {svc['id']}\")\n            return svc[\"id\"]\n\n    if dry_run:\n        print(\"  [DRY] would create 'beat' service\")\n        return None\n\n    # Create via serviceCreate — clones the worker source (same repo + dockerfile)\n    mutation = \"\"\"\n    mutation CreateBeat($projectId: String!, $name: String!) {\n        serviceCreate(input: { projectId: $projectId, name: $name }) {\n            id name\n        }\n    }\n    \"\"\"\n    data = gql(mutation, {\"projectId\": PROJECT_ID, \"name\": \"beat\"}, token)\n    beat_id = data[\"serviceCreate\"][\"id\"]\n    print(f\"  created beat service: {beat_id}\")\n\n    # Set SERVICE_TYPE=beat and copy source config from worker\n    # NOTE: Railway requires setting the source explicitly after creation.\n    # The simplest route is to have the user configure it from dashboard on\n    # first run, OR use the variableUpsert mutation to set SERVICE_TYPE.\n    variable_mutation = \"\"\"\n    mutation SetVar($input: VariableUpsertInput!) {\n        variableUpsert(input: $input)\n    }\n    \"\"\"\n    gql(variable_mutation, {\n        \"input\": {\n            \"projectId\": PROJECT_ID,\n            \"environmentId\": env_id,\n            \"serviceId\": beat_id,\n            \"name\": \"SERVICE_TYPE\",\n            \"value\": \"beat\",\n        },\n    }, token)\n    print(f\"  set SERVICE_TYPE=beat on {beat_id}\")\n\n    return beat_id\n\n\ndef remove_worker_public_domain(token: str, dry_run: bool) -> None:\n    \"\"\"Delete the worker's pointless public domain.\"\"\"\n    query = \"\"\"\n    query WorkerDomains($serviceId: String!) {\n        service(id: $serviceId) {\n            serviceInstances { edges { node { domains { serviceDomains { id domain } } } } }\n        }\n    }\n    \"\"\"\n    data = gql(query, {\"serviceId\": TARGET[\"worker\"][\"id\"]}, token)\n    try:\n        domains = data[\"service\"][\"serviceInstances\"][\"edges\"][0][\"node\"][\"domains\"][\"serviceDomains\"]\n    except (KeyError, IndexError):\n        domains = []\n    if not domains:\n        print(\"  no worker service domains to remove\")\n        return\n\n    mutation = \"\"\"\n    mutation DeleteDomain($id: String!) {\n        serviceDomainDelete(id: $id)\n    }\n    \"\"\"\n    for d in domains:\n        if dry_run:\n            print(f\"  [DRY] would delete worker service domain {d['domain']} ({d['id']})\")\n        else:\n            gql(mutation, {\"id\": d[\"id\"]}, token)\n            print(f\"  deleted worker service domain {d['domain']}\")\n\n\ndef main():\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"--dry-run\", action=\"store_true\", help=\"Show changes without applying\")\n    args = parser.parse_args()\n\n    token = os.environ.get(\"RAILWAY_API_TOKEN\")\n    if not token:\n        print(\"RAILWAY_API_TOKEN env var required\", file=sys.stderr)\n        return 2\n\n    env_id = get_production_environment_id(token)\n    services = list_services(token)\n    print(f\"Found services: {[s['name'] for s in services]}\")\n\n    # 1. Ensure beat service exists\n    beat_id = ensure_beat_service_exists(token, services, env_id, args.dry_run)\n    if beat_id:\n        TARGET[\"beat\"][\"id\"] = beat_id\n\n    # 2. Apply topology to each known service\n    for name, cfg in TARGET.items():\n        if cfg[\"id\"] is None:\n            print(f\"Skipping {name} — no service id (was it created above?)\")\n            continue\n        updates = {}\n        if \"numReplicas\" in cfg:\n            updates[\"numReplicas\"] = cfg[\"numReplicas\"]\n        if cfg.get(\"healthcheckPath\") is not None:\n            updates[\"healthcheckPath\"] = cfg[\"healthcheckPath\"]\n        if \"healthcheckTimeout\" in cfg:\n            updates[\"healthcheckTimeout\"] = cfg[\"healthcheckTimeout\"]\n        if \"drainingSeconds\" in cfg:\n            updates[\"drainingSeconds\"] = cfg[\"drainingSeconds\"]\n\n        if updates:\n            print(f\"Updating {name} ({cfg['id']}):\")\n            update_service_instance(token, cfg[\"id\"], env_id, updates, args.dry_run)\n\n    # 3. Remove worker's public domain\n    print(\"Removing worker public domain:\")\n    remove_worker_public_domain(token, args.dry_run)\n\n    print(\"\\nDone.\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make executable and run dry-run**\n\n```bash\nchmod +x scripts/railway-service-config.py\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\npython3 scripts/railway-service-config.py --dry-run\n```\n\nExpected: prints the changes that would be applied without mutating anything. Review the output carefully.\n\n- [ ] **Step 3: Commit (do NOT run live yet — that happens in Task 4.7 after all code is merged)**\n\n```bash\ngit add scripts/railway-service-config.py\ngit commit -m \"feat: add Railway service topology configurator script\n\nIdempotent Python script that applies the target topology (replicas,\nhealthchecks, drain settings) to each Railway service via the GraphQL\nAPI. Also creates the 'beat' service if missing and removes the worker\nservice's pointless public domain.\n\nDry-run supported via --dry-run. Do NOT apply live until Wave 2 code\nchanges are merged to main.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.5: Apply Redis maxmemory + eviction policy via Railway\n\n**Files:**\n- No code change; Railway service config update\n\n- [ ] **Step 1: Fetch the current Redis startCommand**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ project(id: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\") { services { edges { node { id name serviceInstances { edges { node { startCommand } } } } } } } }\"}' \\\n  | python3 -m json.tool | grep -A1 Redis\n```\n\nExpected: shows the current Redis startCommand (should include `--requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH`).\n\n- [ ] **Step 2: Update the startCommand to add maxmemory + eviction policy**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\n\n# Get production environment id first\nENV_ID=$(curl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ project(id: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\") { environments { edges { node { id name } } } } }\"}' \\\n  | python3 -c \"import json,sys; d=json.load(sys.stdin); print([e['node']['id'] for e in d['data']['project']['environments']['edges'] if e['node']['name']=='production'][0])\")\n\n# Update Redis startCommand\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d \"{\\\"query\\\":\\\"mutation UpdateRedis(\\$input: ServiceInstanceUpdateInput!) { serviceInstanceUpdate(serviceId: \\\\\\\"53904bf4-fcad-426f-aec6-d94de78c3999\\\\\\\", environmentId: \\\\\\\"$ENV_ID\\\\\\\", input: \\$input) }\\\",\\\"variables\\\":{\\\"input\\\":{\\\"startCommand\\\":\\\"/bin/sh -c 'rm -rf \\$RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass \\$REDIS_PASSWORD --save 60 1 --dir \\$RAILWAY_VOLUME_MOUNT_PATH --maxmemory 300mb --maxmemory-policy allkeys-lru'\\\"}}}\" \\\n  | python3 -m json.tool\n```\n\nExpected: response is `{\"data\": {\"serviceInstanceUpdate\": null}}` indicating success. Railway will redeploy Redis with the new startCommand automatically.\n\n- [ ] **Step 2.5: Defer this step — do not run live during WS4**\n\n**Important**: this step should NOT be executed during the refactor PR. Apply it alongside the topology script in Task 4.7 after the Wave 2 PR merges. The command here is documentation for the operator. Add it to the WS4 PR description as a post-merge action item.\n\n- [ ] **Step 3: Document the change in a runbook**\n\nCreate `docs/ops/redis-config.md`:\n\n```markdown\n# Redis service configuration (methodex)\n\n## Current production startCommand\n\n```\n/bin/sh -c 'rm -rf $RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH --maxmemory 300mb --maxmemory-policy allkeys-lru'\n```\n\n## Why maxmemory 300mb + allkeys-lru\n\n- Celery results expire in 10 minutes (`result_expires=600` in celery_app.py)\n- At target B load, we hold roughly 25 concurrent analysis results × a few KB = <1 MB of working data\n- 300 MB is ~300× headroom, and `allkeys-lru` ensures we never OOM — the oldest Celery result gets evicted first\n- Before this was set, Redis had no memory bound and could OOM-kill the broker silently under a pathological load\n\n## To change the limit\n\nEdit `scripts/railway-service-config.py` if adding to the automated script, or run the `serviceInstanceUpdate` mutation documented in `docs/superpowers/plans/2026-04-06-methodex-hobby-scaling-plan.md` Task 4.5.\n```\n\n- [ ] **Step 4: Commit the runbook**\n\n```bash\ngit add docs/ops/redis-config.md\ngit commit -m \"docs: document Redis startCommand + maxmemory rationale\n\nRunbook for the Redis service configuration. Explains why we pick\n300mb + allkeys-lru given Celery's result_expires=600s setting, and\nhow to change the limit.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.6: Run full local + CI verification\n\n**Files:**\n- Read-only\n\n- [ ] **Step 1: Run the full backend test suite in the WS4 worktree**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass, including the 3 new test files from WS4 (`test_database_pool_config.py`, `test_healthcheck_ready.py`, and any WS3 tests that carried over from the rebase).\n\n- [ ] **Step 2: Verify the startup script is shell-syntax valid**\n\n```bash\nbash -n backend/scripts/startup.sh && echo OK\n```\n\nExpected: `OK`.\n\n- [ ] **Step 3: Dry-run the Railway configurator once more**\n\n```bash\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\npython3 scripts/railway-service-config.py --dry-run\n```\n\nReview output. If the dry-run reports any changes you don't expect, investigate before merging.\n\n## Task 4.7: WS4 merge + live infrastructure apply (carefully staged)\n\n**Files:**\n- Post-merge runbook\n\n- [ ] **Step 1: Merge WS4 to main**\n\nOpen PR, get review, merge. This triggers a backend + worker redeploy with the new code (uvicorn `--workers 2`, `-Q` queue routing, new `/health/ready` endpoint, service-type-aware DB pool). **No replica or topology changes yet** — Railway's service config is not code, so just merging the code doesn't change replicas.\n\n- [ ] **Step 2: Verify merged deploy is healthy**\n\n```bash\ncurl -f https://api.methodex.ai/health/live && echo\ncurl -f https://api.methodex.ai/health/ready && echo\n```\n\nExpected: both return 200. If `/health/ready` returns 503, investigate DB/Redis connectivity before proceeding.\n\n- [ ] **Step 3: Apply the Railway topology changes via script**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\npython3 scripts/railway-service-config.py --dry-run   # final review\npython3 scripts/railway-service-config.py             # apply\n```\n\nExpected: creates the `beat` service, sets `numReplicas=2` on backend + worker, sets `healthcheckPath=/health/ready` on backend, sets drain timings, removes worker public domain.\n\n- [ ] **Step 4: Configure the new beat service source (manual, one-time)**\n\nRailway's GraphQL API can create a service but the source (repo + Dockerfile path) must be set through the dashboard on the first run. In the Railway dashboard:\n\n1. Open the new `beat` service\n2. Set Source → GitHub repo `anuragid/qualitative-research-tool`, branch `main`\n3. Set Build → Dockerfile path `backend/Dockerfile.railway`\n4. Verify env vars include `SERVICE_TYPE=beat` (set by the script) plus all the shared env vars (DATABASE_URL, REDIS_URL, etc.) — copy from the worker service if they're not auto-inherited\n5. Trigger an initial deploy\n\nOnce this first manual setup is done, subsequent config changes flow through the committed script.\n\n- [ ] **Step 5: Apply the Redis startCommand change**\n\nRun the command from Task 4.5 Step 2 (the curl + GraphQL mutation).\n\nVerify:\n\n```bash\n# SSH into Redis (via Railway CLI) or check Railway logs to confirm the\n# container restarted with the new startCommand.\n# Alternatively, if Redis responds to AUTH:\nredis-cli -h yamabiko.proxy.rlwy.net -p 50187 -a $REDIS_PASSWORD INFO memory | grep maxmemory\n```\n\nExpected: `maxmemory:314572800` (300 MB in bytes).\n\n- [ ] **Step 6: Verify the new topology is live**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ project(id: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\") { services { edges { node { id name serviceInstances { edges { node { numReplicas healthcheckPath drainingSeconds } } } } } } } }\"}' \\\n  | python3 -m json.tool\n```\n\nExpected output shows:\n- backend: `numReplicas=2, healthcheckPath=/health/ready, drainingSeconds=30`\n- worker: `numReplicas=2, drainingSeconds=60`\n- beat: `numReplicas=1, drainingSeconds=10`\n- Postgres, Redis: unchanged\n\n- [ ] **Step 7: Verify beat is firing exactly once**\n\n```bash\nrailway service beat && railway logs 2>&1 | head -50\n```\n\nExpected: log entries showing `reset_stuck_analyses` tasks dispatched approximately every 5 minutes (not every 2.5 min — that would indicate multiple beat instances).\n\n- [ ] **Step 8: Verify worker no longer runs beat**\n\n```bash\nrailway service worker && railway logs 2>&1 | head -50\n```\n\nExpected: worker logs show task execution but NO beat-scheduler entries (those now only appear in the beat service's logs).\n\n- [ ] **Step 9: Smoke-test the API end-to-end**\n\n```bash\ncurl -f https://api.methodex.ai/health/ready\ncurl -f https://api.methodex.ai/health/live\n```\n\nExpected: both return 200 from both backend replicas (verify by hitting multiple times and watching for different replica IDs in Railway logs).\n\nWS4 is complete. Wave 2 is live in production.\n\n---\n\n# WORKSTREAM 5 (WS5) — Production Validation\n\n**Worktree:** main checkout is fine — this is scripts + memory updates, no code refactor.\n\n**Goal:** Run smoke and burst load tests, confirm memory footprint dropped, update progress memory, close the loop.\n\n**Blocking?** Runs after WS4 is live.\n\n## Task 5.1: Create production smoke test script\n\n**Files:**\n- Create: `scripts/production-smoke-test.py`\n\n- [ ] **Step 1: Create the script**\n\nCreate `scripts/production-smoke-test.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"End-to-end production smoke test for the methodex analysis pipeline.\n\nRequires:\n- A test user with a pre-seeded project containing at least one video\n  that already has a completed transcript\n- METHODEX_API_URL env var (default: https://api.methodex.ai)\n- METHODEX_AUTH_TOKEN env var (Clerk JWT for the test user)\n- METHODEX_TEST_VIDEO_ID env var (UUID of the fixture video)\n\nUsage:\n    export METHODEX_AUTH_TOKEN=<jwt>\n    export METHODEX_TEST_VIDEO_ID=<uuid>\n    python3 scripts/production-smoke-test.py\n\"\"\"\n\nimport os\nimport sys\nimport time\nimport json\nimport urllib.request\nimport urllib.error\n\nAPI = os.environ.get(\"METHODEX_API_URL\", \"https://api.methodex.ai\")\nTOKEN = os.environ.get(\"METHODEX_AUTH_TOKEN\", \"\")\nVIDEO_ID = os.environ.get(\"METHODEX_TEST_VIDEO_ID\", \"\")\nPOLL_INTERVAL = 5\nTIMEOUT = 300  # 5 minutes\n\n\ndef req(method: str, path: str, body: dict | None = None) -> dict:\n    headers = {\n        \"Authorization\": f\"Bearer {TOKEN}\",\n        \"Content-Type\": \"application/json\",\n    }\n    data = json.dumps(body).encode(\"utf-8\") if body else None\n    r = urllib.request.Request(f\"{API}{path}\", data=data, headers=headers, method=method)\n    with urllib.request.urlopen(r, timeout=30) as resp:\n        return json.loads(resp.read().decode(\"utf-8\"))\n\n\ndef main():\n    if not TOKEN or not VIDEO_ID:\n        print(\"METHODEX_AUTH_TOKEN and METHODEX_TEST_VIDEO_ID env vars required\", file=sys.stderr)\n        return 2\n\n    print(f\"🧪 Smoke test: analyze video {VIDEO_ID} against {API}\")\n\n    # 1. Health checks\n    print(\"\\n[1/4] Health check\")\n    for path in (\"/health/live\", \"/health/ready\"):\n        try:\n            resp = urllib.request.urlopen(f\"{API}{path}\", timeout=10)\n            print(f\"  ✅ {path}: {resp.status}\")\n        except urllib.error.HTTPError as e:\n            print(f\"  ❌ {path}: {e.code}\")\n            return 1\n\n    # 2. Trigger analyze\n    print(\"\\n[2/4] Dispatching analyze...\")\n    dispatch = req(\"POST\", f\"/api/videos/{VIDEO_ID}/analyze\")\n    print(f\"  task_id: {dispatch.get('task_id')}\")\n    print(f\"  analysis_id: {dispatch.get('analysis_id')}\")\n    print(f\"  status: {dispatch.get('status')}\")\n\n    # 3. Poll until complete\n    print(\"\\n[3/4] Polling status (every 5s, timeout 5min)...\")\n    start = time.monotonic()\n    last_step = None\n    while time.monotonic() - start < TIMEOUT:\n        status_resp = req(\"GET\", f\"/api/videos/{VIDEO_ID}/analysis/status\")\n        status = status_resp.get(\"status\")\n        current_step = status_resp.get(\"current_step\")\n        if current_step != last_step:\n            print(f\"  [{int(time.monotonic() - start)}s] step={current_step} status={status}\")\n            last_step = current_step\n        if status == \"completed\":\n            print(\"  ✅ analysis completed\")\n            break\n        if status == \"error\":\n            print(f\"  ❌ analysis failed: {status_resp}\")\n            return 1\n        time.sleep(POLL_INTERVAL)\n    else:\n        print(\"  ❌ timeout\")\n        return 1\n\n    # 4. Verify all 5 step_status entries are \"completed\"\n    print(\"\\n[4/4] Verifying step_status transitions...\")\n    final = req(\"GET\", f\"/api/videos/{VIDEO_ID}/analysis/status\")\n    step_status = final.get(\"step_status\") or {}\n    expected_steps = [\"chunk\", \"infer\", \"relate\", \"explain\", \"activate\"]\n    for step in expected_steps:\n        state = step_status.get(step, \"missing\")\n        marker = \"✅\" if state == \"completed\" else \"❌\"\n        print(f\"  {marker} {step}: {state}\")\n\n    if all(step_status.get(s) == \"completed\" for s in expected_steps):\n        print(\"\\n🎉 SMOKE TEST PASSED\")\n        return 0\n    print(\"\\n💥 SMOKE TEST FAILED — not all steps completed\")\n    return 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make executable**\n\n```bash\nchmod +x scripts/production-smoke-test.py\n```\n\n- [ ] **Step 3: Run against production**\n\nRequires a test user JWT and a test video id. If no fixture video exists, create one manually (upload via the UI, run transcription, let it sit in \"transcribed\" state), save the video id and a long-lived test JWT as env vars:\n\n```bash\nexport METHODEX_AUTH_TOKEN=<jwt from a test user>\nexport METHODEX_TEST_VIDEO_ID=<uuid>\npython3 scripts/production-smoke-test.py\n```\n\nExpected: all 5 step transitions reach `completed`, total time under 5 minutes.\n\nIf the test fails, investigate logs (Railway worker logs + Sentry) before declaring WS5 incomplete. The smoke test is a gate — do not proceed to declare success if it fails.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add scripts/production-smoke-test.py\ngit commit -m \"test: add production smoke test script\n\nEnd-to-end test that hits /analyze on a known fixture video and\nverifies all 5 step_status transitions reach 'completed' within 5\nminutes. Run manually after each major deploy as a go/no-go check.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 5.2: Measure post-refactor memory footprint\n\n**Files:**\n- Read-only\n\n- [ ] **Step 1: Query Railway usage for the new billing period**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ estimatedUsage(workspaceId: \\\"37eb4e96-0873-4033-a392-3c6593a68802\\\", projectId: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\", measurements: [MEMORY_USAGE_GB, CPU_USAGE, NETWORK_TX_GB, DISK_USAGE_GB]) { measurement estimatedValue } }\"}' \\\n  | python3 -m json.tool\n```\n\n- [ ] **Step 2: Compare against the pre-flight baseline captured in Task 0 Step 3**\n\nRecord the numbers in `memory/project_hobby_scaling_progress.md` under a new \"Post-refactor metrics\" section. Example update:\n\n```markdown\n## Post-refactor metrics (captured 2026-04-XX after WS4 landed)\n\n| Measurement | Pre-refactor | Post-refactor | Delta |\n|---|---|---|---|\n| MEMORY_USAGE_GB (monthly) | 18,276 | ??? | ??? |\n| NETWORK_TX_GB (monthly) | 21.3 | ??? | ??? |\n| CPU_USAGE (monthly vCPU-hr) | 299 | ??? | ??? |\n| Estimated monthly cost | $5.45 | ??? | ??? |\n```\n\n- [ ] **Step 3: Commit the memory update (no code)**\n\nThe memory file lives outside the repo (it's in `~/.claude/projects/.../memory/`), so no git commit — just ensure the file is updated.\n\n## Task 5.3: Run burst load test\n\n**Files:**\n- Create: `scripts/burst-load-test.py`\n\n- [ ] **Step 1: Create the burst load test**\n\nCreate `scripts/burst-load-test.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"Burst load test for the methodex analysis pipeline.\n\nFires N concurrent POST /analyze requests against N pre-seeded fixture\nvideos, tracks step transitions over time, and reports:\n- Time-to-first-step-start for each video\n- Total wall-clock for all to reach \"completed\"\n- Queue depth trajectory (inferred from step_status polling)\n- Any failures\n\nRequires a list of fixture video IDs belonging to the test user.\n\nUsage:\n    export METHODEX_AUTH_TOKEN=<jwt>\n    export METHODEX_BURST_VIDEO_IDS=\"uuid1,uuid2,uuid3,...\"\n    python3 scripts/burst-load-test.py\n\"\"\"\n\nimport os\nimport sys\nimport time\nimport json\nimport threading\nimport urllib.request\nimport urllib.error\nfrom concurrent.futures import ThreadPoolExecutor\n\nAPI = os.environ.get(\"METHODEX_API_URL\", \"https://api.methodex.ai\")\nTOKEN = os.environ.get(\"METHODEX_AUTH_TOKEN\", \"\")\nVIDEO_IDS = [v.strip() for v in os.environ.get(\"METHODEX_BURST_VIDEO_IDS\", \"\").split(\",\") if v.strip()]\nPOLL_INTERVAL = 5\nTIMEOUT = 600\n\n\ndef req(method: str, path: str) -> dict:\n    headers = {\"Authorization\": f\"Bearer {TOKEN}\", \"Content-Type\": \"application/json\"}\n    r = urllib.request.Request(f\"{API}{path}\", headers=headers, method=method)\n    with urllib.request.urlopen(r, timeout=30) as resp:\n        return json.loads(resp.read().decode(\"utf-8\"))\n\n\ndef dispatch_one(video_id: str) -> tuple[str, float]:\n    t0 = time.monotonic()\n    req(\"POST\", f\"/api/videos/{video_id}/analyze\")\n    return video_id, time.monotonic() - t0\n\n\ndef poll_one(video_id: str, start_time: float) -> dict:\n    history = []\n    while time.monotonic() - start_time < TIMEOUT:\n        try:\n            status = req(\"GET\", f\"/api/videos/{video_id}/analysis/status\")\n        except Exception as e:\n            history.append({\"t\": time.monotonic() - start_time, \"error\": str(e)})\n            time.sleep(POLL_INTERVAL)\n            continue\n        t = time.monotonic() - start_time\n        history.append({\"t\": t, \"step_status\": status.get(\"step_status\"), \"status\": status.get(\"status\")})\n        if status.get(\"status\") in (\"completed\", \"error\"):\n            return {\"video_id\": video_id, \"final\": status, \"history\": history}\n        time.sleep(POLL_INTERVAL)\n    return {\"video_id\": video_id, \"final\": {\"status\": \"timeout\"}, \"history\": history}\n\n\ndef main():\n    if not TOKEN or not VIDEO_IDS:\n        print(\"METHODEX_AUTH_TOKEN and METHODEX_BURST_VIDEO_IDS env vars required\", file=sys.stderr)\n        return 2\n\n    print(f\"🔥 Burst load test: {len(VIDEO_IDS)} concurrent analyses against {API}\")\n\n    # Dispatch all analyses as fast as possible\n    dispatch_start = time.monotonic()\n    with ThreadPoolExecutor(max_workers=len(VIDEO_IDS)) as executor:\n        dispatch_results = list(executor.map(dispatch_one, VIDEO_IDS))\n    dispatch_elapsed = time.monotonic() - dispatch_start\n    print(f\"\\nDispatched {len(VIDEO_IDS)} analyses in {dispatch_elapsed:.1f}s\")\n    for vid, dt in dispatch_results:\n        print(f\"  {vid}: dispatch took {dt * 1000:.0f}ms\")\n\n    # Poll all in parallel until terminal\n    poll_start = time.monotonic()\n    with ThreadPoolExecutor(max_workers=len(VIDEO_IDS)) as executor:\n        poll_results = list(executor.map(lambda v: poll_one(v, poll_start), VIDEO_IDS))\n\n    total_elapsed = time.monotonic() - dispatch_start\n    print(f\"\\n⏱  Total wall-clock for burst: {total_elapsed:.1f}s\")\n\n    # Summary\n    completed = sum(1 for r in poll_results if r[\"final\"].get(\"status\") == \"completed\")\n    errored = sum(1 for r in poll_results if r[\"final\"].get(\"status\") == \"error\")\n    timeout = sum(1 for r in poll_results if r[\"final\"].get(\"status\") == \"timeout\")\n    print(f\"  completed: {completed} / {len(VIDEO_IDS)}\")\n    print(f\"  errored:   {errored}\")\n    print(f\"  timeout:   {timeout}\")\n\n    if completed == len(VIDEO_IDS):\n        print(\"\\n🎉 BURST LOAD TEST PASSED\")\n        return 0\n    print(\"\\n💥 BURST LOAD TEST FAILED\")\n    return 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make executable**\n\n```bash\nchmod +x scripts/burst-load-test.py\n```\n\n- [ ] **Step 3: Run against production with ~10 fixture videos first**\n\nStart small — 10 concurrent analyses — before scaling to the target 25.\n\n```bash\nexport METHODEX_AUTH_TOKEN=<jwt>\nexport METHODEX_BURST_VIDEO_IDS=\"<10 comma-separated uuids>\"\npython3 scripts/burst-load-test.py\n```\n\nReview the output. All 10 should complete within ~3 minutes. If any fail or time out, investigate before scaling.\n\n- [ ] **Step 4: Run again with 25 concurrent**\n\n```bash\nexport METHODEX_BURST_VIDEO_IDS=\"<25 comma-separated uuids>\"\npython3 scripts/burst-load-test.py\n```\n\nExpected: all 25 complete within ~4 minutes, no timeouts, no errors.\n\nIf any step latency looks wrong or tasks pile up in queue, inspect Railway logs + Sentry to find the bottleneck.\n\n- [ ] **Step 5: Commit the burst load test**\n\n```bash\ngit add scripts/burst-load-test.py\ngit commit -m \"test: add burst load test script for multi-replica worker validation\n\nFires N concurrent POST /analyze calls and tracks per-video step\ntransitions until all reach terminal state. Used to validate the\npost-T1/T2 pipeline at target-B scale (25 concurrent).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 5.4: Update progress memory to mark project complete\n\n**Files:**\n- Modify: `~/.claude/projects/-Users-idstuart-Projects-ai-prototyping-5d-analysis/memory/project_hobby_scaling_progress.md`\n\n- [ ] **Step 1: Update the memory file status**\n\nEdit the memory file, change the opening from \"IN-FLIGHT\" to \"COMPLETED 2026-04-XX\" and add a final summary section:\n\n```markdown\n## Completion summary (2026-04-XX)\n\n- **Wave 1** landed: WS1 (CI deploy-wait), WS2 (memory + cleanup), WS3 (chain refactor)\n- **Wave 2** landed: WS4 (beat extraction, multi-replica topology, Redis hardening)\n- **Wave 3** validated: smoke test passed, burst load test (25 concurrent) passed\n- **Memory footprint change**: pre-refactor $X.XX/mo → post-refactor $Y.YY/mo\n- **Task duration change**: monolithic 150s/video → chain mean 30s/step, 145s total (similar wall-clock, no thread pinning)\n- **Burst capacity change**: blocked at 8 concurrent → scales to 25+ without queueing\n\nAll goals from the design spec met. The stack remains on Railway Hobby.\n```\n\nSave. No git commit (memory file is outside the repo).\n\n## Task 5.5: Final handoff\n\n- [ ] **Step 1: Write a short handoff note in the repo**\n\nCreate `docs/ops/post-scaling-operational-notes.md`:\n\n```markdown\n# Post-scaling operational notes (2026-04-XX)\n\n## Architecture changes that went live\n\n- `backend` service now runs with 2 replicas, each with 2 uvicorn workers\n- `worker` service now runs with 2 replicas, each with Celery concurrency 16 (threads pool, consumes from queues: analyze, transcribe, celery)\n- New `beat` service (1 replica forever) runs the Celery Beat scheduler\n- Redis now enforces `maxmemory=300mb` + `allkeys-lru`\n- Backend has `/health/live` and `/health/ready` endpoints; Railway health checks point at `/health/ready`\n\n## Known operational gotchas\n\n- **Do not scale `beat` above 1 replica**, ever. Multiple beat instances fire scheduled tasks N times.\n- **Postgres backups are NOT yet automated.** Separate future spec. Until then, run a manual `pg_dump` to R2 weekly (or more often during active development).\n- If the Redis service redeploys, the startCommand override (maxmemory) may reset. Verify after any Redis restart with `redis-cli INFO memory | grep maxmemory`.\n- Worker replicas each hold ~16 Celery threads. A stuck thread under-utilizes half of one replica — not catastrophic. The watchdog task (every 5 min via beat) will mark stuck analyses as error.\n\n## Troubleshooting\n\n- **Backend returns 503 from /health/ready**: DB or Redis unreachable. Check `railway logs` on the `backend` service for the specific probe failure.\n- **Scheduled tasks firing twice**: check that `beat` is running with `numReplicas=1`. If the worker service accidentally re-added `--beat`, remove it.\n- **Queue backlog**: run `redis-cli -h yamabiko.proxy.rlwy.net -p 50187 -a $REDIS_PASSWORD LLEN analyze` — if it's stuck growing, a worker may have died; check `railway logs` on the worker service.\n```\n\n- [ ] **Step 2: Commit the runbook**\n\n```bash\ngit add docs/ops/post-scaling-operational-notes.md\ngit commit -m \"docs: post-scaling operational notes\n\nRunbook covering the new topology (2 backend replicas, 2 worker\nreplicas, 1 beat), known gotchas (don't scale beat, backups still\nmanual), and troubleshooting steps for common production issues.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n- [ ] **Step 3: WS5 handoff**\n\nAll waves complete. Production is on the new topology, smoke + burst tests pass, docs and memory are updated.\n\nIf anything under load reveals an unforeseen issue — memory pressure, queue backlog, replica contention — file a new spec and do not patch in place.\n\n---\n\n## Self-Review Checklist (mandatory before declaring plan complete)\n\nAfter writing the complete plan, I ran this checklist against the spec:\n\n- ✅ **Spec coverage:** Every track (T1 through T6) from the spec has a corresponding workstream (WS1–WS5) with concrete tasks. T3 (backups) is explicitly out of scope per the spec.\n- ✅ **No placeholders:** No `TBD`, `TODO`, or `implement later` in any task step. Every code block contains the actual code an engineer needs.\n- ✅ **Type/method consistency:** `analyze_chunk_step`, `analyze_infer_step`, etc. referenced consistently. `_check_cancellation` referenced across 3.2, 3.3. `handle_pipeline_error` referenced in 3.5, 3.8, 3.9. `_POOL_CONFIG`/`_pool` in 4.1.\n- ✅ **Exact file paths:** Every task lists the exact files it touches.\n- ✅ **Explicit commits:** Every task ends with a `git commit` step.\n- ✅ **Dependency order documented:** Wave 1 parallel, WS2 rebases on WS3, WS4 after Wave 1 soak, WS5 after WS4.\n- ✅ **Test code inline:** Tests in 3.3, 3.5, 3.6, 4.1, 4.2 have complete code, not just descriptions.\n\nOne area where the plan trusts the executor: **Task 4.7 Step 4** (manual first-time source config of the new beat service in the Railway dashboard) requires human interaction. Automated service creation via Railway's GraphQL API is partial — `serviceCreate` mutation exists but the source (repo + Dockerfile) is typically set by the dashboard UI for first-time setup. This is called out explicitly in the task so an executor knows to pause and do the dashboard step.\n"}, "60ad21dc1d80cb753ad00e7528a169e3be84d2b2": {"t": 1785765395.2575746, "n": 123265, "h": "21bddfe0db1cc14e", "c": "# Methodex Hobby-Plan Scaling — Implementation Plan\n\n> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.\n\n**Goal:** Refactor Methodex's Celery pipeline and Railway topology to serve ~100 users / ~25 concurrent analyses peak on the Hobby plan without single-points-of-failure, by chaining the analysis pipeline into per-step tasks, extracting Celery Beat into its own service, scaling backend+worker to 2 replicas each, and trimming idle memory footprint.\n\n**Architecture:** Five workstreams. WS1 (T6 — CI polish), WS2 (T4 — memory + cleanup), and WS3 (T1 — Celery chain refactor) run in **parallel worktrees**. WS4 (T2+T5 — infrastructure + Redis hardening) runs in **Wave 2** after WS1/WS2/WS3 have soaked on single-replica production for ≥24h. WS5 (validation) runs after WS4 lands. WS2 rebases on WS3 before merging because both edit the tasks/ directory.\n\n**Tech Stack:** FastAPI, Celery 5.4 (threads pool), SQLAlchemy 2.0 (sync psycopg2), PostgreSQL (Railway plugin), Redis (Railway plugin), uvicorn, GitHub Actions, Railway GraphQL API, pytest.\n\n**Design spec:** `docs/superpowers/specs/2026-04-06-methodex-hobby-scaling-design.md`\n\n---\n\n## File Map\n\n| Workstream | Modified | Created | Deleted |\n|---|---|---|---|\n| WS1 (T6 CI polish) | `.github/workflows/ci.yml` | `scripts/ci/wait-for-railway-deploy.py` | — |\n| WS2 (T4 memory + cleanup) | `backend/requirements.txt`, `backend/app/services/llm_service.py`, `backend/app/services/s3_service.py`, `backend/app/services/encryption_service.py`, `backend/app/auth.py` | `backend/tests/test_memory_baseline.py` | — |\n| WS3 (T1 chain refactor) | `backend/app/tasks/analysis_steps.py`, `backend/app/tasks/celery_app.py`, `backend/app/routes/videos.py`, `backend/app/routes/projects.py`, `backend/tests/test_analysis_retry.py`, `backend/tests/test_watchdog_race.py` | `backend/app/tasks/project_analysis_steps.py`, `backend/app/tasks/pipeline_errors.py`, `backend/app/tasks/_pipeline_utils.py`, `backend/tests/test_analysis_chain.py`, `backend/tests/test_project_analysis_chain.py`, `backend/tests/test_pipeline_errors.py` | `backend/app/tasks/analysis_tasks.py` |\n| WS4 (T2+T5 infra + Redis) | `backend/scripts/startup.sh`, `backend/app/database.py`, `backend/app/main.py`, Railway service config (via API) | `scripts/railway-service-config.py`, `backend/tests/test_healthcheck_ready.py`, `backend/tests/test_database_pool_config.py` | — |\n| WS5 (validation) | `memory/project_hobby_scaling_progress.md` (update status) | `scripts/production-smoke-test.py`, `scripts/burst-load-test.py` | — |\n\n---\n\n## Dependency & Execution Order\n\n```\nWAVE 1 (parallel worktrees, independent subagents):\n  ┌────────────┐  ┌──────────────┐  ┌────────────────┐\n  │ WS1 (T6)   │  │ WS2 (T4)     │  │ WS3 (T1)       │\n  │ CI polish  │  │ memory+rename│  │ chain refactor │\n  └─────┬──────┘  └──────┬───────┘  └───────┬────────┘\n        │                │                  │\n        │        rebase on WS3 ─────────────┤\n        │                │                  │\n        └────────────────┴──────────────────┘\n                         │\n                  Merge Wave 1 to main\n                         │\n                  Soak ≥24h on single-replica production\n                         │\nWAVE 2:           ┌──────▼───────┐\n                  │ WS4 (T2+T5)  │\n                  │ infra+Redis  │\n                  └──────┬───────┘\n                         │\n                  Merge Wave 2 to main\n                         │\nWAVE 3:           ┌──────▼───────┐\n                  │ WS5          │\n                  │ validation   │\n                  └──────────────┘\n```\n\n**Hard rules:**\n1. WS2 must rebase on WS3 before merging (both touch `backend/app/tasks/` imports).\n2. WS4 must not begin until WS1, WS2, WS3 are merged and have soaked.\n3. No `git push --force` to main ever. Use `git push` or merge via PR.\n4. Every task ends with a commit. Do not batch commits across tasks.\n5. Run `cd backend && pytest tests/ -v` before committing any backend code change; the test suite must stay green throughout.\n\n---\n\n## Task 0: Pre-flight (run once before launching workstreams)\n\n**Files:**\n- Read-only verification\n\n- [ ] **Step 1: Verify we're on main with a clean working tree**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngit status\ngit log --oneline origin/main..HEAD\n```\n\nExpected: current branch `main`, clean working tree, spec commit `5576daf` visible in `git log origin/main..HEAD`.\n\n- [ ] **Step 2: Verify backend test suite is green before any changes**\n\n```bash\ncd backend && pip install -r requirements.txt -r requirements-dev.txt && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: 0 failing tests, 0 errors.\n\nIf any tests fail on clean main before changes, STOP and investigate — do not start the refactor on a broken baseline.\n\n- [ ] **Step 3: Record current per-service memory footprint baseline**\n\nQuery Railway for current memory usage so we can measure improvement post-T4:\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin).get(\"user\",{}).get(\"token\",\"\"))')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ estimatedUsage(workspaceId: \\\"37eb4e96-0873-4033-a392-3c6593a68802\\\", projectId: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\", measurements: [MEMORY_USAGE_GB, CPU_USAGE, NETWORK_TX_GB, DISK_USAGE_GB]) { measurement estimatedValue } }\"}' \\\n  | python3 -m json.tool\n```\n\nSave the output to a scratch file — it will be compared against post-T4 numbers in WS5.\n\n- [ ] **Step 4: Create worktree directories for parallel execution**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngit worktree add -b ci/railway-deploy-wait ../5d-worktrees/ws1-ci origin/main\ngit worktree add -b cleanup/memory-and-llm-naming ../5d-worktrees/ws2-cleanup origin/main\ngit worktree add -b refactor/celery-chain ../5d-worktrees/ws3-chain origin/main\ngit worktree list\n```\n\nExpected: three worktree entries alongside the main checkout. Each subagent that takes on WS1/WS2/WS3 works inside the matching directory.\n\n**Note:** `origin/main` is the base so worktrees don't include local unpushed commits. If `origin/main` is behind and contains bugs the unpushed commits fix, rebase worktrees onto local main instead — but prefer the clean origin/main base when possible.\n\n---\n\n# WORKSTREAM 1 (WS1) — T6 CI Deploy-Wait\n\n**Worktree:** `../5d-worktrees/ws1-ci` on branch `ci/railway-deploy-wait`\n\n**Goal:** GitHub Actions CI reports green only after Railway has successfully deployed the backend for the current commit SHA, not merely after the test suite passes.\n\n**Blocking?** No — runs fully in parallel with WS2 and WS3.\n\n## Task 1.1: Add the Railway deploy-wait script\n\n**Files:**\n- Create: `scripts/ci/wait-for-railway-deploy.py`\n\n- [ ] **Step 1: Create the script**\n\nCreate `scripts/ci/wait-for-railway-deploy.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"Wait for a Railway backend deployment matching a given commit SHA to reach SUCCESS.\n\nUsed in GitHub Actions to block CI from reporting green until the backend is\nactually live. Polls the Railway GraphQL API every 15 seconds for up to 10 minutes.\n\nEnvironment variables:\n- RAILWAY_API_TOKEN: workspace-scoped Railway API token (GitHub secret)\n- RAILWAY_PROJECT_ID: Railway project id (default: methodex project id)\n- RAILWAY_BACKEND_SERVICE_ID: backend service id\n- COMMIT_SHA: the commit sha we're waiting for (usually $GITHUB_SHA)\n\"\"\"\n\nimport os\nimport sys\nimport time\nimport json\nimport urllib.request\nimport urllib.error\n\nRAILWAY_API = \"https://backboard.railway.app/graphql/v2\"\nPOLL_INTERVAL_SECONDS = 15\nMAX_WAIT_SECONDS = 600\n\nQUERY = \"\"\"\nquery DeploymentsForCommit($projectId: String!, $serviceId: String!) {\n  deployments(\n    first: 20\n    input: { projectId: $projectId, serviceId: $serviceId }\n  ) {\n    edges {\n      node {\n        id\n        status\n        meta\n        createdAt\n      }\n    }\n  }\n}\n\"\"\"\n\n\ndef gql(query: str, variables: dict, token: str) -> dict:\n    body = json.dumps({\"query\": query, \"variables\": variables}).encode(\"utf-8\")\n    req = urllib.request.Request(\n        RAILWAY_API,\n        data=body,\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {token}\",\n        },\n        method=\"POST\",\n    )\n    try:\n        with urllib.request.urlopen(req, timeout=30) as resp:\n            return json.loads(resp.read().decode(\"utf-8\"))\n    except urllib.error.HTTPError as e:\n        print(f\"HTTP {e.code}: {e.read().decode('utf-8', errors='replace')}\", file=sys.stderr)\n        raise\n\n\ndef find_deployment_for_sha(token: str, project_id: str, service_id: str, sha: str):\n    \"\"\"Return (deployment_id, status) for the deployment matching the SHA, or (None, None).\"\"\"\n    data = gql(QUERY, {\"projectId\": project_id, \"serviceId\": service_id}, token)\n    if \"errors\" in data:\n        print(f\"GraphQL errors: {data['errors']}\", file=sys.stderr)\n        return None, None\n    edges = (data.get(\"data\") or {}).get(\"deployments\", {}).get(\"edges\", [])\n    for edge in edges:\n        node = edge.get(\"node\", {})\n        meta = node.get(\"meta\") or {}\n        commit = None\n        if isinstance(meta, dict):\n            commit = meta.get(\"commitHash\") or meta.get(\"commit\") or meta.get(\"sha\")\n        if commit and commit.startswith(sha[:7]):\n            return node.get(\"id\"), node.get(\"status\")\n    return None, None\n\n\ndef main():\n    token = os.environ.get(\"RAILWAY_API_TOKEN\")\n    project_id = os.environ.get(\"RAILWAY_PROJECT_ID\", \"154d302f-8609-4897-a10c-1f0d5bfc4f06\")\n    service_id = os.environ.get(\"RAILWAY_BACKEND_SERVICE_ID\", \"2b70a900-042c-4083-b00b-0d01f3ece5dc\")\n    sha = os.environ.get(\"COMMIT_SHA\") or os.environ.get(\"GITHUB_SHA\")\n\n    if not token:\n        print(\"RAILWAY_API_TOKEN env var is required\", file=sys.stderr)\n        return 2\n    if not sha:\n        print(\"COMMIT_SHA (or GITHUB_SHA) env var is required\", file=sys.stderr)\n        return 2\n\n    print(f\"Waiting for Railway backend deployment of {sha[:7]} to reach SUCCESS...\")\n\n    start = time.monotonic()\n    while time.monotonic() - start < MAX_WAIT_SECONDS:\n        dep_id, status = find_deployment_for_sha(token, project_id, service_id, sha)\n        if status is None:\n            print(f\"  [{int(time.monotonic() - start)}s] No deployment found yet for {sha[:7]}\")\n        elif status == \"SUCCESS\":\n            print(f\"  [{int(time.monotonic() - start)}s] Deployment {dep_id} SUCCESS\")\n            return 0\n        elif status in (\"FAILED\", \"CRASHED\", \"REMOVED\"):\n            print(f\"  [{int(time.monotonic() - start)}s] Deployment {dep_id} {status} — aborting\")\n            return 1\n        else:\n            print(f\"  [{int(time.monotonic() - start)}s] Deployment {dep_id} {status}, still waiting\")\n        time.sleep(POLL_INTERVAL_SECONDS)\n\n    print(f\"Timed out after {MAX_WAIT_SECONDS}s waiting for deployment\", file=sys.stderr)\n    return 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make it executable**\n\n```bash\nchmod +x scripts/ci/wait-for-railway-deploy.py\n```\n\n- [ ] **Step 3: Smoke-test the script locally against the existing production deployment**\n\n```bash\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin).get(\"user\",{}).get(\"token\",\"\"))')\nexport COMMIT_SHA=$(git -C /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool rev-parse origin/main)\npython3 scripts/ci/wait-for-railway-deploy.py\n```\n\nExpected: script prints \"Deployment <id> SUCCESS\" within 30 seconds (the existing production deployment for origin/main already succeeded).\n\nIf the script cannot locate a deployment for the SHA, iterate on the `find_deployment_for_sha` query — Railway's `deployments` edge may expose the commit in a different meta key. Adjust the commit-matching logic until the smoke test passes.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add scripts/ci/wait-for-railway-deploy.py\ngit commit -m \"ci: add Railway deploy-wait script\n\nPolls Railway GraphQL for the backend deployment matching the current\ncommit SHA, blocks until SUCCESS (10 min timeout). Used by a new CI job\nso merges to main only report green when the backend is actually live.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 1.2: Add the deploy-wait job to CI\n\n**Files:**\n- Modify: `.github/workflows/ci.yml`\n\n- [ ] **Step 1: Read the current CI file**\n\n```bash\ncat .github/workflows/ci.yml\n```\n\nIdentify the `backend-ci` job and confirm the `deploy-frontend` job shape to follow conventions.\n\n- [ ] **Step 2: Append a new `wait-backend-deploy` job**\n\nEdit `.github/workflows/ci.yml`, adding a new job block at the bottom (after `deploy-frontend`):\n\n```yaml\n  wait-backend-deploy:\n    name: Wait for Railway Backend Deploy\n    needs: backend-ci\n    if: github.ref == 'refs/heads/main' && github.event_name == 'push'\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v5\n\n      - name: Set up Python 3.11\n        uses: actions/setup-python@v6\n        with:\n          python-version: \"3.11\"\n\n      - name: Wait for Railway deployment to reach SUCCESS\n        env:\n          RAILWAY_API_TOKEN: ${{ secrets.RAILWAY_API_TOKEN }}\n          RAILWAY_PROJECT_ID: \"154d302f-8609-4897-a10c-1f0d5bfc4f06\"\n          RAILWAY_BACKEND_SERVICE_ID: \"2b70a900-042c-4083-b00b-0d01f3ece5dc\"\n          COMMIT_SHA: ${{ github.sha }}\n        run: python3 scripts/ci/wait-for-railway-deploy.py\n```\n\n- [ ] **Step 3: Verify the YAML is valid**\n\n```bash\npython3 -c \"import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))\" && echo OK\n```\n\nExpected: `OK`.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add .github/workflows/ci.yml\ngit commit -m \"ci: add wait-backend-deploy job\n\nBlocks CI from reporting green until Railway confirms the backend\ndeployment reached SUCCESS for the pushed commit. Uses the new\nscripts/ci/wait-for-railway-deploy.py helper. Requires a new\nRAILWAY_API_TOKEN secret to be added in GitHub repo settings.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 1.3: Document the new secret requirement\n\n**Files:**\n- Modify: `README.md` (or create one-liner ops doc)\n\n- [ ] **Step 1: Check README for a secrets section**\n\n```bash\ngrep -in \"secret\\|RAILWAY_API_TOKEN\" README.md 2>/dev/null | head -10\n```\n\n- [ ] **Step 2: If a secrets section exists, add the new secret there; otherwise create `docs/ops/ci-secrets.md`**\n\nCreate `docs/ops/ci-secrets.md` (if README has no secrets section):\n\n```markdown\n# CI Secrets\n\nGitHub Actions secrets required by `.github/workflows/ci.yml`:\n\n| Secret | Purpose | How to obtain |\n|---|---|---|\n| `CLOUDFLARE_API_TOKEN` | Deploy frontend to Cloudflare Pages | Cloudflare dashboard → My Profile → API Tokens |\n| `CLOUDFLARE_ACCOUNT_ID` | Cloudflare account selection | Cloudflare dashboard, right sidebar |\n| `VITE_CLERK_PUBLISHABLE_KEY` | Production Clerk key (pk_live_…) | Clerk dashboard → API Keys |\n| `VITE_API_URL` | `https://api.methodex.ai` | Static |\n| `VITE_CLERK_PROXY_URL` | `/__clerk` (reverse-proxy path) | Static |\n| `VITE_SENTRY_DSN` | Frontend Sentry DSN | Sentry project settings |\n| `SENTRY_AUTH_TOKEN` | Source-map upload | Sentry account → auth tokens |\n| `RAILWAY_API_TOKEN` | Railway GraphQL token for deploy-wait job | Railway dashboard → Account → Tokens (workspace-scoped) |\n\nTo rotate a secret, create a new one in the dashboard, update the GitHub\nActions secret, then revoke the old one.\n```\n\n- [ ] **Step 3: Commit**\n\n```bash\ngit add docs/ops/ci-secrets.md\ngit commit -m \"docs: add CI secrets inventory\n\nLists all GitHub Actions secrets consumed by .github/workflows/ci.yml\nand how to obtain them. Adds the new RAILWAY_API_TOKEN entry required\nby the wait-backend-deploy job.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 1.4: WS1 handoff\n\n- [ ] **Step 1: Final verification in the WS1 worktree**\n\n```bash\ngit log --oneline origin/main..HEAD\n```\n\nExpected: 3 commits (add script, add CI job, add docs).\n\n- [ ] **Step 2: Note manual action required**\n\nThe `RAILWAY_API_TOKEN` GitHub secret must be added before this PR merges to main, or the new CI job will fail. Flag this in the PR description:\n\n> **⚠️ Pre-merge action:** Add `RAILWAY_API_TOKEN` to GitHub Actions secrets (workspace-scoped Railway token). Otherwise the new `wait-backend-deploy` job will fail on the first push to main.\n\nWS1 is complete. The branch is ready for PR review and merge.\n\n---\n\n# WORKSTREAM 2 (WS2) — T4 Memory + llm_service Cleanup\n\n**Worktree:** `../5d-worktrees/ws2-cleanup` on branch `cleanup/memory-and-llm-naming`\n\n**Goal:** Remove unused `langgraph` dependency, lazy-import heavy modules (boto3, openai, cryptography), rename `FREE_MODEL_FALLBACKS → STANDARD_MODEL_FALLBACKS`, fix stale docstrings that describe the app as using free-tier OpenRouter models (it does not).\n\n**Blocking?** Must **rebase on WS3** before merging because both WS2 and WS3 touch the `backend/app/tasks/` directory and imports. Start in parallel; rebase at merge time.\n\n## Task 2.1: Verify langgraph is unused before removing it\n\n**Files:**\n- Read-only verification\n\n- [ ] **Step 1: Grep for any remaining langgraph imports**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngrep -rn \"from langgraph\\|import langgraph\" backend/app backend/tests 2>&1\n```\n\nExpected: zero hits. If any code still imports `langgraph`, STOP — fix those imports to use direct node function calls (see `backend/app/tasks/analysis_tasks.py:1-12` docstring for the pattern) BEFORE removing the dependency. File a finding in the commit message and reopen Task 2.1 if needed.\n\n- [ ] **Step 2: Confirm requirements.txt has the line**\n\n```bash\ngrep -n \"^langgraph\" backend/requirements.txt\n```\n\nExpected: one hit, line ~14.\n\n## Task 2.2: Remove langgraph from requirements.txt\n\n**Files:**\n- Modify: `backend/requirements.txt`\n\n- [ ] **Step 1: Remove the langgraph line**\n\nEdit `backend/requirements.txt`, delete the line `langgraph==1.0.10`.\n\n- [ ] **Step 2: Reinstall to verify the app still imports**\n\n```bash\ncd backend && pip uninstall -y langgraph && pip install -r requirements.txt\npython -c \"import app.main; print('OK')\"\n```\n\nExpected: `OK` with no ImportError.\n\n- [ ] **Step 3: Run the full backend test suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -20\n```\n\nExpected: all tests pass (same count as the pre-flight baseline).\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/requirements.txt\ngit commit -m \"chore: drop unused langgraph dependency\n\nThe codebase moved to direct node function calls in 2026-03 (see\nbackend/app/tasks/analysis_tasks.py docstring) but langgraph stayed\nin requirements.txt. Removing it shrinks per-container idle memory\nby ~50 MB per Python process.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.3: Trim sentry-sdk extras\n\n**Files:**\n- Modify: `backend/requirements.txt`\n\n- [ ] **Step 1: Change sentry-sdk extras**\n\nEdit `backend/requirements.txt`, change:\n\n```\nsentry-sdk[fastapi,celery,openai]>=2.41.0\n```\n\nto:\n\n```\nsentry-sdk[fastapi,celery]>=2.41.0\n```\n\n- [ ] **Step 2: Reinstall and verify**\n\n```bash\ncd backend && pip install -r requirements.txt && python -c \"import sentry_sdk; print(sentry_sdk.VERSION)\"\n```\n\nExpected: version string printed, no import errors.\n\n- [ ] **Step 3: Run the test suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -20\n```\n\nExpected: all tests pass. If `test_sentry_config.py` or `test_llm_monitoring.py` tests fail because they reference the removed openai integration, update them to stop asserting on OpenAI Sentry spans — we deliberately removed that integration.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/requirements.txt\ngit commit -m \"chore: remove openai extra from sentry-sdk\n\nThe OpenAI Sentry integration auto-instruments every LLM call with\ndetailed traces, which is noisy for our workload and not what we use\nSentry for. Keeping [fastapi,celery] which we actually need.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.4: Lazy-import boto3 in s3_service.py\n\n**Files:**\n- Modify: `backend/app/services/s3_service.py`\n\n- [ ] **Step 1: Read the file to find the top-level boto3 import**\n\n```bash\ngrep -n \"^import boto3\\|^from boto3\\|^from botocore\" backend/app/services/s3_service.py\n```\n\n- [ ] **Step 2: Move the boto3 import into the function(s) that use it**\n\nEdit `backend/app/services/s3_service.py`. Find every top-level `import boto3` / `from botocore...` statement and move them into the first function or method that actually uses boto3. For example, if the module has:\n\n```python\nimport boto3\nfrom botocore.client import Config\n```\n\nAnd a class `S3Service` with a method `_get_client()` that uses them, move the imports into `_get_client`:\n\n```python\ndef _get_client(self):\n    import boto3\n    from botocore.client import Config\n    return boto3.client(...)\n```\n\nKeep the imports lazy in every function that touches boto3 (Python caches module imports so repeated calls have near-zero overhead).\n\n- [ ] **Step 3: Verify the backend still imports and the tests still pass**\n\n```bash\ncd backend && python -c \"from app.services.s3_service import s3_service; print('OK')\"\npytest tests/ -v 2>&1 | tail -20\n```\n\nExpected: `OK`, all tests pass. If `test_upload_validation.py` or related tests fail, the lazy import is probably inside a branch that the test never exercises — move it earlier in the function or to a helper that's called on init.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/services/s3_service.py\ngit commit -m \"perf: lazy-import boto3 in s3_service\n\nboto3 + botocore loads ~80 MB on import. The backend API process\nnever touches S3 (only the worker does), so deferring the import\nuntil first use sheds that memory from every backend replica.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.5: Lazy-import openai in llm_service.py\n\n**Files:**\n- Modify: `backend/app/services/llm_service.py`\n\n- [ ] **Step 1: Find the top-level openai import**\n\n```bash\ngrep -n \"^from openai\\|^import openai\" backend/app/services/llm_service.py\n```\n\n- [ ] **Step 2: Move it into the client factory**\n\nFind the method (probably `__init__` or `_get_client`) that constructs the OpenAI client. Move the `from openai import OpenAI` (or equivalent) into that method.\n\nExample pattern if the current code is:\n\n```python\nfrom openai import OpenAI\n\nclass LLMService:\n    def __init__(self):\n        self.client = OpenAI(base_url=..., api_key=...)\n```\n\nChange to:\n\n```python\nclass LLMService:\n    def __init__(self):\n        self._client = None\n\n    @property\n    def client(self):\n        if self._client is None:\n            from openai import OpenAI\n            self._client = OpenAI(base_url=..., api_key=...)\n        return self._client\n```\n\nPreserve every existing call site's behavior.\n\n- [ ] **Step 3: Run the llm_service tests**\n\n```bash\ncd backend && pytest tests/test_llm_service.py tests/test_llm_service_retry.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/services/llm_service.py\ngit commit -m \"perf: lazy-import openai in llm_service\n\nDefers the ~25 MB openai SDK import until the first LLM call.\nUses a cached @property so subsequent calls have no overhead.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.6: Lazy-import cryptography in encryption_service.py\n\n**Files:**\n- Modify: `backend/app/services/encryption_service.py`\n\n- [ ] **Step 1: Find the top-level cryptography import**\n\n```bash\ngrep -n \"cryptography\" backend/app/services/encryption_service.py\n```\n\n- [ ] **Step 2: Move `from cryptography.fernet import Fernet` into the functions that use it**\n\nSame pattern as Task 2.5 — defer the import to first use, cache if needed.\n\n- [ ] **Step 3: Run encryption tests**\n\n```bash\ncd backend && pytest tests/test_encryption.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/services/encryption_service.py\ngit commit -m \"perf: lazy-import cryptography in encryption_service\n\nDefers the ~15 MB cryptography module import until the first\nencrypt/decrypt call. Most requests don't touch BYOK keys so\nthis shaves idle memory from backend replicas.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.7: Rename FREE_MODEL_FALLBACKS → STANDARD_MODEL_FALLBACKS\n\n**Files:**\n- Modify: `backend/app/services/llm_service.py`\n\n- [ ] **Step 1: Find every reference to the symbol**\n\n```bash\ngrep -rn \"FREE_MODEL_FALLBACKS\" backend/\n```\n\n- [ ] **Step 2: Rename every occurrence**\n\nIn `backend/app/services/llm_service.py`, rename the constant and all references:\n\n```python\n# Before\nFREE_MODEL_FALLBACKS = [\n    \"meta-llama/llama-4-scout\",\n    \"nvidia/nemotron-3-super-120b-a12b\",\n    \"deepseek/deepseek-chat-v3-0324\",\n]\n\n# After\nSTANDARD_MODEL_FALLBACKS = [\n    \"meta-llama/llama-4-scout\",\n    \"nvidia/nemotron-3-super-120b-a12b\",\n    \"deepseek/deepseek-chat-v3-0324\",\n]\n```\n\nAnd update every reference in the file (grep output from Step 1).\n\n- [ ] **Step 3: Verify no lingering references**\n\n```bash\ngrep -rn \"FREE_MODEL_FALLBACKS\" backend/\n```\n\nExpected: zero hits.\n\n- [ ] **Step 4: Run the relevant tests**\n\n```bash\ncd backend && pytest tests/test_llm_service.py tests/test_llm_service_retry.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/services/llm_service.py\ngit commit -m \"refactor: rename FREE_MODEL_FALLBACKS to STANDARD_MODEL_FALLBACKS\n\nThe list contains paid open-source models on OpenRouter, not free-tier\nmodels. The 'FREE' prefix was a historical artifact from the original\nmigration plan that confused both humans and AI assistants.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.8: Fix stale free-tier docstrings in llm_service.py\n\n**Files:**\n- Modify: `backend/app/services/llm_service.py`\n\n- [ ] **Step 1: Find the stale comments**\n\n```bash\ngrep -n \"free.*[Oo]penRouter\\|[Ff]ree.*model\" backend/app/services/llm_service.py\n```\n\nExpected hits include:\n- Line ~199–200: \"When the primary model is persistently rate-limited (common with free OpenRouter models)\"\n- Line ~296–297: \"Free models may return null content (e.g. finish_reason: length\"\n\n- [ ] **Step 2: Rewrite those comments**\n\nChange:\n\n```python\n# When the primary model is persistently rate-limited (common with free\n# OpenRouter models), this method automatically tries fallback models.\n```\n\nto:\n\n```python\n# When the primary model is persistently rate-limited, this method\n# automatically tries fallback models from STANDARD_MODEL_FALLBACKS.\n```\n\nAnd:\n\n```python\n# Free models may return null content (e.g. finish_reason: length\n# with no output). Fall back to next model when using shared key.\n```\n\nto:\n\n```python\n# Open-source models may return null content (e.g. finish_reason:\n# length with no output). Fall back to next model when using shared key.\n```\n\n- [ ] **Step 3: Verify no \"free\" references remain in the LLM service**\n\n```bash\ngrep -in \"free\" backend/app/services/llm_service.py\n```\n\nExpected: zero hits related to model tier (hits on unrelated English words like \"free memory\" are fine).\n\n- [ ] **Step 4: Run tests**\n\n```bash\ncd backend && pytest tests/test_llm_service.py tests/test_llm_service_retry.py -v\n```\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/services/llm_service.py\ngit commit -m \"docs: correct stale 'free OpenRouter models' comments in llm_service\n\nThe app has never used free-tier OpenRouter models — DEFAULT_MODEL is\nmeta-llama/llama-4-scout (paid) and there is no ':free' suffix anywhere\nin production code paths. The comments were vestigial language from\nthe original migration plan.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 2.9: WS2 handoff and merge prep\n\n- [ ] **Step 1: Run full backend test suite**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass. Count should match the pre-flight baseline.\n\n- [ ] **Step 2: Rebase on WS3 (chain-refactor) before merging**\n\nWhen WS3 lands on main, rebase WS2 on top:\n\n```bash\ncd ../5d-worktrees/ws2-cleanup\ngit fetch origin\ngit rebase origin/main\n```\n\nResolve any conflicts in `backend/app/tasks/` imports — both branches touch that area. After rebase, re-run `pytest tests/ -v` and fix any test breakage before merging.\n\n- [ ] **Step 3: Verify WS2 commits**\n\n```bash\ngit log --oneline origin/main..HEAD\n```\n\nExpected: 7 commits from this workstream (drop langgraph, trim sentry, lazy-import ×3, rename, docs).\n\nWS2 is complete.\n\n---\n\n# WORKSTREAM 3 (WS3) — T1 Celery Chain Refactor\n\n**Worktree:** `../5d-worktrees/ws3-chain` on branch `refactor/celery-chain`\n\n**Goal:** Convert the monolithic `analyze_video_task` and `analyze_project_task` into Celery `chain`s of per-step tasks. Each step task runs independently, frees its worker thread between steps, and has its own retry lifecycle. Delete the monolithic task file entirely.\n\n**Blocking?** No — runs in parallel with WS1 and WS2. But this must merge BEFORE WS2 so WS2 can rebase on it cleanly.\n\n## Task 3.1: Create shared pipeline utilities module\n\n**Files:**\n- Create: `backend/app/tasks/_pipeline_utils.py`\n- Modify: `backend/app/tasks/celery_app.py` (later)\n\n- [ ] **Step 1: Extract sanitize_error and error-JSON helpers from analysis_tasks.py**\n\nCreate `backend/app/tasks/_pipeline_utils.py`:\n\n```python\n\"\"\"Shared helpers for the analysis pipeline tasks.\n\nExtracted from the pre-refactor analysis_tasks.py so that step tasks\nand the chain error handler can share the same sanitization and\nstructured-error logic.\n\"\"\"\n\nimport json\nimport re\nfrom typing import Optional\n\nfrom app.utils.error_classification import build_structured_error, is_retryable\n\n# Pattern matches common API key formats (OpenRouter sk-or-*, generic long tokens)\n_API_KEY_PATTERN = re.compile(\n    r\"(sk-or-v1-[A-Za-z0-9]{4})[A-Za-z0-9]{20,}\"  # OpenRouter keys\n    r\"|\"\n    r\"(sk-[A-Za-z0-9]{4})[A-Za-z0-9]{20,}\"  # OpenAI-style keys\n    r\"|\"\n    r\"(Bearer\\s+)[A-Za-z0-9_\\-]{20,}\"  # Bearer tokens in error messages\n    r\"|\"\n    r\"([a-f0-9]{4})[a-f0-9]{28,}\"  # AssemblyAI and other hex keys\n)\n\n\ndef sanitize_error(message: str) -> str:\n    \"\"\"Strip potential API key material from error messages before storage.\"\"\"\n    return _API_KEY_PATTERN.sub(\n        lambda m: (m.group(1) or m.group(2) or m.group(3) or m.group(4) or \"\") + \"***REDACTED***\",\n        message,\n    )\n\n\ndef build_error_json(step: str, exc: Exception, message: str) -> str:\n    \"\"\"Build a structured error JSON string from pipeline state info.\"\"\"\n    return json.dumps(build_structured_error(\n        step=step,\n        exc=exc,\n        message=sanitize_error(message),\n    ))\n\n\ndef build_pipeline_error_json(failed_step: str, error_str: str, error_type: Optional[str] = None) -> str:\n    \"\"\"Build a structured error JSON string for a pipeline node failure.\"\"\"\n    etype = error_type or \"unknown\"\n    return json.dumps({\n        \"step\": failed_step,\n        \"error_type\": etype,\n        \"retryable\": is_retryable(etype),\n        \"message\": f\"Analysis failed at step '{failed_step}': {error_str}\",\n        \"details\": error_str,\n    })\n\n\nclass PipelineError(Exception):\n    \"\"\"Internal exception carrying structured error JSON from the pipeline.\"\"\"\n\n    def __init__(self, message: str, structured_json: Optional[str] = None):\n        super().__init__(message)\n        self.structured_json = structured_json\n```\n\n- [ ] **Step 2: Commit**\n\n```bash\ngit add backend/app/tasks/_pipeline_utils.py\ngit commit -m \"refactor: extract pipeline utilities into _pipeline_utils module\n\nExtracts sanitize_error, build_error_json, build_pipeline_error_json,\nand PipelineError from the monolithic analysis_tasks.py so the chain\nrefactor can share them between step tasks and the error handler.\n\nNo functional change — helpers are byte-identical to the originals.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.2: Add cancellation precheck helper for step tasks\n\n**Files:**\n- Modify: `backend/app/tasks/analysis_steps.py`\n\n- [ ] **Step 1: Add a cancellation precheck helper near the top of the file**\n\nIn `backend/app/tasks/analysis_steps.py`, after the existing imports and before `class NonRetryableAnalysisError`, add:\n\n```python\nfrom app.tasks._pipeline_utils import sanitize_error, build_error_json\n\n\ndef _check_cancellation(db: Session, video_id: str) -> bool:\n    \"\"\"Return True if the analysis should stop (watchdog error, row gone).\n\n    Called at the start of every step task in the chain so a halted\n    pipeline doesn't do redundant work on subsequent links.\n    \"\"\"\n    try:\n        db.expire_all()\n        analysis = db.query(VideoAnalysis).filter(\n            VideoAnalysis.video_id == UUID(video_id)\n        ).first()\n        if analysis is None or analysis.status == \"error\":\n            return True\n        return False\n    except Exception:\n        logger.exception(\"_check_cancellation failed, proceeding\")\n        return False\n```\n\n- [ ] **Step 2: Verify tests still pass**\n\n```bash\ncd backend && pytest tests/test_analysis_retry.py tests/test_watchdog_race.py tests/test_analysis_step_non_retryable.py -v\n```\n\nExpected: all pass.\n\n- [ ] **Step 3: Commit**\n\n```bash\ngit add backend/app/tasks/analysis_steps.py\ngit commit -m \"refactor: add _check_cancellation helper for chain halts\n\nEach step task in the upcoming Celery chain will call this at the top\nto exit early if the watchdog (or a prior failed step) already marked\nthe analysis as error. Replaces the in-function _is_cancelled polling\nfrom the monolithic analyze_video_task.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.3: Wire cancellation precheck into every existing video step task\n\n**Files:**\n- Modify: `backend/app/tasks/analysis_steps.py`\n\n- [ ] **Step 1: Add precheck call to each of the 5 step tasks**\n\nFor each of `analyze_chunk_step`, `analyze_infer_step`, `analyze_relate_step`, `analyze_explain_step`, `analyze_activate_step`, add a cancellation check at the very top of the `try:` block, right after the `logger.info(f\"Starting ... for video {video_id}\")` line.\n\nExample for `analyze_chunk_step`:\n\n```python\ndef analyze_chunk_step(self, video_id: str, user_id: str | None = None):\n    \"\"\"\n    Step 1: CHUNK - Break transcript into discrete pieces.\n    \"\"\"\n    try:\n        logger.info(f\"Starting CHUNK step for video {video_id}\")\n\n        # Cancellation precheck — watchdog or prior halted step\n        if _check_cancellation(self.db, video_id):\n            logger.info(f\"Skipping chunk for {video_id} — already in error state\")\n            return {\"video_id\": video_id, \"status\": \"skipped\"}\n\n        # ... existing code continues ...\n```\n\nRepeat for all 5 step tasks.\n\n- [ ] **Step 2: Write a test that verifies the cancellation precheck returns \"skipped\"**\n\nCreate or update `backend/tests/test_analysis_chain.py`:\n\n```python\n\"\"\"Tests for the Celery chain-based analysis pipeline.\"\"\"\n\nfrom unittest.mock import patch\nfrom uuid import uuid4\n\nimport pytest\nfrom sqlalchemy.orm import Session\n\nfrom app.models.database_models import Project, Transcript, Video, VideoAnalysis\nfrom app.tasks.analysis_steps import analyze_chunk_step, _check_cancellation\n\n\ndef _seed_project_video_transcript(db: Session, user_id: str = \"dev_user_local\"):\n    project = Project(name=\"test\", user_id=user_id)\n    db.add(project)\n    db.flush()\n    video = Video(\n        project_id=project.id,\n        filename=\"v.mp4\",\n        s3_key=f\"videos/{project.id}/{uuid4()}/v.mp4\",\n        s3_url=\"https://example/v.mp4\",\n        file_size_bytes=100,\n        status=\"transcribed\",\n    )\n    db.add(video)\n    db.flush()\n    transcript = Transcript(\n        video_id=video.id,\n        status=\"completed\",\n        processed_transcript={\"duration_seconds\": 60},\n        raw_transcript={\"utterances\": []},\n    )\n    db.add(transcript)\n    db.commit()\n    return project, video\n\n\ndef test_cancellation_precheck_returns_skipped_when_analysis_already_in_error(db_session):\n    \"\"\"If watchdog marked analysis as error, step task should return skipped.\"\"\"\n    _, video = _seed_project_video_transcript(db_session)\n    analysis = VideoAnalysis(\n        video_id=video.id,\n        status=\"error\",  # watchdog already marked it\n        step_status={\"chunk\": \"error\"},\n    )\n    db_session.add(analysis)\n    db_session.commit()\n\n    assert _check_cancellation(db_session, str(video.id)) is True\n\n\ndef test_cancellation_precheck_returns_false_when_analysis_is_processing(db_session):\n    \"\"\"If analysis is actively processing, step task should continue.\"\"\"\n    _, video = _seed_project_video_transcript(db_session)\n    analysis = VideoAnalysis(\n        video_id=video.id,\n        status=\"processing\",\n        step_status={\"chunk\": \"processing\"},\n    )\n    db_session.add(analysis)\n    db_session.commit()\n\n    assert _check_cancellation(db_session, str(video.id)) is False\n```\n\n- [ ] **Step 3: Run the new test**\n\n```bash\ncd backend && pytest tests/test_analysis_chain.py -v\n```\n\nExpected: both tests pass. If `db_session` fixture doesn't exist in `conftest.py`, adapt the test to use whatever fixture the existing suite uses for DB sessions (grep `conftest.py` for `db_session` or similar).\n\n- [ ] **Step 4: Run the full analysis-step test suite**\n\n```bash\npytest tests/test_analysis_retry.py tests/test_analysis_step_non_retryable.py tests/test_watchdog_race.py tests/test_analysis_chain.py -v\n```\n\nExpected: all pass. If `test_watchdog_race.py` breaks because it relied on the old `_is_cancelled` behavior, update it to assert the new `_check_cancellation` flow.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/tasks/analysis_steps.py backend/tests/test_analysis_chain.py\ngit commit -m \"refactor: add cancellation precheck to all video step tasks\n\nEvery chain link now exits early if watchdog (or a prior step) marked\nthe analysis as error. Returns {status: skipped} so downstream chain\nlinks see a clean non-error return and also short-circuit naturally.\n\nAdds test_analysis_chain.py with initial coverage for the precheck.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.4: Move initial-state setup from route into analyze_chunk_step\n\n**Files:**\n- Modify: `backend/app/tasks/analysis_steps.py`\n\n- [ ] **Step 1: Update analyze_chunk_step to create/reset VideoAnalysis if needed**\n\nIn `backend/app/tasks/analysis_steps.py`, update `analyze_chunk_step` so it handles the initial-state transitions that were previously done in the route's `trigger_video_analysis` handler:\n\n```python\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_chunk_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_chunk_step(self, video_id: str, user_id: str | None = None):\n    \"\"\"Step 1: CHUNK — break transcript into discrete pieces.\n\n    Also responsible for chain-start state transitions: create or reset\n    the VideoAnalysis row, mark all steps pending, set status=processing.\n    \"\"\"\n    try:\n        logger.info(f\"Starting CHUNK step for video {video_id}\")\n\n        if _check_cancellation(self.db, video_id):\n            logger.info(f\"Skipping chunk for {video_id} — already in error state\")\n            return {\"video_id\": video_id, \"status\": \"skipped\"}\n\n        state = get_video_analysis_state(self.db, UUID(video_id))\n        analysis = state[\"analysis\"]\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        # Chain-start transitions — previously done in the route\n        analysis.status = \"processing\"\n        analysis.current_step = \"chunk\"\n        analysis.step_status = {\n            \"chunk\": \"processing\",\n            \"infer\": \"pending\",\n            \"relate\": \"pending\",\n            \"explain\": \"pending\",\n            \"activate\": \"pending\",\n        }\n        analysis.started_at = datetime.now(timezone.utc)\n\n        # Mark the video as analyzing (may already be set by the route)\n        video = self.db.query(Video).filter(Video.id == UUID(video_id)).first()\n        if video:\n            video.status = \"analyzing\"\n            video.error_message = None\n        self.db.commit()\n\n        result = chunk_node({\n            \"video_id\": video_id,\n            \"transcript\": state[\"transcript\"],\n            \"speaker_labels\": state[\"speaker_labels\"],\n            \"speaker_roles\": state[\"speaker_roles\"],\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"chunks\") is None:\n            _raise_for_node_error(\"chunk\", result)\n\n        analysis.chunks = result.get(\"chunks\")\n        analysis.chunk_completed_at = datetime.now(timezone.utc)\n        analysis.step_status = {**(analysis.step_status or {}), \"chunk\": \"completed\"}\n        self.db.commit()\n\n        logger.info(f\"CHUNK step completed for video {video_id}\")\n        return {\n            \"video_id\": video_id,\n            \"status\": \"success\",\n            \"chunks_count\": len(result.get(\"chunks\", [])),\n        }\n\n    except Exception as e:\n        logger.error(f\"CHUNK step failed for video {video_id}: {e}\")\n        _update_analysis_error(self.db, video_id, \"chunk\")\n        raise\n```\n\n- [ ] **Step 2: Add a test for the chain-start transitions**\n\nAppend to `backend/tests/test_analysis_chain.py`:\n\n```python\ndef test_chunk_step_initializes_step_status_and_sets_processing(db_session, monkeypatch):\n    \"\"\"analyze_chunk_step should init step_status dict and mark analysis processing.\"\"\"\n    _, video = _seed_project_video_transcript(db_session)\n\n    # Stub the chunk_node so we don't hit the LLM\n    from app.tasks import analysis_steps\n    def fake_chunk_node(state):\n        return {\"chunks\": [{\"id\": \"C001\", \"text\": \"hello\"}], \"current_step\": \"chunk\"}\n    monkeypatch.setattr(analysis_steps, \"chunk_node\", fake_chunk_node)\n\n    # Stub BYOK resolver\n    monkeypatch.setattr(\n        analysis_steps, \"_resolve_byok\",\n        lambda db, user_id: (None, None),\n    )\n\n    # Invoke the task function directly (bypassing Celery's queue)\n    task_self = type(\"T\", (), {\"db\": db_session, \"request\": type(\"R\", (), {\"retries\": 0})()})\n    try:\n        analysis_steps.analyze_chunk_step.run.__wrapped__(task_self, str(video.id), \"dev_user_local\")\n    except AttributeError:\n        # Older Celery versions expose the function differently\n        analysis_steps.analyze_chunk_step(str(video.id), \"dev_user_local\")\n\n    analysis = db_session.query(VideoAnalysis).filter_by(video_id=video.id).first()\n    assert analysis is not None\n    assert analysis.status == \"processing\"\n    assert analysis.step_status[\"chunk\"] == \"completed\"\n    assert analysis.step_status[\"infer\"] == \"pending\"\n    assert analysis.step_status[\"relate\"] == \"pending\"\n    assert analysis.step_status[\"explain\"] == \"pending\"\n    assert analysis.step_status[\"activate\"] == \"pending\"\n```\n\n- [ ] **Step 3: Run the test**\n\n```bash\ncd backend && pytest tests/test_analysis_chain.py -v\n```\n\nExpected: all tests pass. If the task-invocation pattern (`run.__wrapped__`) doesn't match this Celery version, adapt to however the existing tests call step tasks directly — grep `backend/tests/test_analysis_retry.py` for the pattern that works.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/tasks/analysis_steps.py backend/tests/test_analysis_chain.py\ngit commit -m \"refactor: move initial-state setup into analyze_chunk_step\n\nChain-start state transitions (create VideoAnalysis row if missing,\ninit step_status dict, set status=processing, clear error_message)\nmove from the route handler into the chunk step task. The route\nbecomes a pure dispatcher.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.5: Create pipeline_errors.py with the chain error handler\n\n**Files:**\n- Create: `backend/app/tasks/pipeline_errors.py`\n- Create: `backend/tests/test_pipeline_errors.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_pipeline_errors.py`:\n\n```python\n\"\"\"Tests for the chain error handler.\"\"\"\n\nfrom uuid import uuid4\nfrom unittest.mock import MagicMock\n\nimport pytest\n\nfrom app.models.database_models import Project, Video, VideoAnalysis\n\n\ndef _seed_video_in_processing(db_session, user_id=\"dev_user_local\"):\n    project = Project(name=\"test\", user_id=user_id)\n    db_session.add(project)\n    db_session.flush()\n    video = Video(\n        project_id=project.id,\n        filename=\"v.mp4\",\n        s3_key=f\"videos/{project.id}/{uuid4()}/v.mp4\",\n        s3_url=\"https://example/v.mp4\",\n        file_size_bytes=100,\n        status=\"analyzing\",\n    )\n    db_session.add(video)\n    db_session.flush()\n    analysis = VideoAnalysis(\n        video_id=video.id,\n        status=\"processing\",\n        step_status={\"chunk\": \"completed\", \"infer\": \"processing\"},\n    )\n    db_session.add(analysis)\n    db_session.commit()\n    return video, analysis\n\n\ndef test_handle_pipeline_error_marks_video_and_analysis_error(db_session):\n    \"\"\"When a chain link fails, the error handler should mark both records.\"\"\"\n    from app.tasks.pipeline_errors import handle_pipeline_error\n\n    video, analysis = _seed_video_in_processing(db_session)\n    fake_exc = RuntimeError(\"LLM timeout\")\n    fake_request = MagicMock()\n    fake_request.task = \"analyze_infer_step\"\n\n    task_self = type(\"T\", (), {\"db\": db_session})()\n    handle_pipeline_error.run.__func__(\n        task_self, fake_request, fake_exc, \"fake_traceback\", str(video.id)\n    )\n\n    db_session.refresh(video)\n    db_session.refresh(analysis)\n    assert video.status == \"error\"\n    assert video.error_message  # JSON string\n    assert analysis.status == \"error\"\n    assert analysis.completed_at is not None\n\n\ndef test_handle_pipeline_error_is_idempotent(db_session):\n    \"\"\"Running the error handler twice should not change state after the first.\"\"\"\n    from app.tasks.pipeline_errors import handle_pipeline_error\n\n    video, analysis = _seed_video_in_processing(db_session)\n    fake_exc = RuntimeError(\"boom\")\n    fake_request = MagicMock()\n    fake_request.task = \"analyze_infer_step\"\n    task_self = type(\"T\", (), {\"db\": db_session})()\n\n    handle_pipeline_error.run.__func__(task_self, fake_request, fake_exc, \"tb\", str(video.id))\n    first_error_msg = video.error_message\n\n    handle_pipeline_error.run.__func__(task_self, fake_request, fake_exc, \"tb\", str(video.id))\n    db_session.refresh(video)\n    # Should be unchanged on the second call\n    assert video.error_message == first_error_msg\n    assert video.status == \"error\"\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_pipeline_errors.py -v\n```\n\nExpected: FAIL with `ModuleNotFoundError: No module named 'app.tasks.pipeline_errors'`.\n\n- [ ] **Step 3: Create the pipeline_errors.py module**\n\nCreate `backend/app/tasks/pipeline_errors.py`:\n\n```python\n\"\"\"Chain error handler for the Celery analysis pipeline.\n\nWhen a link in the analysis chain fails after retries are exhausted,\nCelery invokes this task via .on_error() with the task request, the\nexception, a traceback string, and our explicit video_id argument.\n\nThe handler is idempotent — if an individual step task's except block\nalready marked the analysis as error (the common case), this handler\nis a no-op.\n\"\"\"\n\nimport logging\nfrom datetime import datetime, timezone\nfrom uuid import UUID\n\nfrom app.models.database_models import Video, VideoAnalysis\nfrom app.tasks._pipeline_utils import build_error_json, sanitize_error\nfrom app.tasks.base import DatabaseTask\nfrom app.tasks.celery_app import celery_app\n\nlogger = logging.getLogger(__name__)\n\n\n@celery_app.task(base=DatabaseTask, bind=True, name=\"handle_pipeline_error\")\ndef handle_pipeline_error(self, request, exc, traceback, video_id: str):\n    \"\"\"Chain error handler — marks video + analysis as error, idempotent.\"\"\"\n    try:\n        self.db.rollback()\n\n        video = self.db.query(Video).filter(Video.id == UUID(video_id)).first()\n        analysis = self.db.query(VideoAnalysis).filter(\n            VideoAnalysis.video_id == UUID(video_id)\n        ).first()\n\n        # Determine which step failed from the request context\n        failed_step = \"unknown\"\n        if request is not None:\n            task_name = getattr(request, \"task\", None)\n            if task_name and isinstance(task_name, str):\n                # e.g. \"analyze_infer_step\" → \"infer\"\n                failed_step = task_name.replace(\"analyze_\", \"\").replace(\"_step\", \"\")\n\n        error_json = build_error_json(\n            step=failed_step,\n            exc=exc if isinstance(exc, Exception) else Exception(str(exc)),\n            message=str(exc),\n        )\n\n        # Idempotent update: only set error state if not already set\n        if video and video.status not in (\"error\", \"analyzed\"):\n            video.status = \"error\"\n            video.error_message = error_json\n            logger.info(f\"handle_pipeline_error: marked video {video_id} as error (step={failed_step})\")\n        elif video:\n            logger.info(f\"handle_pipeline_error: video {video_id} already in {video.status}, no-op\")\n\n        if analysis and analysis.status != \"error\":\n            analysis.status = \"error\"\n            analysis.completed_at = datetime.now(timezone.utc)\n\n        self.db.commit()\n\n    except Exception as e:\n        logger.error(f\"handle_pipeline_error itself failed: {sanitize_error(str(e))}\", exc_info=True)\n        try:\n            self.db.rollback()\n        except Exception:\n            pass\n```\n\n- [ ] **Step 4: Run the test to verify it passes**\n\n```bash\ncd backend && pytest tests/test_pipeline_errors.py -v\n```\n\nExpected: PASS.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/app/tasks/pipeline_errors.py backend/tests/test_pipeline_errors.py\ngit commit -m \"feat: add handle_pipeline_error chain error handler\n\nNew task invoked by Celery via chain.on_error() when any step task\nfails after retries. Marks the video + VideoAnalysis rows as error\n(idempotent — no-op if already marked). Extracts the failing step\nname from request.task for structured logging.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.6: Create per-step tasks for the 3-node project analysis\n\n**Files:**\n- Create: `backend/app/tasks/project_analysis_steps.py`\n- Create: `backend/tests/test_project_analysis_chain.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_project_analysis_chain.py`:\n\n```python\n\"\"\"Tests for the Celery chain-based project analysis pipeline.\"\"\"\n\nfrom uuid import uuid4\n\nimport pytest\n\nfrom app.models.database_models import (\n    Project,\n    ProjectAnalysis,\n    Video,\n    VideoAnalysis,\n)\n\n\ndef _seed_project_with_completed_videos(db_session, user_id=\"dev_user_local\"):\n    project = Project(name=\"test\", user_id=user_id, description=\"Research goal\")\n    db_session.add(project)\n    db_session.flush()\n    for i in range(2):\n        video = Video(\n            project_id=project.id,\n            filename=f\"v{i}.mp4\",\n            s3_key=f\"videos/{project.id}/{uuid4()}/v{i}.mp4\",\n            s3_url=f\"https://example/v{i}.mp4\",\n            file_size_bytes=100,\n            status=\"analyzed\",\n        )\n        db_session.add(video)\n        db_session.flush()\n        analysis = VideoAnalysis(\n            video_id=video.id,\n            status=\"completed\",\n            patterns=[{\"id\": f\"P{i}\", \"text\": \"pattern\"}],\n            insights=[{\"id\": f\"I{i}\", \"text\": \"insight\"}],\n            design_principles=[{\"id\": f\"DP{i}\", \"text\": \"principle\"}],\n        )\n        db_session.add(analysis)\n    db_session.commit()\n    return project\n\n\ndef test_cross_relate_step_aggregates_patterns_and_runs_node(db_session, monkeypatch):\n    \"\"\"cross_relate step should read completed video analyses and run cross_relate_node.\"\"\"\n    from app.tasks import project_analysis_steps\n\n    project = _seed_project_with_completed_videos(db_session)\n\n    def fake_cross_relate(state):\n        assert len(state[\"video_patterns\"]) == 2\n        return {\"cross_video_patterns\": [{\"id\": \"CP1\", \"text\": \"cross pattern\"}]}\n\n    monkeypatch.setattr(project_analysis_steps, \"cross_relate_node\", fake_cross_relate)\n    monkeypatch.setattr(project_analysis_steps, \"_resolve_byok\", lambda db, user_id: (None, None))\n\n    task_self = type(\"T\", (), {\"db\": db_session, \"request\": type(\"R\", (), {\"retries\": 0})()})\n    project_analysis_steps.analyze_cross_relate_step.run.__func__(\n        task_self, str(project.id), \"dev_user_local\"\n    )\n\n    pa = db_session.query(ProjectAnalysis).filter_by(project_id=project.id).first()\n    assert pa is not None\n    assert pa.cross_video_patterns == [{\"id\": \"CP1\", \"text\": \"cross pattern\"}]\n    assert pa.status == \"processing\"\n\n\ndef test_cross_activate_step_marks_project_analysis_completed(db_session, monkeypatch):\n    \"\"\"cross_activate (terminal step) should mark status completed with completed_at.\"\"\"\n    from app.tasks import project_analysis_steps\n\n    project = _seed_project_with_completed_videos(db_session)\n    pa = ProjectAnalysis(\n        project_id=project.id,\n        status=\"processing\",\n        video_ids=[],\n        cross_video_patterns=[{\"id\": \"CP1\"}],\n        cross_video_insights=[{\"id\": \"CI1\"}],\n    )\n    db_session.add(pa)\n    db_session.commit()\n\n    def fake_cross_activate(state):\n        return {\"cross_video_principles\": [{\"id\": \"CDP1\", \"text\": \"principle\"}]}\n\n    monkeypatch.setattr(project_analysis_steps, \"cross_activate_node\", fake_cross_activate)\n    monkeypatch.setattr(project_analysis_steps, \"_resolve_byok\", lambda db, user_id: (None, None))\n\n    task_self = type(\"T\", (), {\"db\": db_session, \"request\": type(\"R\", (), {\"retries\": 0})()})\n    project_analysis_steps.analyze_cross_activate_step.run.__func__(\n        task_self, str(project.id), \"dev_user_local\"\n    )\n\n    db_session.refresh(pa)\n    assert pa.status == \"completed\"\n    assert pa.completed_at is not None\n    assert pa.cross_video_principles == [{\"id\": \"CDP1\", \"text\": \"principle\"}]\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_project_analysis_chain.py -v\n```\n\nExpected: FAIL with `ModuleNotFoundError: No module named 'app.tasks.project_analysis_steps'`.\n\n- [ ] **Step 3: Create the project_analysis_steps.py module**\n\nCreate `backend/app/tasks/project_analysis_steps.py`:\n\n```python\n\"\"\"Separate Celery tasks for step-by-step cross-video project analysis.\n\nMirrors backend/app/tasks/analysis_steps.py but for the 3-node project\nanalysis pipeline: cross_relate → cross_explain → cross_activate.\nEach step reads state from the ProjectAnalysis row, runs one node,\nand writes results back.\n\"\"\"\n\nimport logging\nfrom datetime import datetime, timezone\nfrom typing import Any, Dict\nfrom uuid import UUID\n\nfrom sqlalchemy.orm import Session\n\nfrom app.agents.nodes.cross_activate import cross_activate_node\nfrom app.agents.nodes.cross_explain import cross_explain_node\nfrom app.agents.nodes.cross_relate import cross_relate_node\nfrom app.models.database_models import Project, ProjectAnalysis, Video, VideoAnalysis\nfrom app.services.byok_service import resolve_byok as _resolve_byok\nfrom app.tasks.analysis_steps import NonRetryableAnalysisError, _raise_for_node_error\nfrom app.tasks.base import DatabaseTask\nfrom app.tasks.celery_app import celery_app\n\nlogger = logging.getLogger(__name__)\n\n\ndef _check_project_cancellation(db: Session, project_id: str) -> bool:\n    \"\"\"Return True if the project analysis should stop.\"\"\"\n    try:\n        db.expire_all()\n        pa = db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if pa is None or pa.status == \"error\":\n            return True\n        return False\n    except Exception:\n        logger.exception(\"_check_project_cancellation failed, proceeding\")\n        return False\n\n\ndef _get_or_create_project_analysis(db: Session, project_id: UUID) -> ProjectAnalysis:\n    \"\"\"Get existing ProjectAnalysis or create a new one with aggregated state.\"\"\"\n    pa = db.query(ProjectAnalysis).filter(ProjectAnalysis.project_id == project_id).first()\n    if pa is None:\n        # Aggregate from completed video analyses\n        video_analyses = db.query(VideoAnalysis).join(Video).filter(\n            Video.project_id == project_id,\n            VideoAnalysis.status == \"completed\",\n        ).all()\n        if not video_analyses:\n            raise Exception(\"At least one completed video analysis is required\")\n\n        video_ids = [va.video_id for va in video_analyses]\n        pa = ProjectAnalysis(\n            project_id=project_id,\n            video_ids=video_ids,\n            status=\"processing\",\n            started_at=datetime.now(timezone.utc),\n        )\n        db.add(pa)\n        db.commit()\n        db.refresh(pa)\n    return pa\n\n\ndef _update_project_analysis_error(db: Session, project_id: str, step_name: str):\n    \"\"\"Mark ProjectAnalysis as error, safe to call on dirty session.\"\"\"\n    try:\n        db.rollback()\n        pa = db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if pa:\n            pa.status = \"error\"\n            pa.completed_at = datetime.now(timezone.utc)\n        db.commit()\n    except Exception as commit_error:\n        logger.error(f\"Failed to update project error status for {step_name}: {commit_error}\")\n        try:\n            db.rollback()\n        except Exception:\n            pass\n\n\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_cross_relate_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_cross_relate_step(self, project_id: str, user_id: str | None = None):\n    \"\"\"Step 1 of 3: find meta-patterns across videos.\"\"\"\n    try:\n        logger.info(f\"Starting CROSS_RELATE step for project {project_id}\")\n\n        if _check_project_cancellation(self.db, project_id):\n            logger.info(f\"Skipping cross_relate for {project_id} — already in error state\")\n            return {\"project_id\": project_id, \"status\": \"skipped\"}\n\n        pa = _get_or_create_project_analysis(self.db, UUID(project_id))\n\n        # Aggregate patterns from completed videos\n        video_analyses = self.db.query(VideoAnalysis).join(Video).filter(\n            Video.project_id == UUID(project_id),\n            VideoAnalysis.status == \"completed\",\n        ).all()\n\n        all_patterns = []\n        all_insights = []\n        all_principles = []\n        video_ids = []\n        for va in video_analyses:\n            video_ids.append(str(va.video_id))\n            if va.patterns:\n                all_patterns.extend(va.patterns)\n            if va.insights:\n                all_insights.extend(va.insights)\n            if va.design_principles:\n                all_principles.extend(va.design_principles)\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        result = cross_relate_node({\n            \"project_id\": project_id,\n            \"video_ids\": video_ids,\n            \"video_patterns\": all_patterns,\n            \"video_insights\": all_insights,\n            \"video_principles\": all_principles,\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"cross_video_patterns\") is None:\n            _raise_for_node_error(\"cross_relate\", result)\n\n        pa.cross_video_patterns = result.get(\"cross_video_patterns\")\n        pa.status = \"processing\"\n        self.db.commit()\n\n        logger.info(f\"CROSS_RELATE step completed for project {project_id}\")\n        return {\"project_id\": project_id, \"status\": \"success\"}\n\n    except Exception as e:\n        logger.error(f\"CROSS_RELATE step failed for project {project_id}: {e}\")\n        _update_project_analysis_error(self.db, project_id, \"cross_relate\")\n        raise\n\n\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_cross_explain_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_cross_explain_step(self, project_id: str, user_id: str | None = None):\n    \"\"\"Step 2 of 3: synthesize cross-video insights from meta-patterns.\"\"\"\n    try:\n        logger.info(f\"Starting CROSS_EXPLAIN step for project {project_id}\")\n\n        if _check_project_cancellation(self.db, project_id):\n            return {\"project_id\": project_id, \"status\": \"skipped\"}\n\n        pa = self.db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if not pa or not pa.cross_video_patterns:\n            raise Exception(\"No cross-video patterns available for cross_explain\")\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        result = cross_explain_node({\n            \"project_id\": project_id,\n            \"cross_video_patterns\": pa.cross_video_patterns,\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"cross_video_insights\") is None:\n            _raise_for_node_error(\"cross_explain\", result)\n\n        pa.cross_video_insights = result.get(\"cross_video_insights\")\n        self.db.commit()\n\n        logger.info(f\"CROSS_EXPLAIN step completed for project {project_id}\")\n        return {\"project_id\": project_id, \"status\": \"success\"}\n\n    except Exception as e:\n        logger.error(f\"CROSS_EXPLAIN step failed for project {project_id}: {e}\")\n        _update_project_analysis_error(self.db, project_id, \"cross_explain\")\n        raise\n\n\n@celery_app.task(\n    base=DatabaseTask,\n    bind=True,\n    name=\"analyze_cross_activate_step\",\n    autoretry_for=(Exception,),\n    dont_autoretry_for=(NonRetryableAnalysisError,),\n    retry_backoff=True,\n    retry_backoff_max=600,\n    retry_jitter=True,\n    max_retries=3,\n)\ndef analyze_cross_activate_step(self, project_id: str, user_id: str | None = None):\n    \"\"\"Step 3 of 3: derive system-level design principles. Terminal step.\"\"\"\n    try:\n        logger.info(f\"Starting CROSS_ACTIVATE step for project {project_id}\")\n\n        if _check_project_cancellation(self.db, project_id):\n            return {\"project_id\": project_id, \"status\": \"skipped\"}\n\n        pa = self.db.query(ProjectAnalysis).filter(\n            ProjectAnalysis.project_id == UUID(project_id)\n        ).first()\n        if not pa or not pa.cross_video_insights:\n            raise Exception(\"No cross-video insights available for cross_activate\")\n\n        byok_api_key, byok_model = _resolve_byok(self.db, user_id)\n\n        result = cross_activate_node({\n            \"project_id\": project_id,\n            \"cross_video_insights\": pa.cross_video_insights,\n            \"api_key\": byok_api_key,\n            \"model\": byok_model,\n        })\n\n        if result.get(\"error\") or result.get(\"cross_video_principles\") is None:\n            _raise_for_node_error(\"cross_activate\", result)\n\n        pa.cross_video_principles = result.get(\"cross_video_principles\")\n        pa.status = \"completed\"\n        pa.completed_at = datetime.now(timezone.utc)\n        self.db.commit()\n\n        logger.info(f\"CROSS_ACTIVATE step completed for project {project_id}\")\n        return {\"project_id\": project_id, \"status\": \"success\"}\n\n    except Exception as e:\n        logger.error(f\"CROSS_ACTIVATE step failed for project {project_id}: {e}\")\n        _update_project_analysis_error(self.db, project_id, \"cross_activate\")\n        raise\n```\n\n- [ ] **Step 4: Run the test to verify it passes**\n\n```bash\ncd backend && pytest tests/test_project_analysis_chain.py -v\n```\n\nExpected: PASS.\n\n- [ ] **Step 5: Run full suite to make sure nothing regressed**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/app/tasks/project_analysis_steps.py backend/tests/test_project_analysis_chain.py\ngit commit -m \"feat: add per-step project analysis tasks for chain refactor\n\nNew file project_analysis_steps.py mirrors the pattern of\nanalysis_steps.py but for the 3-node cross-video pipeline:\ncross_relate → cross_explain → cross_activate. Each step is an\nindependent Celery task with autoretry, cancellation precheck,\nand DB-backed state transitions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.7: Register new tasks in celery_app.py and add queue routing\n\n**Files:**\n- Modify: `backend/app/tasks/celery_app.py`\n\n- [ ] **Step 1: Update the Celery include list and add task_routes**\n\nEdit `backend/app/tasks/celery_app.py`. Update the `Celery(...)` constructor to include the new task modules, and add `task_routes` + `task_default_queue`:\n\n```python\ncelery_app = Celery(\n    \"qualitative_research_tasks\",\n    broker=settings.CELERY_BROKER_URL,\n    backend=settings.CELERY_RESULT_BACKEND,\n    include=[\n        \"app.tasks.transcription_tasks\",\n        \"app.tasks.analysis_steps\",\n        \"app.tasks.project_analysis_steps\",  # NEW\n        \"app.tasks.pipeline_errors\",          # NEW\n        \"app.tasks.watchdog_tasks\",\n        \"app.tasks.model_validation_tasks\",\n    ],\n)\n\ncelery_app.conf.update(\n    # ... existing config ...\n\n    # Queue routing — analyze tasks on their own queue so they can be\n    # scaled or isolated independently later.\n    task_default_queue=\"celery\",\n    task_routes={\n        \"analyze_chunk_step\":      {\"queue\": \"analyze\"},\n        \"analyze_infer_step\":      {\"queue\": \"analyze\"},\n        \"analyze_relate_step\":     {\"queue\": \"analyze\"},\n        \"analyze_explain_step\":    {\"queue\": \"analyze\"},\n        \"analyze_activate_step\":   {\"queue\": \"analyze\"},\n        \"analyze_cross_relate_step\":   {\"queue\": \"analyze\"},\n        \"analyze_cross_explain_step\":  {\"queue\": \"analyze\"},\n        \"analyze_cross_activate_step\": {\"queue\": \"analyze\"},\n        \"handle_pipeline_error\":       {\"queue\": \"analyze\"},\n        \"transcribe_video\":            {\"queue\": \"transcribe\"},\n        \"check_transcription\":         {\"queue\": \"transcribe\"},\n        # watchdog + model validation use the default \"celery\" queue\n    },\n)\n```\n\n- [ ] **Step 2: Verify Celery can still discover and register tasks**\n\n```bash\ncd backend && python -c \"from app.tasks.celery_app import celery_app; print(sorted(celery_app.tasks.keys()))\" | tr ',' '\\n'\n```\n\nExpected output includes the 5 video step tasks, the 3 cross project step tasks, handle_pipeline_error, transcribe_video, check_transcription, reset_stuck_analyses, validate_openrouter_models.\n\n- [ ] **Step 3: Run the full backend test suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/tasks/celery_app.py\ngit commit -m \"refactor: register new tasks + add queue routing\n\ncelery_app.py now includes project_analysis_steps and pipeline_errors\nmodules. task_routes routes all 8 analysis step tasks and the error\nhandler to the 'analyze' queue, transcription to 'transcribe', and\nwatchdog/model-validation to the default 'celery' queue.\n\nWorker will need to be started with -Q analyze,transcribe,celery\nto consume from all three — that change lands in WS4.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.8: Dispatch chain from the video analyze route\n\n**Files:**\n- Modify: `backend/app/routes/videos.py`\n\n- [ ] **Step 1: Find the existing analyze_video_task.delay dispatch**\n\n```bash\ngrep -n \"analyze_video_task\" backend/app/routes/videos.py\n```\n\n- [ ] **Step 2: Replace with chain dispatch**\n\nIn the `trigger_video_analysis` function (around line 572), replace the monolithic dispatch:\n\n```python\n# OLD:\nfrom app.tasks.analysis_tasks import analyze_video_task\ntask = analyze_video_task.delay(str(video_id), current_user_id)\n```\n\nwith the chain dispatch:\n\n```python\n# NEW:\nfrom celery import chain\nfrom app.tasks.analysis_steps import (\n    analyze_activate_step,\n    analyze_chunk_step,\n    analyze_explain_step,\n    analyze_infer_step,\n    analyze_relate_step,\n)\nfrom app.tasks.pipeline_errors import handle_pipeline_error\n\npipeline = chain(\n    analyze_chunk_step.si(str(video_id), current_user_id),\n    analyze_infer_step.si(str(video_id), current_user_id),\n    analyze_relate_step.si(str(video_id), current_user_id),\n    analyze_explain_step.si(str(video_id), current_user_id),\n    analyze_activate_step.si(str(video_id), current_user_id),\n).on_error(handle_pipeline_error.s(video_id=str(video_id)))\n\ntask = pipeline.apply_async()\n```\n\nKeep the response shape identical (`task_id`, `status`, etc.). Remove the now-unused setup code that was creating the `VideoAnalysis` row or initializing `step_status` — that logic moved into `analyze_chunk_step` in Task 3.4.\n\n- [ ] **Step 3: Run the route tests**\n\n```bash\ncd backend && pytest tests/ -v -k \"analyze or video or route\" 2>&1 | tail -30\n```\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/routes/videos.py\ngit commit -m \"refactor: dispatch Celery chain from video analyze route\n\nPOST /videos/{id}/analyze now dispatches a chain of the 5 per-step\ntasks (chunk → infer → relate → explain → activate) with immutable\nsignatures (.si) so each task receives its own explicit args. The\nerror handler is attached via .on_error().\n\nThe response shape is unchanged so the frontend polls the same\nstatus endpoint and sees the same transitions.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.9: Dispatch chain from the project analyze route\n\n**Files:**\n- Modify: `backend/app/routes/projects.py`\n\n- [ ] **Step 1: Find the existing analyze_project_task.delay dispatch**\n\n```bash\ngrep -n \"analyze_project_task\" backend/app/routes/projects.py\n```\n\n- [ ] **Step 2: Replace with chain dispatch**\n\nSame pattern as Task 3.8:\n\n```python\nfrom celery import chain\nfrom app.tasks.project_analysis_steps import (\n    analyze_cross_activate_step,\n    analyze_cross_explain_step,\n    analyze_cross_relate_step,\n)\nfrom app.tasks.pipeline_errors import handle_pipeline_error\n\npipeline = chain(\n    analyze_cross_relate_step.si(str(project_id), current_user_id),\n    analyze_cross_explain_step.si(str(project_id), current_user_id),\n    analyze_cross_activate_step.si(str(project_id), current_user_id),\n).on_error(handle_pipeline_error.s(video_id=str(project_id)))\n# NOTE: the error handler's video_id parameter is a slight misnomer\n# for projects — leave it for now; it'll just tag the error log with\n# the project_id. Rename to record_id or similar as future cleanup.\n\ntask = pipeline.apply_async()\n```\n\n- [ ] **Step 3: Run tests**\n\n```bash\ncd backend && pytest tests/ -v -k \"project or analyze\" 2>&1 | tail -30\n```\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/app/routes/projects.py\ngit commit -m \"refactor: dispatch Celery chain from project analyze route\n\nPOST /projects/{id}/analyze now dispatches a chain of the 3 per-step\ncross-video tasks (cross_relate → cross_explain → cross_activate).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.10: Delete analysis_tasks.py\n\n**Files:**\n- Delete: `backend/app/tasks/analysis_tasks.py`\n- Modify: any file still importing from it\n\n- [ ] **Step 1: Find remaining imports of analysis_tasks**\n\n```bash\ngrep -rn \"from app.tasks.analysis_tasks\\|from app.tasks import analysis_tasks\" backend/\n```\n\nExpected: should be empty after Tasks 3.8 and 3.9 (routes no longer import it). If any test file still imports from it (likely `test_analysis_retry.py`), update those tests next — see Step 3.\n\n- [ ] **Step 2: Delete the file**\n\n```bash\ngit rm backend/app/tasks/analysis_tasks.py\n```\n\n- [ ] **Step 3: Update any test that still references the deleted module**\n\nFor each failing import from Step 1:\n\n- `test_analysis_retry.py`: if it asserts on `analyze_video_task` retry behavior, rewrite it to assert on `analyze_chunk_step` retry behavior instead (chunk step now owns the retry semantics for the first pipeline link).\n- `test_watchdog_race.py`: replace references to `_is_cancelled` with the new `_check_cancellation` helper.\n\n- [ ] **Step 4: Run the full test suite**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass. If tests still fail because of functional gaps (not mere naming), file a finding in the commit message and adjust — don't paper over with test skips.\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add -A\ngit commit -m \"refactor: delete monolithic analysis_tasks.py\n\nThe monolithic analyze_video_task and analyze_project_task are\nreplaced by Celery chains of per-step tasks in analysis_steps.py\nand project_analysis_steps.py. The in-function retry/cancellation\nlogic is replaced by per-task autoretry + _check_cancellation\nprecheck. Helpers moved to _pipeline_utils.py.\n\n-673 / +0 lines net.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 3.11: Full integration smoke test on WS3\n\n**Files:**\n- Read-only\n\n- [ ] **Step 1: Run the complete test suite one more time**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -40\n```\n\nExpected: all tests pass, with the same count as the pre-flight baseline (minus any tests that were intentionally removed because they tested the deleted monolithic task).\n\n- [ ] **Step 2: Verify celery_app discovers all expected tasks**\n\n```bash\npython -c \"from app.tasks.celery_app import celery_app; import json; print(json.dumps(sorted(celery_app.tasks.keys()), indent=2))\"\n```\n\nExpected tasks in the output:\n- analyze_chunk_step, analyze_infer_step, analyze_relate_step, analyze_explain_step, analyze_activate_step\n- analyze_cross_relate_step, analyze_cross_explain_step, analyze_cross_activate_step\n- handle_pipeline_error\n- transcribe_video, check_transcription\n- reset_stuck_analyses, validate_openrouter_models\n\nAnd NOT in the output: analyze_video, analyze_project (the deleted monolithic tasks).\n\n- [ ] **Step 3: WS3 handoff**\n\n```bash\ngit log --oneline origin/main..HEAD\n```\n\nExpected: ~10 commits (one per task in this workstream).\n\nWS3 is complete. Ready for PR review and merge. **WS2 must rebase on WS3 after it merges.**\n\n---\n\n# WORKSTREAM 4 (WS4) — T2+T5 Infrastructure, Multi-Replica, Redis Hardening\n\n**Worktree:** a fresh worktree created AFTER Wave 1 merges — do not start until WS1/WS2/WS3 are all merged to origin/main and have soaked for ≥24h.\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\ngit fetch origin\ngit worktree add -b infra/multi-replica-topology ../5d-worktrees/ws4-infra origin/main\ncd ../5d-worktrees/ws4-infra\n```\n\n**Goal:** Extract Celery Beat into its own Railway service. Scale backend and worker to `numReplicas=2`. Add `/health/ready` readiness endpoint. Resize DB pool via environment-aware config. Apply Redis maxmemory + eviction policy. Remove worker's unused public domain.\n\n**Blocking?** Yes — must land after Wave 1 and before WS5 validation.\n\n## Task 4.1: Add service-type-aware DB pool config\n\n**Files:**\n- Modify: `backend/app/database.py`\n- Create: `backend/tests/test_database_pool_config.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_database_pool_config.py`:\n\n```python\n\"\"\"Tests for service-type-aware database pool sizing.\"\"\"\n\nimport importlib\nimport os\n\nimport pytest\n\n\n@pytest.mark.parametrize(\"service_type,expected_pool_size,expected_overflow\", [\n    (\"api\", 5, 5),\n    (\"worker\", 4, 4),\n    (\"beat\", 1, 1),\n])\ndef test_pool_size_by_service_type(service_type, expected_pool_size, expected_overflow, monkeypatch):\n    \"\"\"Pool size should vary by SERVICE_TYPE to avoid Postgres connection exhaustion.\"\"\"\n    monkeypatch.setenv(\"SERVICE_TYPE\", service_type)\n    # Re-import database module so it picks up the new SERVICE_TYPE\n    import app.database\n    importlib.reload(app.database)\n    # The engine's pool carries the size\n    assert app.database.engine.pool.size() == expected_pool_size or \\\n        app.database._pool[\"pool_size\"] == expected_pool_size\n    assert app.database._pool[\"max_overflow\"] == expected_overflow\n\n\ndef test_unknown_service_type_falls_back_to_api_pool(monkeypatch):\n    \"\"\"Unknown SERVICE_TYPE should not crash; fall back to api pool sizing.\"\"\"\n    monkeypatch.setenv(\"SERVICE_TYPE\", \"something-weird\")\n    import app.database\n    importlib.reload(app.database)\n    assert app.database._pool[\"pool_size\"] == 5\n    assert app.database._pool[\"max_overflow\"] == 5\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_database_pool_config.py -v\n```\n\nExpected: FAIL (because `_pool` doesn't exist yet in `database.py`).\n\n- [ ] **Step 3: Update database.py to read SERVICE_TYPE**\n\nEdit `backend/app/database.py`:\n\n```python\n\"\"\"Database connection and session management.\"\"\"\n\nimport os\nfrom typing import Generator\n\nfrom sqlalchemy import create_engine\nfrom sqlalchemy.orm import DeclarativeBase, Session, sessionmaker\n\nfrom app.config import settings\n\n# Build connect_args based on database backend\n_connect_args: dict = {}\nif \"postgresql\" in settings.DATABASE_URL:\n    _connect_args[\"options\"] = \"-c statement_timeout=30000\"  # 30s query timeout\n\n# Service-type-aware pool sizing. Picked so total connection count\n# across all replicas stays comfortably under Postgres max_connections=100.\n#\n# backend: 5 + 5 overflow per uvicorn worker × 2 workers × 2 replicas = 40 max\n# worker:  4 + 4 overflow per replica × 2 replicas = 16 max\n# beat:    1 + 1 overflow × 1 replica = 2 max\n# Total worst case: ~58 connections. Plenty of headroom under 100.\n_SERVICE_TYPE = os.environ.get(\"SERVICE_TYPE\", \"api\")\n_POOL_CONFIG = {\n    \"api\":    {\"pool_size\": 5, \"max_overflow\": 5},\n    \"worker\": {\"pool_size\": 4, \"max_overflow\": 4},\n    \"beat\":   {\"pool_size\": 1, \"max_overflow\": 1},\n}\n_pool = _POOL_CONFIG.get(_SERVICE_TYPE, _POOL_CONFIG[\"api\"])\n\n# Create database engine\nengine = create_engine(\n    settings.DATABASE_URL,\n    connect_args=_connect_args,\n    pool_pre_ping=True,  # Verify connections before using them\n    pool_size=_pool[\"pool_size\"],\n    max_overflow=_pool[\"max_overflow\"],\n    pool_recycle=1800,  # Recycle connections after 30 minutes\n    echo=settings.DEBUG,\n)\n\n# Create session factory\nSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)\n\n\n# Base class for ORM models (modern SQLAlchemy 2.0 style)\nclass Base(DeclarativeBase):\n    pass\n\n\ndef get_db() -> Generator[Session, None, None]:\n    \"\"\"Dependency for FastAPI routes to get database session.\"\"\"\n    db = SessionLocal()\n    try:\n        yield db\n    finally:\n        db.close()\n```\n\n- [ ] **Step 4: Run the test to verify it passes**\n\n```bash\ncd backend && pytest tests/test_database_pool_config.py -v\n```\n\nExpected: all tests pass.\n\n- [ ] **Step 5: Run the full suite to make sure nothing regressed**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass.\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/app/database.py backend/tests/test_database_pool_config.py\ngit commit -m \"feat: service-type-aware DB pool sizing\n\nReads SERVICE_TYPE env var and picks a pool size that keeps total\nconcurrent connections under Postgres max_connections=100 across\nthe new 2-replica backend + 2-replica worker + 1-replica beat\ntopology. Total worst-case usage: ~58 connections.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.2: Add /health/live and /health/ready endpoints\n\n**Files:**\n- Modify: `backend/app/main.py`\n- Create: `backend/tests/test_healthcheck_ready.py`\n\n- [ ] **Step 1: Write the failing test**\n\nCreate `backend/tests/test_healthcheck_ready.py`:\n\n```python\n\"\"\"Tests for the /health/live and /health/ready endpoints.\"\"\"\n\nfrom unittest.mock import patch\n\nimport pytest\nfrom fastapi.testclient import TestClient\n\n\n@pytest.fixture\ndef client():\n    from app.main import app\n    return TestClient(app)\n\n\ndef test_health_live_returns_200(client):\n    \"\"\"Liveness check is always 200 as long as the process is running.\"\"\"\n    resp = client.get(\"/health/live\")\n    assert resp.status_code == 200\n    assert resp.json() == {\"status\": \"alive\"}\n\n\ndef test_health_returns_200_backwards_compat(client):\n    \"\"\"The old /health endpoint must still work for existing monitoring.\"\"\"\n    resp = client.get(\"/health\")\n    assert resp.status_code == 200\n\n\ndef test_health_ready_returns_200_when_db_and_redis_ok(client):\n    \"\"\"Readiness check returns 200 when both DB and Redis are reachable.\"\"\"\n    resp = client.get(\"/health/ready\")\n    assert resp.status_code == 200\n    assert resp.json() == {\"status\": \"ready\"}\n\n\ndef test_health_ready_returns_503_when_db_fails(client):\n    \"\"\"Readiness check returns 503 when DB ping fails.\"\"\"\n    with patch(\"app.main._probe_db\", side_effect=Exception(\"db down\")):\n        resp = client.get(\"/health/ready\")\n        assert resp.status_code == 503\n        assert resp.json()[\"status\"] == \"db_down\"\n\n\ndef test_health_ready_returns_503_when_redis_fails(client):\n    \"\"\"Readiness check returns 503 when Redis ping fails.\"\"\"\n    with patch(\"app.main._probe_redis\", side_effect=Exception(\"redis down\")):\n        resp = client.get(\"/health/ready\")\n        assert resp.status_code == 503\n        assert resp.json()[\"status\"] == \"redis_down\"\n```\n\n- [ ] **Step 2: Run the failing test**\n\n```bash\ncd backend && pytest tests/test_healthcheck_ready.py -v\n```\n\nExpected: FAIL (endpoints don't exist yet).\n\n- [ ] **Step 3: Add the endpoints and probe helpers to main.py**\n\nIn `backend/app/main.py`, replace the existing `/health` endpoint with:\n\n```python\nfrom sqlalchemy import text\nfrom fastapi.responses import JSONResponse\n\n\ndef _probe_db() -> None:\n    \"\"\"Raise if the DB is not reachable. Used by /health/ready.\"\"\"\n    from app.database import SessionLocal\n    with SessionLocal() as db:\n        db.execute(text(\"SELECT 1\"))\n\n\ndef _probe_redis() -> None:\n    \"\"\"Raise if Redis (Celery broker) is not reachable. Used by /health/ready.\"\"\"\n    from app.tasks.celery_app import celery_app\n    with celery_app.broker_connection() as conn:\n        conn.ensure_connection(max_retries=1)\n\n\n@app.get(\"/\")\nasync def root():\n    \"\"\"Root endpoint.\"\"\"\n    return {\"status\": \"ok\"}\n\n\n@app.get(\"/health\")\nasync def health_check():\n    \"\"\"Backwards-compatible liveness alias.\"\"\"\n    return {\"status\": \"healthy\"}\n\n\n@app.get(\"/health/live\")\nasync def health_live():\n    \"\"\"Liveness — is the process running?\"\"\"\n    return {\"status\": \"alive\"}\n\n\n@app.get(\"/health/ready\")\nasync def health_ready():\n    \"\"\"Readiness — can this replica serve traffic (DB + Redis OK)?\"\"\"\n    try:\n        _probe_db()\n    except Exception:\n        return JSONResponse(status_code=503, content={\"status\": \"db_down\"})\n    try:\n        _probe_redis()\n    except Exception:\n        return JSONResponse(status_code=503, content={\"status\": \"redis_down\"})\n    return {\"status\": \"ready\"}\n```\n\n- [ ] **Step 4: Run the tests**\n\n```bash\ncd backend && pytest tests/test_healthcheck_ready.py -v\n```\n\nExpected: all tests pass. If the `_probe_redis` mock doesn't fire because the module re-imports it, use `monkeypatch` on the specific `app.main._probe_redis` attribute instead.\n\n- [ ] **Step 5: Run the full suite**\n\n```bash\npytest tests/ -v 2>&1 | tail -30\n```\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/app/main.py backend/tests/test_healthcheck_ready.py\ngit commit -m \"feat: add /health/live and /health/ready endpoints\n\nSplits the single /health endpoint into liveness (always 200 as long\nas the process runs) and readiness (verifies DB + Redis connectivity).\nRailway will point healthcheckPath at /health/ready so broken replicas\nget pulled from load balancer rotation automatically.\n\nThe old /health stays as an alias for backwards compat with existing\nmonitoring.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.3: Update startup.sh for beat service + multi-worker uvicorn + worker queues\n\n**Files:**\n- Modify: `backend/scripts/startup.sh`\n\n- [ ] **Step 1: Add beat service branch, --workers to uvicorn, -Q to worker**\n\nEdit `backend/scripts/startup.sh`. Replace the final `if/else` block that starts the service with:\n\n```sh\n# Start the application\nif [ \"$SERVICE\" = \"worker\" ]; then\n    echo \"🔨 Starting Celery worker...\"\n    # --pool=threads: I/O-bound (LLM, AssemblyAI, R2)\n    # --concurrency: CELERY_CONCURRENCY env var (default 16 post-scaling)\n    # -Q analyze,transcribe,celery: consume from all three queues\n    exec celery -A app.tasks.celery_app worker \\\n        --pool=threads \\\n        --concurrency=${CELERY_CONCURRENCY:-16} \\\n        --loglevel=info \\\n        --without-heartbeat \\\n        --without-mingle \\\n        --without-gossip \\\n        -Q analyze,transcribe,celery\nelif [ \"$SERVICE\" = \"beat\" ]; then\n    echo \"⏰ Starting Celery beat scheduler...\"\n    # Beat service runs ONLY the scheduler. Hard-pinned to 1 replica.\n    # Periodic tasks fire exactly once per schedule regardless of\n    # how many worker replicas are running.\n    exec celery -A app.tasks.celery_app beat --loglevel=info\nelse\n    echo \"🌐 Starting API server...\"\n    if [ \"$IS_PRODUCTION\" = true ]; then\n        # Production: 2 uvicorn workers per replica, no reload\n        exec uvicorn app.main:app \\\n            --host 0.0.0.0 \\\n            --port ${PORT:-8000} \\\n            --workers 2 \\\n            --proxy-headers \\\n            --forwarded-allow-ips='*'\n    else\n        # Development: with reload\n        exec uvicorn app.main:app --host 0.0.0.0 --port ${PORT:-8000} --reload\n    fi\nfi\n```\n\nNote that the worker branch no longer has `--beat`. That's intentional — Beat moves to its own dedicated service.\n\n- [ ] **Step 2: Update the Redis-wait block so it also runs for the beat service**\n\nEarlier in the same file, find the block that waits for Redis only when `SERVICE = worker`. Update it to also wait for Redis when `SERVICE = beat`:\n\n```sh\n# Wait for Redis to be ready (required for Celery broker, beat scheduler)\nif [ \"$SERVICE\" = \"worker\" ] || [ \"$SERVICE\" = \"beat\" ]; then\n    echo \"⏳ Waiting for Redis...\"\n    # ... existing Redis-wait code unchanged ...\nfi\n```\n\n- [ ] **Step 3: Shellcheck the script**\n\n```bash\nshellcheck backend/scripts/startup.sh 2>&1 | head -20\n```\n\nIf shellcheck isn't installed, skip — the script is a straightforward bash file.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add backend/scripts/startup.sh\ngit commit -m \"feat: update startup.sh for beat service + multi-worker uvicorn\n\n- New 'beat' service branch runs celery beat scheduler only (no worker)\n- Worker branch loses --beat (moved to dedicated service) and gains\n  -Q analyze,transcribe,celery so it consumes from all queues\n- uvicorn gets --workers 2 in production for within-replica concurrency\n- Redis wait block now also runs for the beat service\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.4: Create Railway service configurator script\n\n**Files:**\n- Create: `scripts/railway-service-config.py`\n\n- [ ] **Step 1: Create the script**\n\nCreate `scripts/railway-service-config.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"Idempotent Railway service configuration for the methodex project.\n\nApplies the target topology (replicas, healthchecks, drain settings) to\neach service via the Railway GraphQL API. Safe to re-run — Railway's\nmutation is idempotent and will no-op on unchanged fields.\n\nAlso creates the new 'beat' service if it doesn't yet exist, and removes\nthe worker's unused public domain.\n\nUsage:\n    export RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\n    python3 scripts/railway-service-config.py --dry-run   # show what would change\n    python3 scripts/railway-service-config.py             # apply\n\nExit codes:\n    0 — success (or dry-run completed)\n    1 — any mutation or validation failure\n    2 — missing env vars\n\"\"\"\n\nimport argparse\nimport json\nimport os\nimport sys\nimport urllib.error\nimport urllib.request\n\nRAILWAY_API = \"https://backboard.railway.app/graphql/v2\"\n\nPROJECT_ID = \"154d302f-8609-4897-a10c-1f0d5bfc4f06\"\nWORKSPACE_ID = \"37eb4e96-0873-4033-a392-3c6593a68802\"\n\n# Target topology — edit here to change replica/health settings\nTARGET = {\n    \"backend\": {\n        \"id\": \"2b70a900-042c-4083-b00b-0d01f3ece5dc\",\n        \"numReplicas\": 2,\n        \"healthcheckPath\": \"/health/ready\",\n        \"healthcheckTimeout\": 10,\n        \"drainingSeconds\": 30,\n    },\n    \"worker\": {\n        \"id\": \"08097b12-1501-4dff-a990-edcd95c73ed4\",\n        \"numReplicas\": 2,\n        \"healthcheckPath\": None,   # workers have no HTTP endpoint\n        \"drainingSeconds\": 60,\n    },\n    \"beat\": {\n        # id populated at runtime if the service already exists,\n        # otherwise created during this script\n        \"id\": None,\n        \"numReplicas\": 1,\n        \"healthcheckPath\": None,\n        \"drainingSeconds\": 10,\n        \"createIfMissing\": True,\n    },\n}\n\n\ndef gql(query: str, variables: dict, token: str) -> dict:\n    body = json.dumps({\"query\": query, \"variables\": variables}).encode(\"utf-8\")\n    req = urllib.request.Request(\n        RAILWAY_API,\n        data=body,\n        headers={\n            \"Content-Type\": \"application/json\",\n            \"Authorization\": f\"Bearer {token}\",\n        },\n        method=\"POST\",\n    )\n    with urllib.request.urlopen(req, timeout=30) as resp:\n        data = json.loads(resp.read().decode(\"utf-8\"))\n    if \"errors\" in data:\n        raise RuntimeError(f\"GraphQL errors: {data['errors']}\")\n    return data[\"data\"]\n\n\ndef list_services(token: str) -> list[dict]:\n    query = \"\"\"\n    query ListServices($projectId: String!) {\n        project(id: $projectId) {\n            services { edges { node { id name } } }\n        }\n    }\n    \"\"\"\n    data = gql(query, {\"projectId\": PROJECT_ID}, token)\n    return [e[\"node\"] for e in data[\"project\"][\"services\"][\"edges\"]]\n\n\ndef update_service_instance(token: str, service_id: str, env_id: str, updates: dict, dry_run: bool) -> None:\n    \"\"\"Mutate serviceInstanceUpdate with the given updates.\"\"\"\n    if dry_run:\n        print(f\"  [DRY] would update service {service_id}: {updates}\")\n        return\n\n    mutation = \"\"\"\n    mutation UpdateInstance($serviceId: String!, $environmentId: String!, $input: ServiceInstanceUpdateInput!) {\n        serviceInstanceUpdate(serviceId: $serviceId, environmentId: $environmentId, input: $input)\n    }\n    \"\"\"\n    data = gql(mutation, {\n        \"serviceId\": service_id,\n        \"environmentId\": env_id,\n        \"input\": updates,\n    }, token)\n    print(f\"  applied to {service_id}: {list(updates.keys())}\")\n\n\ndef get_production_environment_id(token: str) -> str:\n    \"\"\"Return the production environment id for our project.\"\"\"\n    query = \"\"\"\n    query Envs($projectId: String!) {\n        project(id: $projectId) {\n            environments { edges { node { id name } } }\n        }\n    }\n    \"\"\"\n    data = gql(query, {\"projectId\": PROJECT_ID}, token)\n    for e in data[\"project\"][\"environments\"][\"edges\"]:\n        if e[\"node\"][\"name\"] == \"production\":\n            return e[\"node\"][\"id\"]\n    raise RuntimeError(\"production environment not found\")\n\n\ndef ensure_beat_service_exists(token: str, existing_services: list[dict], env_id: str, dry_run: bool) -> str | None:\n    \"\"\"Create the 'beat' service if it doesn't exist. Return its id.\"\"\"\n    for svc in existing_services:\n        if svc[\"name\"] == \"beat\":\n            print(f\"  beat service already exists: {svc['id']}\")\n            return svc[\"id\"]\n\n    if dry_run:\n        print(\"  [DRY] would create 'beat' service\")\n        return None\n\n    # Create via serviceCreate — clones the worker source (same repo + dockerfile)\n    mutation = \"\"\"\n    mutation CreateBeat($projectId: String!, $name: String!) {\n        serviceCreate(input: { projectId: $projectId, name: $name }) {\n            id name\n        }\n    }\n    \"\"\"\n    data = gql(mutation, {\"projectId\": PROJECT_ID, \"name\": \"beat\"}, token)\n    beat_id = data[\"serviceCreate\"][\"id\"]\n    print(f\"  created beat service: {beat_id}\")\n\n    # Set SERVICE_TYPE=beat and copy source config from worker\n    # NOTE: Railway requires setting the source explicitly after creation.\n    # The simplest route is to have the user configure it from dashboard on\n    # first run, OR use the variableUpsert mutation to set SERVICE_TYPE.\n    variable_mutation = \"\"\"\n    mutation SetVar($input: VariableUpsertInput!) {\n        variableUpsert(input: $input)\n    }\n    \"\"\"\n    gql(variable_mutation, {\n        \"input\": {\n            \"projectId\": PROJECT_ID,\n            \"environmentId\": env_id,\n            \"serviceId\": beat_id,\n            \"name\": \"SERVICE_TYPE\",\n            \"value\": \"beat\",\n        },\n    }, token)\n    print(f\"  set SERVICE_TYPE=beat on {beat_id}\")\n\n    return beat_id\n\n\ndef remove_worker_public_domain(token: str, dry_run: bool) -> None:\n    \"\"\"Delete the worker's pointless public domain.\"\"\"\n    query = \"\"\"\n    query WorkerDomains($serviceId: String!) {\n        service(id: $serviceId) {\n            serviceInstances { edges { node { domains { serviceDomains { id domain } } } } }\n        }\n    }\n    \"\"\"\n    data = gql(query, {\"serviceId\": TARGET[\"worker\"][\"id\"]}, token)\n    try:\n        domains = data[\"service\"][\"serviceInstances\"][\"edges\"][0][\"node\"][\"domains\"][\"serviceDomains\"]\n    except (KeyError, IndexError):\n        domains = []\n    if not domains:\n        print(\"  no worker service domains to remove\")\n        return\n\n    mutation = \"\"\"\n    mutation DeleteDomain($id: String!) {\n        serviceDomainDelete(id: $id)\n    }\n    \"\"\"\n    for d in domains:\n        if dry_run:\n            print(f\"  [DRY] would delete worker service domain {d['domain']} ({d['id']})\")\n        else:\n            gql(mutation, {\"id\": d[\"id\"]}, token)\n            print(f\"  deleted worker service domain {d['domain']}\")\n\n\ndef main():\n    parser = argparse.ArgumentParser()\n    parser.add_argument(\"--dry-run\", action=\"store_true\", help=\"Show changes without applying\")\n    args = parser.parse_args()\n\n    token = os.environ.get(\"RAILWAY_API_TOKEN\")\n    if not token:\n        print(\"RAILWAY_API_TOKEN env var required\", file=sys.stderr)\n        return 2\n\n    env_id = get_production_environment_id(token)\n    services = list_services(token)\n    print(f\"Found services: {[s['name'] for s in services]}\")\n\n    # 1. Ensure beat service exists\n    beat_id = ensure_beat_service_exists(token, services, env_id, args.dry_run)\n    if beat_id:\n        TARGET[\"beat\"][\"id\"] = beat_id\n\n    # 2. Apply topology to each known service\n    for name, cfg in TARGET.items():\n        if cfg[\"id\"] is None:\n            print(f\"Skipping {name} — no service id (was it created above?)\")\n            continue\n        updates = {}\n        if \"numReplicas\" in cfg:\n            updates[\"numReplicas\"] = cfg[\"numReplicas\"]\n        if cfg.get(\"healthcheckPath\") is not None:\n            updates[\"healthcheckPath\"] = cfg[\"healthcheckPath\"]\n        if \"healthcheckTimeout\" in cfg:\n            updates[\"healthcheckTimeout\"] = cfg[\"healthcheckTimeout\"]\n        if \"drainingSeconds\" in cfg:\n            updates[\"drainingSeconds\"] = cfg[\"drainingSeconds\"]\n\n        if updates:\n            print(f\"Updating {name} ({cfg['id']}):\")\n            update_service_instance(token, cfg[\"id\"], env_id, updates, args.dry_run)\n\n    # 3. Remove worker's public domain\n    print(\"Removing worker public domain:\")\n    remove_worker_public_domain(token, args.dry_run)\n\n    print(\"\\nDone.\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make executable and run dry-run**\n\n```bash\nchmod +x scripts/railway-service-config.py\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\npython3 scripts/railway-service-config.py --dry-run\n```\n\nExpected: prints the changes that would be applied without mutating anything. Review the output carefully.\n\n- [ ] **Step 3: Commit (do NOT run live yet — that happens in Task 4.7 after all code is merged)**\n\n```bash\ngit add scripts/railway-service-config.py\ngit commit -m \"feat: add Railway service topology configurator script\n\nIdempotent Python script that applies the target topology (replicas,\nhealthchecks, drain settings) to each Railway service via the GraphQL\nAPI. Also creates the 'beat' service if missing and removes the worker\nservice's pointless public domain.\n\nDry-run supported via --dry-run. Do NOT apply live until Wave 2 code\nchanges are merged to main.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.5: Apply Redis maxmemory + eviction policy via Railway\n\n**Files:**\n- No code change; Railway service config update\n\n- [ ] **Step 1: Fetch the current Redis startCommand**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ project(id: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\") { services { edges { node { id name serviceInstances { edges { node { startCommand } } } } } } } }\"}' \\\n  | python3 -m json.tool | grep -A1 Redis\n```\n\nExpected: shows the current Redis startCommand (should include `--requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH`).\n\n- [ ] **Step 2: Update the startCommand to add maxmemory + eviction policy**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\n\n# Get production environment id first\nENV_ID=$(curl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ project(id: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\") { environments { edges { node { id name } } } } }\"}' \\\n  | python3 -c \"import json,sys; d=json.load(sys.stdin); print([e['node']['id'] for e in d['data']['project']['environments']['edges'] if e['node']['name']=='production'][0])\")\n\n# Update Redis startCommand\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d \"{\\\"query\\\":\\\"mutation UpdateRedis(\\$input: ServiceInstanceUpdateInput!) { serviceInstanceUpdate(serviceId: \\\\\\\"53904bf4-fcad-426f-aec6-d94de78c3999\\\\\\\", environmentId: \\\\\\\"$ENV_ID\\\\\\\", input: \\$input) }\\\",\\\"variables\\\":{\\\"input\\\":{\\\"startCommand\\\":\\\"/bin/sh -c 'rm -rf \\$RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass \\$REDIS_PASSWORD --save 60 1 --dir \\$RAILWAY_VOLUME_MOUNT_PATH --maxmemory 300mb --maxmemory-policy allkeys-lru'\\\"}}}\" \\\n  | python3 -m json.tool\n```\n\nExpected: response is `{\"data\": {\"serviceInstanceUpdate\": null}}` indicating success. Railway will redeploy Redis with the new startCommand automatically.\n\n- [ ] **Step 2.5: Defer this step — do not run live during WS4**\n\n**Important**: this step should NOT be executed during the refactor PR. Apply it alongside the topology script in Task 4.7 after the Wave 2 PR merges. The command here is documentation for the operator. Add it to the WS4 PR description as a post-merge action item.\n\n- [ ] **Step 3: Document the change in a runbook**\n\nCreate `docs/ops/redis-config.md`:\n\n```markdown\n# Redis service configuration (methodex)\n\n## Current production startCommand\n\n```\n/bin/sh -c 'rm -rf $RAILWAY_VOLUME_MOUNT_PATH/lost+found/ && exec docker-entrypoint.sh redis-server --requirepass $REDIS_PASSWORD --save 60 1 --dir $RAILWAY_VOLUME_MOUNT_PATH --maxmemory 300mb --maxmemory-policy allkeys-lru'\n```\n\n## Why maxmemory 300mb + allkeys-lru\n\n- Celery results expire in 10 minutes (`result_expires=600` in celery_app.py)\n- At target B load, we hold roughly 25 concurrent analysis results × a few KB = <1 MB of working data\n- 300 MB is ~300× headroom, and `allkeys-lru` ensures we never OOM — the oldest Celery result gets evicted first\n- Before this was set, Redis had no memory bound and could OOM-kill the broker silently under a pathological load\n\n## To change the limit\n\nEdit `scripts/railway-service-config.py` if adding to the automated script, or run the `serviceInstanceUpdate` mutation documented in `docs/superpowers/plans/2026-04-06-methodex-hobby-scaling-plan.md` Task 4.5.\n```\n\n- [ ] **Step 4: Commit the runbook**\n\n```bash\ngit add docs/ops/redis-config.md\ngit commit -m \"docs: document Redis startCommand + maxmemory rationale\n\nRunbook for the Redis service configuration. Explains why we pick\n300mb + allkeys-lru given Celery's result_expires=600s setting, and\nhow to change the limit.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 4.6: Run full local + CI verification\n\n**Files:**\n- Read-only\n\n- [ ] **Step 1: Run the full backend test suite in the WS4 worktree**\n\n```bash\ncd backend && pytest tests/ -v 2>&1 | tail -30\n```\n\nExpected: all tests pass, including the 3 new test files from WS4 (`test_database_pool_config.py`, `test_healthcheck_ready.py`, and any WS3 tests that carried over from the rebase).\n\n- [ ] **Step 2: Verify the startup script is shell-syntax valid**\n\n```bash\nbash -n backend/scripts/startup.sh && echo OK\n```\n\nExpected: `OK`.\n\n- [ ] **Step 3: Dry-run the Railway configurator once more**\n\n```bash\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\npython3 scripts/railway-service-config.py --dry-run\n```\n\nReview output. If the dry-run reports any changes you don't expect, investigate before merging.\n\n## Task 4.7: WS4 merge + live infrastructure apply (carefully staged)\n\n**Files:**\n- Post-merge runbook\n\n- [ ] **Step 1: Merge WS4 to main**\n\nOpen PR, get review, merge. This triggers a backend + worker redeploy with the new code (uvicorn `--workers 2`, `-Q` queue routing, new `/health/ready` endpoint, service-type-aware DB pool). **No replica or topology changes yet** — Railway's service config is not code, so just merging the code doesn't change replicas.\n\n- [ ] **Step 2: Verify merged deploy is healthy**\n\n```bash\ncurl -f https://api.methodex.ai/health/live && echo\ncurl -f https://api.methodex.ai/health/ready && echo\n```\n\nExpected: both return 200. If `/health/ready` returns 503, investigate DB/Redis connectivity before proceeding.\n\n- [ ] **Step 3: Apply the Railway topology changes via script**\n\n```bash\ncd /Users/idstuart/Projects/ai-prototyping/5d-analysis/qualitative-research-tool\nexport RAILWAY_API_TOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\npython3 scripts/railway-service-config.py --dry-run   # final review\npython3 scripts/railway-service-config.py             # apply\n```\n\nExpected: creates the `beat` service, sets `numReplicas=2` on backend + worker, sets `healthcheckPath=/health/ready` on backend, sets drain timings, removes worker public domain.\n\n- [ ] **Step 4: Configure the new beat service source (manual, one-time)**\n\nRailway's GraphQL API can create a service but the source (repo + Dockerfile path) must be set through the dashboard on the first run. In the Railway dashboard:\n\n1. Open the new `beat` service\n2. Set Source → GitHub repo `anuragid/qualitative-research-tool`, branch `main`\n3. Set Build → Dockerfile path `backend/Dockerfile.railway`\n4. Verify env vars include `SERVICE_TYPE=beat` (set by the script) plus all the shared env vars (DATABASE_URL, REDIS_URL, etc.) — copy from the worker service if they're not auto-inherited\n5. Trigger an initial deploy\n\nOnce this first manual setup is done, subsequent config changes flow through the committed script.\n\n- [ ] **Step 5: Apply the Redis startCommand change**\n\nRun the command from Task 4.5 Step 2 (the curl + GraphQL mutation).\n\nVerify:\n\n```bash\n# SSH into Redis (via Railway CLI) or check Railway logs to confirm the\n# container restarted with the new startCommand.\n# Alternatively, if Redis responds to AUTH:\nredis-cli -h yamabiko.proxy.rlwy.net -p 50187 -a $REDIS_PASSWORD INFO memory | grep maxmemory\n```\n\nExpected: `maxmemory:314572800` (300 MB in bytes).\n\n- [ ] **Step 6: Verify the new topology is live**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ project(id: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\") { services { edges { node { id name serviceInstances { edges { node { numReplicas healthcheckPath drainingSeconds } } } } } } } }\"}' \\\n  | python3 -m json.tool\n```\n\nExpected output shows:\n- backend: `numReplicas=2, healthcheckPath=/health/ready, drainingSeconds=30`\n- worker: `numReplicas=2, drainingSeconds=60`\n- beat: `numReplicas=1, drainingSeconds=10`\n- Postgres, Redis: unchanged\n\n- [ ] **Step 7: Verify beat is firing exactly once**\n\n```bash\nrailway service beat && railway logs 2>&1 | head -50\n```\n\nExpected: log entries showing `reset_stuck_analyses` tasks dispatched approximately every 5 minutes (not every 2.5 min — that would indicate multiple beat instances).\n\n- [ ] **Step 8: Verify worker no longer runs beat**\n\n```bash\nrailway service worker && railway logs 2>&1 | head -50\n```\n\nExpected: worker logs show task execution but NO beat-scheduler entries (those now only appear in the beat service's logs).\n\n- [ ] **Step 9: Smoke-test the API end-to-end**\n\n```bash\ncurl -f https://api.methodex.ai/health/ready\ncurl -f https://api.methodex.ai/health/live\n```\n\nExpected: both return 200 from both backend replicas (verify by hitting multiple times and watching for different replica IDs in Railway logs).\n\nWS4 is complete. Wave 2 is live in production.\n\n---\n\n# WORKSTREAM 5 (WS5) — Production Validation\n\n**Worktree:** main checkout is fine — this is scripts + memory updates, no code refactor.\n\n**Goal:** Run smoke and burst load tests, confirm memory footprint dropped, update progress memory, close the loop.\n\n**Blocking?** Runs after WS4 is live.\n\n## Task 5.1: Create production smoke test script\n\n**Files:**\n- Create: `scripts/production-smoke-test.py`\n\n- [ ] **Step 1: Create the script**\n\nCreate `scripts/production-smoke-test.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"End-to-end production smoke test for the methodex analysis pipeline.\n\nRequires:\n- A test user with a pre-seeded project containing at least one video\n  that already has a completed transcript\n- METHODEX_API_URL env var (default: https://api.methodex.ai)\n- METHODEX_AUTH_TOKEN env var (Clerk JWT for the test user)\n- METHODEX_TEST_VIDEO_ID env var (UUID of the fixture video)\n\nUsage:\n    export METHODEX_AUTH_TOKEN=<jwt>\n    export METHODEX_TEST_VIDEO_ID=<uuid>\n    python3 scripts/production-smoke-test.py\n\"\"\"\n\nimport os\nimport sys\nimport time\nimport json\nimport urllib.request\nimport urllib.error\n\nAPI = os.environ.get(\"METHODEX_API_URL\", \"https://api.methodex.ai\")\nTOKEN = os.environ.get(\"METHODEX_AUTH_TOKEN\", \"\")\nVIDEO_ID = os.environ.get(\"METHODEX_TEST_VIDEO_ID\", \"\")\nPOLL_INTERVAL = 5\nTIMEOUT = 300  # 5 minutes\n\n\ndef req(method: str, path: str, body: dict | None = None) -> dict:\n    headers = {\n        \"Authorization\": f\"Bearer {TOKEN}\",\n        \"Content-Type\": \"application/json\",\n    }\n    data = json.dumps(body).encode(\"utf-8\") if body else None\n    r = urllib.request.Request(f\"{API}{path}\", data=data, headers=headers, method=method)\n    with urllib.request.urlopen(r, timeout=30) as resp:\n        return json.loads(resp.read().decode(\"utf-8\"))\n\n\ndef main():\n    if not TOKEN or not VIDEO_ID:\n        print(\"METHODEX_AUTH_TOKEN and METHODEX_TEST_VIDEO_ID env vars required\", file=sys.stderr)\n        return 2\n\n    print(f\"🧪 Smoke test: analyze video {VIDEO_ID} against {API}\")\n\n    # 1. Health checks\n    print(\"\\n[1/4] Health check\")\n    for path in (\"/health/live\", \"/health/ready\"):\n        try:\n            resp = urllib.request.urlopen(f\"{API}{path}\", timeout=10)\n            print(f\"  ✅ {path}: {resp.status}\")\n        except urllib.error.HTTPError as e:\n            print(f\"  ❌ {path}: {e.code}\")\n            return 1\n\n    # 2. Trigger analyze\n    print(\"\\n[2/4] Dispatching analyze...\")\n    dispatch = req(\"POST\", f\"/api/videos/{VIDEO_ID}/analyze\")\n    print(f\"  task_id: {dispatch.get('task_id')}\")\n    print(f\"  analysis_id: {dispatch.get('analysis_id')}\")\n    print(f\"  status: {dispatch.get('status')}\")\n\n    # 3. Poll until complete\n    print(\"\\n[3/4] Polling status (every 5s, timeout 5min)...\")\n    start = time.monotonic()\n    last_step = None\n    while time.monotonic() - start < TIMEOUT:\n        status_resp = req(\"GET\", f\"/api/videos/{VIDEO_ID}/analysis/status\")\n        status = status_resp.get(\"status\")\n        current_step = status_resp.get(\"current_step\")\n        if current_step != last_step:\n            print(f\"  [{int(time.monotonic() - start)}s] step={current_step} status={status}\")\n            last_step = current_step\n        if status == \"completed\":\n            print(\"  ✅ analysis completed\")\n            break\n        if status == \"error\":\n            print(f\"  ❌ analysis failed: {status_resp}\")\n            return 1\n        time.sleep(POLL_INTERVAL)\n    else:\n        print(\"  ❌ timeout\")\n        return 1\n\n    # 4. Verify all 5 step_status entries are \"completed\"\n    print(\"\\n[4/4] Verifying step_status transitions...\")\n    final = req(\"GET\", f\"/api/videos/{VIDEO_ID}/analysis/status\")\n    step_status = final.get(\"step_status\") or {}\n    expected_steps = [\"chunk\", \"infer\", \"relate\", \"explain\", \"activate\"]\n    for step in expected_steps:\n        state = step_status.get(step, \"missing\")\n        marker = \"✅\" if state == \"completed\" else \"❌\"\n        print(f\"  {marker} {step}: {state}\")\n\n    if all(step_status.get(s) == \"completed\" for s in expected_steps):\n        print(\"\\n🎉 SMOKE TEST PASSED\")\n        return 0\n    print(\"\\n💥 SMOKE TEST FAILED — not all steps completed\")\n    return 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make executable**\n\n```bash\nchmod +x scripts/production-smoke-test.py\n```\n\n- [ ] **Step 3: Run against production**\n\nRequires a test user JWT and a test video id. If no fixture video exists, create one manually (upload via the UI, run transcription, let it sit in \"transcribed\" state), save the video id and a long-lived test JWT as env vars:\n\n```bash\nexport METHODEX_AUTH_TOKEN=<jwt from a test user>\nexport METHODEX_TEST_VIDEO_ID=<uuid>\npython3 scripts/production-smoke-test.py\n```\n\nExpected: all 5 step transitions reach `completed`, total time under 5 minutes.\n\nIf the test fails, investigate logs (Railway worker logs + Sentry) before declaring WS5 incomplete. The smoke test is a gate — do not proceed to declare success if it fails.\n\n- [ ] **Step 4: Commit**\n\n```bash\ngit add scripts/production-smoke-test.py\ngit commit -m \"test: add production smoke test script\n\nEnd-to-end test that hits /analyze on a known fixture video and\nverifies all 5 step_status transitions reach 'completed' within 5\nminutes. Run manually after each major deploy as a go/no-go check.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 5.2: Measure post-refactor memory footprint\n\n**Files:**\n- Read-only\n\n- [ ] **Step 1: Query Railway usage for the new billing period**\n\n```bash\nTOKEN=$(cat ~/.railway/config.json | python3 -c 'import json,sys; print(json.load(sys.stdin)[\"user\"][\"token\"])')\ncurl -s -X POST https://backboard.railway.app/graphql/v2 \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $TOKEN\" \\\n  -d '{\"query\":\"{ estimatedUsage(workspaceId: \\\"37eb4e96-0873-4033-a392-3c6593a68802\\\", projectId: \\\"154d302f-8609-4897-a10c-1f0d5bfc4f06\\\", measurements: [MEMORY_USAGE_GB, CPU_USAGE, NETWORK_TX_GB, DISK_USAGE_GB]) { measurement estimatedValue } }\"}' \\\n  | python3 -m json.tool\n```\n\n- [ ] **Step 2: Compare against the pre-flight baseline captured in Task 0 Step 3**\n\nRecord the numbers in `memory/project_hobby_scaling_progress.md` under a new \"Post-refactor metrics\" section. Example update:\n\n```markdown\n## Post-refactor metrics (captured 2026-04-XX after WS4 landed)\n\n| Measurement | Pre-refactor | Post-refactor | Delta |\n|---|---|---|---|\n| MEMORY_USAGE_GB (monthly) | 18,276 | ??? | ??? |\n| NETWORK_TX_GB (monthly) | 21.3 | ??? | ??? |\n| CPU_USAGE (monthly vCPU-hr) | 299 | ??? | ??? |\n| Estimated monthly cost | $5.45 | ??? | ??? |\n```\n\n- [ ] **Step 3: Commit the memory update (no code)**\n\nThe memory file lives outside the repo (it's in `~/.claude/projects/.../memory/`), so no git commit — just ensure the file is updated.\n\n## Task 5.3: Run burst load test\n\n**Files:**\n- Create: `scripts/burst-load-test.py`\n\n- [ ] **Step 1: Create the burst load test**\n\nCreate `scripts/burst-load-test.py`:\n\n```python\n#!/usr/bin/env python3\n\"\"\"Burst load test for the methodex analysis pipeline.\n\nFires N concurrent POST /analyze requests against N pre-seeded fixture\nvideos, tracks step transitions over time, and reports:\n- Time-to-first-step-start for each video\n- Total wall-clock for all to reach \"completed\"\n- Queue depth trajectory (inferred from step_status polling)\n- Any failures\n\nRequires a list of fixture video IDs belonging to the test user.\n\nUsage:\n    export METHODEX_AUTH_TOKEN=<jwt>\n    export METHODEX_BURST_VIDEO_IDS=\"uuid1,uuid2,uuid3,...\"\n    python3 scripts/burst-load-test.py\n\"\"\"\n\nimport os\nimport sys\nimport time\nimport json\nimport threading\nimport urllib.request\nimport urllib.error\nfrom concurrent.futures import ThreadPoolExecutor\n\nAPI = os.environ.get(\"METHODEX_API_URL\", \"https://api.methodex.ai\")\nTOKEN = os.environ.get(\"METHODEX_AUTH_TOKEN\", \"\")\nVIDEO_IDS = [v.strip() for v in os.environ.get(\"METHODEX_BURST_VIDEO_IDS\", \"\").split(\",\") if v.strip()]\nPOLL_INTERVAL = 5\nTIMEOUT = 600\n\n\ndef req(method: str, path: str) -> dict:\n    headers = {\"Authorization\": f\"Bearer {TOKEN}\", \"Content-Type\": \"application/json\"}\n    r = urllib.request.Request(f\"{API}{path}\", headers=headers, method=method)\n    with urllib.request.urlopen(r, timeout=30) as resp:\n        return json.loads(resp.read().decode(\"utf-8\"))\n\n\ndef dispatch_one(video_id: str) -> tuple[str, float]:\n    t0 = time.monotonic()\n    req(\"POST\", f\"/api/videos/{video_id}/analyze\")\n    return video_id, time.monotonic() - t0\n\n\ndef poll_one(video_id: str, start_time: float) -> dict:\n    history = []\n    while time.monotonic() - start_time < TIMEOUT:\n        try:\n            status = req(\"GET\", f\"/api/videos/{video_id}/analysis/status\")\n        except Exception as e:\n            history.append({\"t\": time.monotonic() - start_time, \"error\": str(e)})\n            time.sleep(POLL_INTERVAL)\n            continue\n        t = time.monotonic() - start_time\n        history.append({\"t\": t, \"step_status\": status.get(\"step_status\"), \"status\": status.get(\"status\")})\n        if status.get(\"status\") in (\"completed\", \"error\"):\n            return {\"video_id\": video_id, \"final\": status, \"history\": history}\n        time.sleep(POLL_INTERVAL)\n    return {\"video_id\": video_id, \"final\": {\"status\": \"timeout\"}, \"history\": history}\n\n\ndef main():\n    if not TOKEN or not VIDEO_IDS:\n        print(\"METHODEX_AUTH_TOKEN and METHODEX_BURST_VIDEO_IDS env vars required\", file=sys.stderr)\n        return 2\n\n    print(f\"🔥 Burst load test: {len(VIDEO_IDS)} concurrent analyses against {API}\")\n\n    # Dispatch all analyses as fast as possible\n    dispatch_start = time.monotonic()\n    with ThreadPoolExecutor(max_workers=len(VIDEO_IDS)) as executor:\n        dispatch_results = list(executor.map(dispatch_one, VIDEO_IDS))\n    dispatch_elapsed = time.monotonic() - dispatch_start\n    print(f\"\\nDispatched {len(VIDEO_IDS)} analyses in {dispatch_elapsed:.1f}s\")\n    for vid, dt in dispatch_results:\n        print(f\"  {vid}: dispatch took {dt * 1000:.0f}ms\")\n\n    # Poll all in parallel until terminal\n    poll_start = time.monotonic()\n    with ThreadPoolExecutor(max_workers=len(VIDEO_IDS)) as executor:\n        poll_results = list(executor.map(lambda v: poll_one(v, poll_start), VIDEO_IDS))\n\n    total_elapsed = time.monotonic() - dispatch_start\n    print(f\"\\n⏱  Total wall-clock for burst: {total_elapsed:.1f}s\")\n\n    # Summary\n    completed = sum(1 for r in poll_results if r[\"final\"].get(\"status\") == \"completed\")\n    errored = sum(1 for r in poll_results if r[\"final\"].get(\"status\") == \"error\")\n    timeout = sum(1 for r in poll_results if r[\"final\"].get(\"status\") == \"timeout\")\n    print(f\"  completed: {completed} / {len(VIDEO_IDS)}\")\n    print(f\"  errored:   {errored}\")\n    print(f\"  timeout:   {timeout}\")\n\n    if completed == len(VIDEO_IDS):\n        print(\"\\n🎉 BURST LOAD TEST PASSED\")\n        return 0\n    print(\"\\n💥 BURST LOAD TEST FAILED\")\n    return 1\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n```\n\n- [ ] **Step 2: Make executable**\n\n```bash\nchmod +x scripts/burst-load-test.py\n```\n\n- [ ] **Step 3: Run against production with ~10 fixture videos first**\n\nStart small — 10 concurrent analyses — before scaling to the target 25.\n\n```bash\nexport METHODEX_AUTH_TOKEN=<jwt>\nexport METHODEX_BURST_VIDEO_IDS=\"<10 comma-separated uuids>\"\npython3 scripts/burst-load-test.py\n```\n\nReview the output. All 10 should complete within ~3 minutes. If any fail or time out, investigate before scaling.\n\n- [ ] **Step 4: Run again with 25 concurrent**\n\n```bash\nexport METHODEX_BURST_VIDEO_IDS=\"<25 comma-separated uuids>\"\npython3 scripts/burst-load-test.py\n```\n\nExpected: all 25 complete within ~4 minutes, no timeouts, no errors.\n\nIf any step latency looks wrong or tasks pile up in queue, inspect Railway logs + Sentry to find the bottleneck.\n\n- [ ] **Step 5: Commit the burst load test**\n\n```bash\ngit add scripts/burst-load-test.py\ngit commit -m \"test: add burst load test script for multi-replica worker validation\n\nFires N concurrent POST /analyze calls and tracks per-video step\ntransitions until all reach terminal state. Used to validate the\npost-T1/T2 pipeline at target-B scale (25 concurrent).\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n## Task 5.4: Update progress memory to mark project complete\n\n**Files:**\n- Modify: `~/.claude/projects/-Users-idstuart-Projects-ai-prototyping-5d-analysis/memory/project_hobby_scaling_progress.md`\n\n- [ ] **Step 1: Update the memory file status**\n\nEdit the memory file, change the opening from \"IN-FLIGHT\" to \"COMPLETED 2026-04-XX\" and add a final summary section:\n\n```markdown\n## Completion summary (2026-04-XX)\n\n- **Wave 1** landed: WS1 (CI deploy-wait), WS2 (memory + cleanup), WS3 (chain refactor)\n- **Wave 2** landed: WS4 (beat extraction, multi-replica topology, Redis hardening)\n- **Wave 3** validated: smoke test passed, burst load test (25 concurrent) passed\n- **Memory footprint change**: pre-refactor $X.XX/mo → post-refactor $Y.YY/mo\n- **Task duration change**: monolithic 150s/video → chain mean 30s/step, 145s total (similar wall-clock, no thread pinning)\n- **Burst capacity change**: blocked at 8 concurrent → scales to 25+ without queueing\n\nAll goals from the design spec met. The stack remains on Railway Hobby.\n```\n\nSave. No git commit (memory file is outside the repo).\n\n## Task 5.5: Final handoff\n\n- [ ] **Step 1: Write a short handoff note in the repo**\n\nCreate `docs/ops/post-scaling-operational-notes.md`:\n\n```markdown\n# Post-scaling operational notes (2026-04-XX)\n\n## Architecture changes that went live\n\n- `backend` service now runs with 2 replicas, each with 2 uvicorn workers\n- `worker` service now runs with 2 replicas, each with Celery concurrency 16 (threads pool, consumes from queues: analyze, transcribe, celery)\n- New `beat` service (1 replica forever) runs the Celery Beat scheduler\n- Redis now enforces `maxmemory=300mb` + `allkeys-lru`\n- Backend has `/health/live` and `/health/ready` endpoints; Railway health checks point at `/health/ready`\n\n## Known operational gotchas\n\n- **Do not scale `beat` above 1 replica**, ever. Multiple beat instances fire scheduled tasks N times.\n- **Postgres backups are NOT yet automated.** Separate future spec. Until then, run a manual `pg_dump` to R2 weekly (or more often during active development).\n- If the Redis service redeploys, the startCommand override (maxmemory) may reset. Verify after any Redis restart with `redis-cli INFO memory | grep maxmemory`.\n- Worker replicas each hold ~16 Celery threads. A stuck thread under-utilizes half of one replica — not catastrophic. The watchdog task (every 5 min via beat) will mark stuck analyses as error.\n\n## Troubleshooting\n\n- **Backend returns 503 from /health/ready**: DB or Redis unreachable. Check `railway logs` on the `backend` service for the specific probe failure.\n- **Scheduled tasks firing twice**: check that `beat` is running with `numReplicas=1`. If the worker service accidentally re-added `--beat`, remove it.\n- **Queue backlog**: run `redis-cli -h yamabiko.proxy.rlwy.net -p 50187 -a $REDIS_PASSWORD LLEN analyze` — if it's stuck growing, a worker may have died; check `railway logs` on the worker service.\n```\n\n- [ ] **Step 2: Commit the runbook**\n\n```bash\ngit add docs/ops/post-scaling-operational-notes.md\ngit commit -m \"docs: post-scaling operational notes\n\nRunbook covering the new topology (2 backend replicas, 2 worker\nreplicas, 1 beat), known gotchas (don't scale beat, backups still\nmanual), and troubleshooting steps for common production issues.\n\nCo-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>\"\n```\n\n- [ ] **Step 3: WS5 handoff**\n\nAll waves complete. Production is on the new topology, smoke + burst tests pass, docs and memory are updated.\n\nIf anything under load reveals an unforeseen issue — memory pressure, queue backlog, replica contention — file a new spec and do not patch in place.\n\n---\n\n## Self-Review Checklist (mandatory before declaring plan complete)\n\nAfter writing the complete plan, I ran this checklist against the spec:\n\n- ✅ **Spec coverage:** Every track (T1 through T6) from the spec has a corresponding workstream (WS1–WS5) with concrete tasks. T3 (backups) is explicitly out of scope per the spec.\n- ✅ **No placeholders:** No `TBD`, `TODO`, or `implement later` in any task step. Every code block contains the actual code an engineer needs.\n- ✅ **Type/method consistency:** `analyze_chunk_step`, `analyze_infer_step`, etc. referenced consistently. `_check_cancellation` referenced across 3.2, 3.3. `handle_pipeline_error` referenced in 3.5, 3.8, 3.9. `_POOL_CONFIG`/`_pool` in 4.1.\n- ✅ **Exact file paths:** Every task lists the exact files it touches.\n- ✅ **Explicit commits:** Every task ends with a `git commit` step.\n- ✅ **Dependency order documented:** Wave 1 parallel, WS2 rebases on WS3, WS4 after Wave 1 soak, WS5 after WS4.\n- ✅ **Test code inline:** Tests in 3.3, 3.5, 3.6, 4.1, 4.2 have complete code, not just descriptions.\n\nOne area where the plan trusts the executor: **Task 4.7 Step 4** (manual first-time source config of the new beat service in the Railway dashboard) requires human interaction. Automated service creation via Railway's GraphQL API is partial — `serviceCreate` mutation exists but the source (repo + Dockerfile) is typically set by the dashboard UI for first-time setup. This is called out explicitly in the task so an executor knows to pause and do the dashboard step.\n"}}, "backend/tests/test_users_settings.py": {"6760f774b17e4ad5f6e628622f921d5337703fd6": {"t": 1785746914.5154326, "n": 24075, "h": "451b76b63e6b3608", "c": "\"\"\"Tests for the split user-settings endpoints introduced in 2026-04 redesign.\n\nFixture notes\n-------------\n- ``client`` is an async HTTPX client (anyio). All tests must be async.\n- Auth uses ``Bearer dev-bypass`` → the dev auth bypass returns ``dev_user_local``.\n- ``db_user_factory`` (defined below as a local fixture) seeds the\n  ``dev_user_local`` row in the per-test SQLite DB with the caller-supplied\n  attributes.  Each test gets an isolated DB from ``conftest.client``'s\n  ``tmp_path``, so there is no state leakage between tests.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom datetime import datetime, timezone\nfrom unittest.mock import patch\n\nimport pytest\nfrom fastapi import status\n\nfrom app.constants import DEFAULT_STANDARD_MODEL\nfrom app.services.openrouter_balance import BalanceInfo, OpenRouterBalanceError\n\npytestmark = pytest.mark.anyio\n\n_AUTH = {\"Authorization\": \"Bearer dev-bypass\"}\n_DEV_USER_ID = \"dev_user_local\"\n\n\n# ── BalanceInfo helpers ───────────────────────────────────────────────────────\n# BalanceInfo is a frozen dataclass; all fields must be supplied explicitly.\n\n\ndef _healthy_balance(api_key: str = \"sk-or-v1-test1234\") -> BalanceInfo:\n    total_credits = 10.0\n    total_usage = 1.48\n    balance_remaining = total_credits - total_usage\n    return BalanceInfo(\n        total_credits=total_credits,\n        total_usage=total_usage,\n        balance_remaining=balance_remaining,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=False,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=True,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\ndef _empty_balance(api_key: str = \"sk-or-v1-empty0\") -> BalanceInfo:\n    return BalanceInfo(\n        total_credits=0.0,\n        total_usage=0.0,\n        balance_remaining=0.0,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=True,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=False,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\n# ── db_user_factory fixture ───────────────────────────────────────────────────\n\n\n@pytest.fixture\ndef db_user_factory(client):\n    \"\"\"Factory that seeds the dev_user_local row in the per-test DB.\n\n    Uses the same DB session override that the ``client`` fixture installs\n    on the FastAPI app, so the insert lands in the per-test SQLite file.\n\n    Usage::\n\n        user = db_user_factory()                          # no key\n        user = db_user_factory(preferred_model=\"foo/bar\")\n        user = db_user_factory(encrypted_api_key=b\"x\", key_hint=\"abcd\")\n\n    Returns the created User ORM object so tests can inspect its id.\n    \"\"\"\n    from app.database import get_db\n    from app.main import app\n    from app.models.database_models import User\n\n    def _factory(\n        *,\n        encrypted_api_key=None,\n        key_hint=None,\n        preferred_model=None,\n        model_tier=None,\n        key_total_credits=None,\n        key_total_usage=None,\n        key_is_free_tier=None,\n    ):\n        # Use the override that ``client`` installed on the app — that's\n        # the session pointing at the per-test SQLite file.\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            user = User(\n                id=_DEV_USER_ID,\n                email=\"dev@example.com\",\n                role=\"user\",\n            )\n            if encrypted_api_key is not None:\n                user.encrypted_api_key = encrypted_api_key\n            if key_hint is not None:\n                user.key_hint = key_hint\n            if preferred_model is not None:\n                user.preferred_model = preferred_model\n            if model_tier is not None:\n                user.model_tier = model_tier\n            if key_total_credits is not None:\n                user.key_total_credits = key_total_credits\n            if key_total_usage is not None:\n                user.key_total_usage = key_total_usage\n            if key_is_free_tier is not None:\n                user.key_is_free_tier = key_is_free_tier\n            db.add(user)\n            db.commit()\n            db.refresh(user)\n            return user\n        finally:\n            db.close()\n\n    return _factory\n\n\n# ── GET /settings ─────────────────────────────────────────────────────────────\n\n\nclass TestGetSettings:\n    async def test_returns_model_tier_field(self, client, db_user_factory):\n        \"\"\"GET /settings response includes the model_tier field.\"\"\"\n        db_user_factory(model_tier=\"included\")\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert \"model_tier\" in body\n        assert body[\"model_tier\"] == \"included\"\n\n    async def test_includes_low_balance_threshold_from_config(\n        self, client, db_user_factory\n    ):\n        \"\"\"GET /settings serializes the server-driven low-balance threshold.\n\n        Locks the byok-balance contract: the threshold is sourced from\n        settings.LOW_BALANCE_THRESHOLD_USD, not a frontend fallback.\n        \"\"\"\n        from app.config import settings\n\n        db_user_factory(model_tier=\"included\")\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert \"low_balance_threshold_usd\" in body\n        assert body[\"low_balance_threshold_usd\"] == settings.LOW_BALANCE_THRESHOLD_USD\n\n    async def test_returns_byok_model_tier(self, client, db_user_factory):\n        \"\"\"GET /settings for a BYOK user returns model_tier='byok'.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-key\",\n            key_hint=\"abcd\",\n            model_tier=\"byok\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.get(\n                \"/api/users/settings\",\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"model_tier\"] == \"byok\"\n        assert body[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n\n    async def test_defaults_to_included_when_tier_unset(self, client, db_user_factory):\n        \"\"\"GET /settings defaults model_tier to 'included' when not explicitly set.\"\"\"\n        db_user_factory()  # model_tier not set\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"model_tier\"] == \"included\"\n\n\n# ── POST /settings/api-key ────────────────────────────────────────────────────\n\n\nclass TestPostApiKey:\n    async def test_valid_key_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"Happy path: post a valid key with credits, get the full settings response back.\"\"\"\n        db_user_factory()  # creates dev_user_local with no key\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"1234\"\n        assert body[\"balance\"][\"balance_remaining\"] == pytest.approx(8.52)\n        assert body[\"balance\"][\"has_credits\"] is True\n        # The endpoint must NOT touch preferred_model\n        assert body[\"preferred_model\"] is None\n\n    async def test_zero_credit_key_returns_400(self, client, db_user_factory):\n        \"\"\"The Baffour Adu case: brand-new key with $0 → reject before saving.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_empty_balance(\"sk-or-v1-zerox0\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-zerox0\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"$0 credits\" in response.json()[\"detail\"]\n\n        # Verify the key was NOT persisted — use the same override the client uses\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n        finally:\n            db.close()\n\n    async def test_unreachable_openrouter_returns_400(self, client, db_user_factory):\n        \"\"\"fetch_balance_sync raises → 400 with 'invalid or unreachable'.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            side_effect=OpenRouterBalanceError(\"upstream timeout\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-broken1\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"invalid api key or openrouter is temporarily unreachable\" in (\n            response.json()[\"detail\"].lower()\n        )\n\n    async def test_replacing_existing_key_overwrites_hint(self, client, db_user_factory):\n        \"\"\"Replacing an existing key saves the new hint and discards the old one.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"old-encrypted\",\n            key_hint=\"oldA\",\n        )\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-newx5678\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-newx5678\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"key_hint\"] == \"5678\"\n\n    async def test_does_not_touch_preferred_model(self, client, db_user_factory):\n        \"\"\"Even if a saved standard model exists, this endpoint must leave it alone.\"\"\"\n        db_user_factory(preferred_model=\"meta-llama/llama-4-scout\")\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n\n    async def test_blank_key_returns_422(self, client, db_user_factory):\n        \"\"\"Whitespace-only key is rejected by schema validation before hitting the route.\"\"\"\n        db_user_factory()\n        response = await client.post(\n            \"/api/users/settings/api-key\",\n            json={\"api_key\": \"          \"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n\n# ── PUT /settings/preferred-model ────────────────────────────────────────\n\n\nclass TestPutPreferredModel:\n    async def test_no_key_standard_model_returns_200(self, client, db_user_factory):\n        db_user_factory()  # no key\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n        assert response.json()[\"model_tier\"] == \"included\"\n\n    async def test_included_tier_premium_model_returns_400(self, client, db_user_factory):\n        \"\"\"Included tier + non-standard model = 400.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"not available on the included tier\" in response.json()[\"detail\"].lower()\n\n    async def test_byok_tier_no_key_returns_403(self, client, db_user_factory):\n        \"\"\"BYOK tier but no key on file = 403.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_403_FORBIDDEN\n        assert \"openrouter api key\" in response.json()[\"detail\"].lower()\n\n    async def test_with_key_byok_tier_premium_model_returns_200(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n        assert response.json()[\"model_tier\"] == \"byok\"\n\n    async def test_with_key_included_tier_standard_model_returns_200(\n        self, client, db_user_factory\n    ):\n        \"\"\"A BYOK user can fall back to the included tier with a standard model.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"deepseek/deepseek-chat-v3-0324\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"deepseek/deepseek-chat-v3-0324\"\n        assert response.json()[\"model_tier\"] == \"included\"\n\n    async def test_invalid_format_returns_422(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"no-slash-here\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_invalid_tier_returns_422(self, client, db_user_factory):\n        \"\"\"Invalid model_tier value is rejected by schema validation.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"premium\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_missing_tier_returns_422(self, client, db_user_factory):\n        \"\"\"model_tier is required in the request payload.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_byok_tier_zero_balance_returns_402(self, client, db_user_factory):\n        \"\"\"BYOK tier with key but $0 balance -> 402 with structured error detail.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_empty_balance(\"sk-or-v1-empty0\"),\n        ):\n            response = await client.put(\n                \"/api/users/settings/preferred-model\",\n                json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_402_PAYMENT_REQUIRED\n        assert \"no remaining credits\" in response.json()[\"detail\"].lower()\n\n    async def test_byok_tier_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"BYOK tier with key and credits -> 200, both model_tier and preferred_model saved.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.put(\n                \"/api/users/settings/preferred-model\",\n                json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n        assert body[\"model_tier\"] == \"byok\"\n\n    async def test_included_tier_with_byok_key_on_file_stays_included(\n        self, client, db_user_factory\n    ):\n        \"\"\"User has a BYOK key on file but selects included tier with a standard model.\n\n        The PUT succeeds, tier stays included, and the BYOK key is untouched.\n        \"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"byok-key-present\",\n            key_hint=\"xyzw\",\n            model_tier=\"byok\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"model_tier\"] == \"included\"\n        assert body[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n        # Key is still present\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"xyzw\"\n\n    async def test_does_not_touch_api_key(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"keep-me\",\n            key_hint=\"zzzz\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"has_api_key\"] is True\n        assert response.json()[\"key_hint\"] == \"zzzz\"\n\n\n# ── DELETE /settings/api-key ─────────────────────────────────────────────\n\n\nclass TestDeleteApiKey:\n    async def test_with_key_and_premium_model_resets_to_default(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n            model_tier=\"byok\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n            assert refreshed.model_tier == \"included\"\n        finally:\n            db.close()\n\n    async def test_with_key_and_standard_model_still_resets_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"Always reset on delete — even from a non-default standard.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"deepseek/deepseek-chat-v3-0324\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n        finally:\n            db.close()\n\n    async def test_without_key_is_idempotent(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n    async def test_delete_resets_tier_to_included_and_model_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"After DELETE, model_tier is reset to 'included' and preferred_model\n        to the default standard model — even if the user was on BYOK with a\n        premium model.\n        \"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n            model_tier=\"byok\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.model_tier == \"included\"\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n        finally:\n            db.close()\n\n    async def test_balance_columns_cleared(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            key_total_credits=10.0,\n            key_total_usage=2.0,\n            key_is_free_tier=False,\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.key_total_credits is None\n            assert refreshed.key_total_usage is None\n            assert refreshed.key_is_free_tier is None\n        finally:\n            db.close()\n"}, "69e7bac8773b366dbfcb0180f873471b6251cf0e": {"t": 1785765428.5757666, "n": 24075, "h": "451b76b63e6b3608", "c": "\"\"\"Tests for the split user-settings endpoints introduced in 2026-04 redesign.\n\nFixture notes\n-------------\n- ``client`` is an async HTTPX client (anyio). All tests must be async.\n- Auth uses ``Bearer dev-bypass`` → the dev auth bypass returns ``dev_user_local``.\n- ``db_user_factory`` (defined below as a local fixture) seeds the\n  ``dev_user_local`` row in the per-test SQLite DB with the caller-supplied\n  attributes.  Each test gets an isolated DB from ``conftest.client``'s\n  ``tmp_path``, so there is no state leakage between tests.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom datetime import datetime, timezone\nfrom unittest.mock import patch\n\nimport pytest\nfrom fastapi import status\n\nfrom app.constants import DEFAULT_STANDARD_MODEL\nfrom app.services.openrouter_balance import BalanceInfo, OpenRouterBalanceError\n\npytestmark = pytest.mark.anyio\n\n_AUTH = {\"Authorization\": \"Bearer dev-bypass\"}\n_DEV_USER_ID = \"dev_user_local\"\n\n\n# ── BalanceInfo helpers ───────────────────────────────────────────────────────\n# BalanceInfo is a frozen dataclass; all fields must be supplied explicitly.\n\n\ndef _healthy_balance(api_key: str = \"sk-or-v1-test1234\") -> BalanceInfo:\n    total_credits = 10.0\n    total_usage = 1.48\n    balance_remaining = total_credits - total_usage\n    return BalanceInfo(\n        total_credits=total_credits,\n        total_usage=total_usage,\n        balance_remaining=balance_remaining,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=False,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=True,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\ndef _empty_balance(api_key: str = \"sk-or-v1-empty0\") -> BalanceInfo:\n    return BalanceInfo(\n        total_credits=0.0,\n        total_usage=0.0,\n        balance_remaining=0.0,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=True,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=False,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\n# ── db_user_factory fixture ───────────────────────────────────────────────────\n\n\n@pytest.fixture\ndef db_user_factory(client):\n    \"\"\"Factory that seeds the dev_user_local row in the per-test DB.\n\n    Uses the same DB session override that the ``client`` fixture installs\n    on the FastAPI app, so the insert lands in the per-test SQLite file.\n\n    Usage::\n\n        user = db_user_factory()                          # no key\n        user = db_user_factory(preferred_model=\"foo/bar\")\n        user = db_user_factory(encrypted_api_key=b\"x\", key_hint=\"abcd\")\n\n    Returns the created User ORM object so tests can inspect its id.\n    \"\"\"\n    from app.database import get_db\n    from app.main import app\n    from app.models.database_models import User\n\n    def _factory(\n        *,\n        encrypted_api_key=None,\n        key_hint=None,\n        preferred_model=None,\n        model_tier=None,\n        key_total_credits=None,\n        key_total_usage=None,\n        key_is_free_tier=None,\n    ):\n        # Use the override that ``client`` installed on the app — that's\n        # the session pointing at the per-test SQLite file.\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            user = User(\n                id=_DEV_USER_ID,\n                email=\"dev@example.com\",\n                role=\"user\",\n            )\n            if encrypted_api_key is not None:\n                user.encrypted_api_key = encrypted_api_key\n            if key_hint is not None:\n                user.key_hint = key_hint\n            if preferred_model is not None:\n                user.preferred_model = preferred_model\n            if model_tier is not None:\n                user.model_tier = model_tier\n            if key_total_credits is not None:\n                user.key_total_credits = key_total_credits\n            if key_total_usage is not None:\n                user.key_total_usage = key_total_usage\n            if key_is_free_tier is not None:\n                user.key_is_free_tier = key_is_free_tier\n            db.add(user)\n            db.commit()\n            db.refresh(user)\n            return user\n        finally:\n            db.close()\n\n    return _factory\n\n\n# ── GET /settings ─────────────────────────────────────────────────────────────\n\n\nclass TestGetSettings:\n    async def test_returns_model_tier_field(self, client, db_user_factory):\n        \"\"\"GET /settings response includes the model_tier field.\"\"\"\n        db_user_factory(model_tier=\"included\")\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert \"model_tier\" in body\n        assert body[\"model_tier\"] == \"included\"\n\n    async def test_includes_low_balance_threshold_from_config(\n        self, client, db_user_factory\n    ):\n        \"\"\"GET /settings serializes the server-driven low-balance threshold.\n\n        Locks the byok-balance contract: the threshold is sourced from\n        settings.LOW_BALANCE_THRESHOLD_USD, not a frontend fallback.\n        \"\"\"\n        from app.config import settings\n\n        db_user_factory(model_tier=\"included\")\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert \"low_balance_threshold_usd\" in body\n        assert body[\"low_balance_threshold_usd\"] == settings.LOW_BALANCE_THRESHOLD_USD\n\n    async def test_returns_byok_model_tier(self, client, db_user_factory):\n        \"\"\"GET /settings for a BYOK user returns model_tier='byok'.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-key\",\n            key_hint=\"abcd\",\n            model_tier=\"byok\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.get(\n                \"/api/users/settings\",\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"model_tier\"] == \"byok\"\n        assert body[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n\n    async def test_defaults_to_included_when_tier_unset(self, client, db_user_factory):\n        \"\"\"GET /settings defaults model_tier to 'included' when not explicitly set.\"\"\"\n        db_user_factory()  # model_tier not set\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"model_tier\"] == \"included\"\n\n\n# ── POST /settings/api-key ────────────────────────────────────────────────────\n\n\nclass TestPostApiKey:\n    async def test_valid_key_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"Happy path: post a valid key with credits, get the full settings response back.\"\"\"\n        db_user_factory()  # creates dev_user_local with no key\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"1234\"\n        assert body[\"balance\"][\"balance_remaining\"] == pytest.approx(8.52)\n        assert body[\"balance\"][\"has_credits\"] is True\n        # The endpoint must NOT touch preferred_model\n        assert body[\"preferred_model\"] is None\n\n    async def test_zero_credit_key_returns_400(self, client, db_user_factory):\n        \"\"\"The Baffour Adu case: brand-new key with $0 → reject before saving.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_empty_balance(\"sk-or-v1-zerox0\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-zerox0\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"$0 credits\" in response.json()[\"detail\"]\n\n        # Verify the key was NOT persisted — use the same override the client uses\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n        finally:\n            db.close()\n\n    async def test_unreachable_openrouter_returns_400(self, client, db_user_factory):\n        \"\"\"fetch_balance_sync raises → 400 with 'invalid or unreachable'.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            side_effect=OpenRouterBalanceError(\"upstream timeout\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-broken1\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"invalid api key or openrouter is temporarily unreachable\" in (\n            response.json()[\"detail\"].lower()\n        )\n\n    async def test_replacing_existing_key_overwrites_hint(self, client, db_user_factory):\n        \"\"\"Replacing an existing key saves the new hint and discards the old one.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"old-encrypted\",\n            key_hint=\"oldA\",\n        )\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-newx5678\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-newx5678\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"key_hint\"] == \"5678\"\n\n    async def test_does_not_touch_preferred_model(self, client, db_user_factory):\n        \"\"\"Even if a saved standard model exists, this endpoint must leave it alone.\"\"\"\n        db_user_factory(preferred_model=\"meta-llama/llama-4-scout\")\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n\n    async def test_blank_key_returns_422(self, client, db_user_factory):\n        \"\"\"Whitespace-only key is rejected by schema validation before hitting the route.\"\"\"\n        db_user_factory()\n        response = await client.post(\n            \"/api/users/settings/api-key\",\n            json={\"api_key\": \"          \"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n\n# ── PUT /settings/preferred-model ────────────────────────────────────────\n\n\nclass TestPutPreferredModel:\n    async def test_no_key_standard_model_returns_200(self, client, db_user_factory):\n        db_user_factory()  # no key\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n        assert response.json()[\"model_tier\"] == \"included\"\n\n    async def test_included_tier_premium_model_returns_400(self, client, db_user_factory):\n        \"\"\"Included tier + non-standard model = 400.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"not available on the included tier\" in response.json()[\"detail\"].lower()\n\n    async def test_byok_tier_no_key_returns_403(self, client, db_user_factory):\n        \"\"\"BYOK tier but no key on file = 403.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_403_FORBIDDEN\n        assert \"openrouter api key\" in response.json()[\"detail\"].lower()\n\n    async def test_with_key_byok_tier_premium_model_returns_200(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n        assert response.json()[\"model_tier\"] == \"byok\"\n\n    async def test_with_key_included_tier_standard_model_returns_200(\n        self, client, db_user_factory\n    ):\n        \"\"\"A BYOK user can fall back to the included tier with a standard model.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"deepseek/deepseek-chat-v3-0324\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"deepseek/deepseek-chat-v3-0324\"\n        assert response.json()[\"model_tier\"] == \"included\"\n\n    async def test_invalid_format_returns_422(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"no-slash-here\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_invalid_tier_returns_422(self, client, db_user_factory):\n        \"\"\"Invalid model_tier value is rejected by schema validation.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"premium\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_missing_tier_returns_422(self, client, db_user_factory):\n        \"\"\"model_tier is required in the request payload.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_byok_tier_zero_balance_returns_402(self, client, db_user_factory):\n        \"\"\"BYOK tier with key but $0 balance -> 402 with structured error detail.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_empty_balance(\"sk-or-v1-empty0\"),\n        ):\n            response = await client.put(\n                \"/api/users/settings/preferred-model\",\n                json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_402_PAYMENT_REQUIRED\n        assert \"no remaining credits\" in response.json()[\"detail\"].lower()\n\n    async def test_byok_tier_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"BYOK tier with key and credits -> 200, both model_tier and preferred_model saved.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.put(\n                \"/api/users/settings/preferred-model\",\n                json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n        assert body[\"model_tier\"] == \"byok\"\n\n    async def test_included_tier_with_byok_key_on_file_stays_included(\n        self, client, db_user_factory\n    ):\n        \"\"\"User has a BYOK key on file but selects included tier with a standard model.\n\n        The PUT succeeds, tier stays included, and the BYOK key is untouched.\n        \"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"byok-key-present\",\n            key_hint=\"xyzw\",\n            model_tier=\"byok\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"model_tier\"] == \"included\"\n        assert body[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n        # Key is still present\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"xyzw\"\n\n    async def test_does_not_touch_api_key(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"keep-me\",\n            key_hint=\"zzzz\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"has_api_key\"] is True\n        assert response.json()[\"key_hint\"] == \"zzzz\"\n\n\n# ── DELETE /settings/api-key ─────────────────────────────────────────────\n\n\nclass TestDeleteApiKey:\n    async def test_with_key_and_premium_model_resets_to_default(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n            model_tier=\"byok\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n            assert refreshed.model_tier == \"included\"\n        finally:\n            db.close()\n\n    async def test_with_key_and_standard_model_still_resets_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"Always reset on delete — even from a non-default standard.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"deepseek/deepseek-chat-v3-0324\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n        finally:\n            db.close()\n\n    async def test_without_key_is_idempotent(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n    async def test_delete_resets_tier_to_included_and_model_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"After DELETE, model_tier is reset to 'included' and preferred_model\n        to the default standard model — even if the user was on BYOK with a\n        premium model.\n        \"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n            model_tier=\"byok\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.model_tier == \"included\"\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n        finally:\n            db.close()\n\n    async def test_balance_columns_cleared(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            key_total_credits=10.0,\n            key_total_usage=2.0,\n            key_is_free_tier=False,\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.key_total_credits is None\n            assert refreshed.key_total_usage is None\n            assert refreshed.key_is_free_tier is None\n        finally:\n            db.close()\n"}, "3e53a2574a3b2d5f1cc9e1a87808d241f0be279e": {"t": 1785765430.0367227, "n": 23312, "h": "2e6d477268820de1", "c": "\"\"\"Tests for the split user-settings endpoints introduced in 2026-04 redesign.\n\nFixture notes\n-------------\n- ``client`` is an async HTTPX client (anyio). All tests must be async.\n- Auth uses ``Bearer dev-bypass`` → the dev auth bypass returns ``dev_user_local``.\n- ``db_user_factory`` (defined below as a local fixture) seeds the\n  ``dev_user_local`` row in the per-test SQLite DB with the caller-supplied\n  attributes.  Each test gets an isolated DB from ``conftest.client``'s\n  ``tmp_path``, so there is no state leakage between tests.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom datetime import datetime, timezone\nfrom unittest.mock import patch\n\nimport pytest\nfrom fastapi import status\n\nfrom app.constants import DEFAULT_STANDARD_MODEL\nfrom app.services.openrouter_balance import BalanceInfo, OpenRouterBalanceError\n\npytestmark = pytest.mark.anyio\n\n_AUTH = {\"Authorization\": \"Bearer dev-bypass\"}\n_DEV_USER_ID = \"dev_user_local\"\n\n\n# ── BalanceInfo helpers ───────────────────────────────────────────────────────\n# BalanceInfo is a frozen dataclass; all fields must be supplied explicitly.\n\n\ndef _healthy_balance(api_key: str = \"sk-or-v1-test1234\") -> BalanceInfo:\n    total_credits = 10.0\n    total_usage = 1.48\n    balance_remaining = total_credits - total_usage\n    return BalanceInfo(\n        total_credits=total_credits,\n        total_usage=total_usage,\n        balance_remaining=balance_remaining,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=False,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=True,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\ndef _empty_balance(api_key: str = \"sk-or-v1-empty0\") -> BalanceInfo:\n    return BalanceInfo(\n        total_credits=0.0,\n        total_usage=0.0,\n        balance_remaining=0.0,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=True,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=False,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\n# ── db_user_factory fixture ───────────────────────────────────────────────────\n\n\n@pytest.fixture\ndef db_user_factory(client):\n    \"\"\"Factory that seeds the dev_user_local row in the per-test DB.\n\n    Uses the same DB session override that the ``client`` fixture installs\n    on the FastAPI app, so the insert lands in the per-test SQLite file.\n\n    Usage::\n\n        user = db_user_factory()                          # no key\n        user = db_user_factory(preferred_model=\"foo/bar\")\n        user = db_user_factory(encrypted_api_key=b\"x\", key_hint=\"abcd\")\n\n    Returns the created User ORM object so tests can inspect its id.\n    \"\"\"\n    from app.database import get_db\n    from app.main import app\n    from app.models.database_models import User\n\n    def _factory(\n        *,\n        encrypted_api_key=None,\n        key_hint=None,\n        preferred_model=None,\n        model_tier=None,\n        key_total_credits=None,\n        key_total_usage=None,\n        key_is_free_tier=None,\n    ):\n        # Use the override that ``client`` installed on the app — that's\n        # the session pointing at the per-test SQLite file.\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            user = User(\n                id=_DEV_USER_ID,\n                email=\"dev@example.com\",\n                role=\"user\",\n            )\n            if encrypted_api_key is not None:\n                user.encrypted_api_key = encrypted_api_key\n            if key_hint is not None:\n                user.key_hint = key_hint\n            if preferred_model is not None:\n                user.preferred_model = preferred_model\n            if model_tier is not None:\n                user.model_tier = model_tier\n            if key_total_credits is not None:\n                user.key_total_credits = key_total_credits\n            if key_total_usage is not None:\n                user.key_total_usage = key_total_usage\n            if key_is_free_tier is not None:\n                user.key_is_free_tier = key_is_free_tier\n            db.add(user)\n            db.commit()\n            db.refresh(user)\n            return user\n        finally:\n            db.close()\n\n    return _factory\n\n\n# ── GET /settings ─────────────────────────────────────────────────────────────\n\n\nclass TestGetSettings:\n    async def test_returns_model_tier_field(self, client, db_user_factory):\n        \"\"\"GET /settings response includes the model_tier field.\"\"\"\n        db_user_factory(model_tier=\"included\")\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert \"model_tier\" in body\n        assert body[\"model_tier\"] == \"included\"\n\n    async def test_returns_byok_model_tier(self, client, db_user_factory):\n        \"\"\"GET /settings for a BYOK user returns model_tier='byok'.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-key\",\n            key_hint=\"abcd\",\n            model_tier=\"byok\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.get(\n                \"/api/users/settings\",\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"model_tier\"] == \"byok\"\n        assert body[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n\n    async def test_defaults_to_included_when_tier_unset(self, client, db_user_factory):\n        \"\"\"GET /settings defaults model_tier to 'included' when not explicitly set.\"\"\"\n        db_user_factory()  # model_tier not set\n        response = await client.get(\n            \"/api/users/settings\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"model_tier\"] == \"included\"\n\n\n# ── POST /settings/api-key ────────────────────────────────────────────────────\n\n\nclass TestPostApiKey:\n    async def test_valid_key_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"Happy path: post a valid key with credits, get the full settings response back.\"\"\"\n        db_user_factory()  # creates dev_user_local with no key\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"1234\"\n        assert body[\"balance\"][\"balance_remaining\"] == pytest.approx(8.52)\n        assert body[\"balance\"][\"has_credits\"] is True\n        # The endpoint must NOT touch preferred_model\n        assert body[\"preferred_model\"] is None\n\n    async def test_zero_credit_key_returns_400(self, client, db_user_factory):\n        \"\"\"The Baffour Adu case: brand-new key with $0 → reject before saving.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_empty_balance(\"sk-or-v1-zerox0\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-zerox0\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"$0 credits\" in response.json()[\"detail\"]\n\n        # Verify the key was NOT persisted — use the same override the client uses\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n        finally:\n            db.close()\n\n    async def test_unreachable_openrouter_returns_400(self, client, db_user_factory):\n        \"\"\"fetch_balance_sync raises → 400 with 'invalid or unreachable'.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            side_effect=OpenRouterBalanceError(\"upstream timeout\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-broken1\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"invalid api key or openrouter is temporarily unreachable\" in (\n            response.json()[\"detail\"].lower()\n        )\n\n    async def test_replacing_existing_key_overwrites_hint(self, client, db_user_factory):\n        \"\"\"Replacing an existing key saves the new hint and discards the old one.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"old-encrypted\",\n            key_hint=\"oldA\",\n        )\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-newx5678\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-newx5678\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"key_hint\"] == \"5678\"\n\n    async def test_does_not_touch_preferred_model(self, client, db_user_factory):\n        \"\"\"Even if a saved standard model exists, this endpoint must leave it alone.\"\"\"\n        db_user_factory(preferred_model=\"meta-llama/llama-4-scout\")\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n\n    async def test_blank_key_returns_422(self, client, db_user_factory):\n        \"\"\"Whitespace-only key is rejected by schema validation before hitting the route.\"\"\"\n        db_user_factory()\n        response = await client.post(\n            \"/api/users/settings/api-key\",\n            json={\"api_key\": \"          \"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n\n# ── PUT /settings/preferred-model ────────────────────────────────────────\n\n\nclass TestPutPreferredModel:\n    async def test_no_key_standard_model_returns_200(self, client, db_user_factory):\n        db_user_factory()  # no key\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n        assert response.json()[\"model_tier\"] == \"included\"\n\n    async def test_included_tier_premium_model_returns_400(self, client, db_user_factory):\n        \"\"\"Included tier + non-standard model = 400.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"not available on the included tier\" in response.json()[\"detail\"].lower()\n\n    async def test_byok_tier_no_key_returns_403(self, client, db_user_factory):\n        \"\"\"BYOK tier but no key on file = 403.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_403_FORBIDDEN\n        assert \"openrouter api key\" in response.json()[\"detail\"].lower()\n\n    async def test_with_key_byok_tier_premium_model_returns_200(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n        assert response.json()[\"model_tier\"] == \"byok\"\n\n    async def test_with_key_included_tier_standard_model_returns_200(\n        self, client, db_user_factory\n    ):\n        \"\"\"A BYOK user can fall back to the included tier with a standard model.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"deepseek/deepseek-chat-v3-0324\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"deepseek/deepseek-chat-v3-0324\"\n        assert response.json()[\"model_tier\"] == \"included\"\n\n    async def test_invalid_format_returns_422(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"no-slash-here\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_invalid_tier_returns_422(self, client, db_user_factory):\n        \"\"\"Invalid model_tier value is rejected by schema validation.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"premium\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_missing_tier_returns_422(self, client, db_user_factory):\n        \"\"\"model_tier is required in the request payload.\"\"\"\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_byok_tier_zero_balance_returns_402(self, client, db_user_factory):\n        \"\"\"BYOK tier with key but $0 balance -> 402 with structured error detail.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_empty_balance(\"sk-or-v1-empty0\"),\n        ):\n            response = await client.put(\n                \"/api/users/settings/preferred-model\",\n                json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_402_PAYMENT_REQUIRED\n        assert \"no remaining credits\" in response.json()[\"detail\"].lower()\n\n    async def test_byok_tier_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"BYOK tier with key and credits -> 200, both model_tier and preferred_model saved.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        with patch(\n            \"app.routes.users.get_cached_balance\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.put(\n                \"/api/users/settings/preferred-model\",\n                json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\", \"model_tier\": \"byok\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n        assert body[\"model_tier\"] == \"byok\"\n\n    async def test_included_tier_with_byok_key_on_file_stays_included(\n        self, client, db_user_factory\n    ):\n        \"\"\"User has a BYOK key on file but selects included tier with a standard model.\n\n        The PUT succeeds, tier stays included, and the BYOK key is untouched.\n        \"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"byok-key-present\",\n            key_hint=\"xyzw\",\n            model_tier=\"byok\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"model_tier\"] == \"included\"\n        assert body[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n        # Key is still present\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"xyzw\"\n\n    async def test_does_not_touch_api_key(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"keep-me\",\n            key_hint=\"zzzz\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\", \"model_tier\": \"included\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"has_api_key\"] is True\n        assert response.json()[\"key_hint\"] == \"zzzz\"\n\n\n# ── DELETE /settings/api-key ─────────────────────────────────────────────\n\n\nclass TestDeleteApiKey:\n    async def test_with_key_and_premium_model_resets_to_default(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n            model_tier=\"byok\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n            assert refreshed.model_tier == \"included\"\n        finally:\n            db.close()\n\n    async def test_with_key_and_standard_model_still_resets_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"Always reset on delete — even from a non-default standard.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"deepseek/deepseek-chat-v3-0324\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n        finally:\n            db.close()\n\n    async def test_without_key_is_idempotent(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n    async def test_delete_resets_tier_to_included_and_model_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"After DELETE, model_tier is reset to 'included' and preferred_model\n        to the default standard model — even if the user was on BYOK with a\n        premium model.\n        \"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n            model_tier=\"byok\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.model_tier == \"included\"\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n        finally:\n            db.close()\n\n    async def test_balance_columns_cleared(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            key_total_credits=10.0,\n            key_total_usage=2.0,\n            key_is_free_tier=False,\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.key_total_credits is None\n            assert refreshed.key_total_usage is None\n            assert refreshed.key_is_free_tier is None\n        finally:\n            db.close()\n"}, "f93736161ee989a780d295dd8a83825d13ba426d": {"t": 1785765431.7680516, "n": 15421, "h": "144c993a3eebdb81", "c": "\"\"\"Tests for the split user-settings endpoints introduced in 2026-04 redesign.\n\nFixture notes\n-------------\n- ``client`` is an async HTTPX client (anyio). All tests must be async.\n- Auth uses ``Bearer dev-bypass`` → the dev auth bypass returns ``dev_user_local``.\n- ``db_user_factory`` (defined below as a local fixture) seeds the\n  ``dev_user_local`` row in the per-test SQLite DB with the caller-supplied\n  attributes.  Each test gets an isolated DB from ``conftest.client``'s\n  ``tmp_path``, so there is no state leakage between tests.\n\"\"\"\n\nfrom __future__ import annotations\n\nfrom datetime import datetime, timezone\nfrom unittest.mock import patch\n\nimport pytest\nfrom fastapi import status\n\nfrom app.constants import DEFAULT_STANDARD_MODEL\nfrom app.services.openrouter_balance import BalanceInfo, OpenRouterBalanceError\n\npytestmark = pytest.mark.anyio\n\n_AUTH = {\"Authorization\": \"Bearer dev-bypass\"}\n_DEV_USER_ID = \"dev_user_local\"\n\n\n# ── BalanceInfo helpers ───────────────────────────────────────────────────────\n# BalanceInfo is a frozen dataclass; all fields must be supplied explicitly.\n\n\ndef _healthy_balance(api_key: str = \"sk-or-v1-test1234\") -> BalanceInfo:\n    total_credits = 10.0\n    total_usage = 1.48\n    balance_remaining = total_credits - total_usage\n    return BalanceInfo(\n        total_credits=total_credits,\n        total_usage=total_usage,\n        balance_remaining=balance_remaining,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=False,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=True,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\ndef _empty_balance(api_key: str = \"sk-or-v1-empty0\") -> BalanceInfo:\n    return BalanceInfo(\n        total_credits=0.0,\n        total_usage=0.0,\n        balance_remaining=0.0,\n        key_limit=None,\n        key_limit_remaining=None,\n        is_free_tier=True,\n        key_label=f\"{api_key[:8]}...{api_key[-4:]}\",\n        has_credits=False,\n        checked_at=datetime(2026, 4, 6, 12, 0, 0, tzinfo=timezone.utc),\n        stale=False,\n    )\n\n\n# ── db_user_factory fixture ───────────────────────────────────────────────────\n\n\n@pytest.fixture\ndef db_user_factory(client):\n    \"\"\"Factory that seeds the dev_user_local row in the per-test DB.\n\n    Uses the same DB session override that the ``client`` fixture installs\n    on the FastAPI app, so the insert lands in the per-test SQLite file.\n\n    Usage::\n\n        user = db_user_factory()                          # no key\n        user = db_user_factory(preferred_model=\"foo/bar\")\n        user = db_user_factory(encrypted_api_key=b\"x\", key_hint=\"abcd\")\n\n    Returns the created User ORM object so tests can inspect its id.\n    \"\"\"\n    from app.database import get_db\n    from app.main import app\n    from app.models.database_models import User\n\n    def _factory(\n        *,\n        encrypted_api_key=None,\n        key_hint=None,\n        preferred_model=None,\n        key_total_credits=None,\n        key_total_usage=None,\n        key_is_free_tier=None,\n    ):\n        # Use the override that ``client`` installed on the app — that's\n        # the session pointing at the per-test SQLite file.\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            user = User(\n                id=_DEV_USER_ID,\n                email=\"dev@example.com\",\n                role=\"user\",\n            )\n            if encrypted_api_key is not None:\n                user.encrypted_api_key = encrypted_api_key\n            if key_hint is not None:\n                user.key_hint = key_hint\n            if preferred_model is not None:\n                user.preferred_model = preferred_model\n            if key_total_credits is not None:\n                user.key_total_credits = key_total_credits\n            if key_total_usage is not None:\n                user.key_total_usage = key_total_usage\n            if key_is_free_tier is not None:\n                user.key_is_free_tier = key_is_free_tier\n            db.add(user)\n            db.commit()\n            db.refresh(user)\n            return user\n        finally:\n            db.close()\n\n    return _factory\n\n\n# ── POST /settings/api-key ────────────────────────────────────────────────────\n\n\nclass TestPostApiKey:\n    async def test_valid_key_with_credits_returns_200(self, client, db_user_factory):\n        \"\"\"Happy path: post a valid key with credits, get the full settings response back.\"\"\"\n        db_user_factory()  # creates dev_user_local with no key\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-test1234\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_200_OK\n        body = response.json()\n        assert body[\"has_api_key\"] is True\n        assert body[\"key_hint\"] == \"1234\"\n        assert body[\"balance\"][\"balance_remaining\"] == pytest.approx(8.52)\n        assert body[\"balance\"][\"has_credits\"] is True\n        # The endpoint must NOT touch preferred_model\n        assert body[\"preferred_model\"] is None\n\n    async def test_zero_credit_key_returns_400(self, client, db_user_factory):\n        \"\"\"The Baffour Adu case: brand-new key with $0 → reject before saving.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_empty_balance(\"sk-or-v1-zerox0\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-zerox0\"},\n                headers=_AUTH,\n            )\n\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"$0 credits\" in response.json()[\"detail\"]\n\n        # Verify the key was NOT persisted — use the same override the client uses\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n        finally:\n            db.close()\n\n    async def test_unreachable_openrouter_returns_400(self, client, db_user_factory):\n        \"\"\"fetch_balance_sync raises → 400 with 'invalid or unreachable'.\"\"\"\n        db_user_factory()\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            side_effect=OpenRouterBalanceError(\"upstream timeout\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-broken1\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_400_BAD_REQUEST\n        assert \"invalid api key or openrouter is temporarily unreachable\" in (\n            response.json()[\"detail\"].lower()\n        )\n\n    async def test_replacing_existing_key_overwrites_hint(self, client, db_user_factory):\n        \"\"\"Replacing an existing key saves the new hint and discards the old one.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"old-encrypted\",\n            key_hint=\"oldA\",\n        )\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(\"sk-or-v1-newx5678\"),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-newx5678\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"key_hint\"] == \"5678\"\n\n    async def test_does_not_touch_preferred_model(self, client, db_user_factory):\n        \"\"\"Even if a saved standard model exists, this endpoint must leave it alone.\"\"\"\n        db_user_factory(preferred_model=\"meta-llama/llama-4-scout\")\n        with patch(\n            \"app.routes.users.fetch_balance_sync\",\n            return_value=_healthy_balance(),\n        ):\n            response = await client.post(\n                \"/api/users/settings/api-key\",\n                json={\"api_key\": \"sk-or-v1-test1234\"},\n                headers=_AUTH,\n            )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n\n    async def test_blank_key_returns_422(self, client, db_user_factory):\n        \"\"\"Whitespace-only key is rejected by schema validation before hitting the route.\"\"\"\n        db_user_factory()\n        response = await client.post(\n            \"/api/users/settings/api-key\",\n            json={\"api_key\": \"          \"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n\n# ── PUT /settings/preferred-model ────────────────────────────────────────\n\n\nclass TestPutPreferredModel:\n    async def test_no_key_standard_model_returns_200(self, client, db_user_factory):\n        db_user_factory()  # no key\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"meta-llama/llama-4-scout\"\n\n    async def test_no_key_premium_model_returns_403(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_403_FORBIDDEN\n        assert \"openrouter api key\" in response.json()[\"detail\"].lower()\n\n    async def test_with_key_premium_model_returns_200(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"anthropic/claude-sonnet-4.6\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"anthropic/claude-sonnet-4.6\"\n\n    async def test_with_key_standard_model_returns_200(\n        self, client, db_user_factory\n    ):\n        \"\"\"A BYOK user can fall back to a standard model.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"some-encrypted\",\n            key_hint=\"abcd\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"deepseek/deepseek-chat-v3-0324\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"preferred_model\"] == \"deepseek/deepseek-chat-v3-0324\"\n\n    async def test_invalid_format_returns_422(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"no-slash-here\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_422_UNPROCESSABLE_ENTITY\n\n    async def test_does_not_touch_api_key(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"keep-me\",\n            key_hint=\"zzzz\",\n        )\n        response = await client.put(\n            \"/api/users/settings/preferred-model\",\n            json={\"preferred_model\": \"meta-llama/llama-4-scout\"},\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n        assert response.json()[\"has_api_key\"] is True\n        assert response.json()[\"key_hint\"] == \"zzzz\"\n\n\n# ── DELETE /settings/api-key ─────────────────────────────────────────────\n\n\nclass TestDeleteApiKey:\n    async def test_with_key_and_premium_model_resets_to_default(\n        self, client, db_user_factory\n    ):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"anthropic/claude-sonnet-4.6\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.encrypted_api_key is None\n            assert refreshed.key_hint is None\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n        finally:\n            db.close()\n\n    async def test_with_key_and_standard_model_still_resets_to_default(\n        self, client, db_user_factory\n    ):\n        \"\"\"Always reset on delete — even from a non-default standard.\"\"\"\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            preferred_model=\"deepseek/deepseek-chat-v3-0324\",\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.preferred_model == DEFAULT_STANDARD_MODEL\n        finally:\n            db.close()\n\n    async def test_without_key_is_idempotent(self, client, db_user_factory):\n        db_user_factory()\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n    async def test_balance_columns_cleared(self, client, db_user_factory):\n        db_user_factory(\n            encrypted_api_key=b\"will-be-cleared\",\n            key_hint=\"abcd\",\n            key_total_credits=10.0,\n            key_total_usage=2.0,\n            key_is_free_tier=False,\n        )\n        response = await client.delete(\n            \"/api/users/settings/api-key\",\n            headers=_AUTH,\n        )\n        assert response.status_code == status.HTTP_200_OK\n\n        from app.database import get_db\n        from app.main import app\n        from app.models.database_models import User\n\n        override = app.dependency_overrides.get(get_db, get_db)\n        db = next(override())\n        try:\n            refreshed = db.query(User).filter(User.id == _DEV_USER_ID).first()\n            assert refreshed.key_total_credits is None\n            assert refreshed.key_total_usage is None\n            assert refreshed.key_is_free_tier is None\n        finally:\n            db.close()\n"}}, "backend/.env.example": {"6760f774b17e4ad5f6e628622f921d5337703fd6": {"t": 1786068543.1416807, "n": 2015, "h": "f1a75889d36e9680", "c": "# Backend environment variables only.\n# Do NOT add VITE_* variables here — those belong in frontend/.env.\n# See frontend/.env.example for frontend config.\n\n# Cloudflare R2 Storage (S3-compatible)\nR2_ACCESS_KEY_ID=your_r2_access_key_here\nR2_SECRET_ACCESS_KEY=your_r2_secret_key_here\nR2_ENDPOINT_URL=https://your-account-id.r2.cloudflarestorage.com\nR2_BUCKET_NAME=your-bucket-name\n\n# AI API Keys - OpenRouter (OpenAI-compatible)\nOPENROUTER_API_KEY=sk-or-v1-your-key-here\nDEFAULT_MODEL=deepseek/deepseek-chat-v3-0324\nASSEMBLYAI_API_KEY=your_assemblyai_key_here\n\n# Clerk Authentication\nCLERK_SECRET_KEY=sk_test_your_clerk_secret_key_here\nCLERK_PUBLISHABLE_KEY=pk_test_your_clerk_publishable_key_here\n# CLERK_ISSUER pins the expected JWT `iss` claim. When set, tokens minted by\n# the same JWKS for a different Clerk context are rejected. REQUIRED in\n# production (boot fails without it). This is your Clerk Frontend API URL:\n#   - Development instance: https://<slug>.clerk.accounts.dev\n#   - Production custom domain: https://clerk.<your-domain>  (e.g. https://clerk.methodex.ai)\n# Find the exact value in the Clerk Dashboard (API Keys -> \"Frontend API URL\").\n# Leave empty for local dev to skip the issuer check.\nCLERK_ISSUER=\n\n# Database (Local Docker)\nDATABASE_URL=postgresql://postgres:postgres@localhost:5432/qualitative_research\n\n# Redis (Local Docker)\nREDIS_URL=redis://localhost:6379/0\n\n# Application Settings\nAPP_ENV=development\nDEBUG=True\nALLOWED_ORIGINS=http://localhost:5173,http://localhost:3000\n\n# API Settings\nAPI_V1_PREFIX=/api\nPROJECT_NAME=Qualitative Research Tool\n\n# Encryption key for BYOK API keys (generate with: python -c \"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())\")\nENCRYPTION_KEY=\n\n# Sentry Error Monitoring (optional — Sentry is silently disabled when DSN is empty)\nSENTRY_DSN=\nSENTRY_ORG=your_sentry_org_here\nSENTRY_AUTH_TOKEN=your_sentry_auth_token_here\n\n# File Upload Settings\nMAX_FILE_SIZE_MB=500\nALLOWED_VIDEO_EXTENSIONS=[\".mp4\",\".mov\",\".webm\",\".avi\"]\n"}}}}