1 line
33 KiB
JSON
1 line
33 KiB
JSON
{"repo": "boduga/alix", "files": {"tests/manual/README.md": {"4fd792714db3d31bbbf6e9d122d99900443855c1": {"t": 1785747255.7930236, "n": 15968, "h": "bcd0277540d93648", "c": "# ALiX Manual Test Suite\n\n> **Purpose:** Manual test scripts for ALiX v0.2.0-rc.1\n> **Tester:** Run each test step-by-step and mark results\n> **Setup:** `cd <project-root> && npm run build` before running tests\n>\n> **Automated tests also available:** See `tests/manual/suite-*.test.ts`. Run all with:\n> ```bash\n> npm run build && node --test dist/tests/manual/suite-*.test.js\n> ```\n\n---\n\n## How to Use\n\n1. Each test has a **Setup**, **Steps**, and **Expected Result**\n2. Run the command listed, observe the output\n3. Mark `[PASS]` or `[FAIL]` after each test\n4. If a test fails, note the actual output in the **Notes** section\n5. Most tests have an equivalent automated version in `suite-*.test.ts`\n\n---\n\n## Test Suite A: Basic CLI (`alix run`)\n\n### A.1: Hello World (read-only, no plan)\n\n```bash\nnode bin/alix.js run \"echo hello\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- No plan generated (no \"## Summary\" or \"## Changes\" output)\n- Output shows `hello`\n- Session ID printed\n- Completes in < 10 seconds\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.2: List files (read-only, no plan)\n\n```bash\nnode bin/alix.js run \"ls\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- No plan generated\n- Lists files in project root (package.json, src/, tests/, etc.)\n- Completes quickly\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.3: Direct execution with `--no-plan`\n\n```bash\nnode bin/alix.js run \"who is the president of Nigeria\" --session-mode bypass --no-plan\n```\n\n**Expected:**\n- Skips plan generation entirely (no plan output)\n- Returns an answer (may not be accurate without web search)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.4: Development task with plan mode (approve)\n\n```bash\necho \"y\" | node bin/alix.js run \"add a healthz endpoint\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan is generated with Summary, Changes, Verification, Risk Assessment sections\n- Plan printed to stdout\n- After approval (via `echo \"y\"`), execution proceeds\n- Tool calls are made (file reads, patch apply)\n- Summary of work printed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.5: Development task with plan rejection\n\n```bash\necho \"n\" | node bin/alix.js run \"add a healthz endpoint\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan generated and printed\n- Rejected with \"Plan rejected. Task cancelled.\"\n- No tool execution happens\n- Session returned immediately\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.6: Plan detail view (d key)\n\n```bash\nprintf 'd\\ny\\n' | node bin/alix.js run \"add a healthz endpoint\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan printed\n- After pressing `d`, expanded details shown (file counts)\n- Plan re-printed\n- Then `y` approves and executes\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.7: Streaming output\n\n```bash\nnode bin/alix.js run \"echo hello\" --session-mode bypass\n```\n\n**Expected:**\n- Output appears character-by-character or line-by-line (streaming)\n- Visible output before the command completes\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.8: Task with scope expansion (denied)\n\nRun in a fresh test directory, or use `--mode=ask`:\n\n```bash\nnode bin/alix.js run \"create a new file called test.txt with content 'hello'\" --session-mode ask --no-stream\n```\n\n**Expected:**\n- Plan generated and printed\n- Plan approved, execution starts\n- When tool wants to create a file, permission prompt appears\n- Type `n` to deny → task continues without creating the file\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.9: --help flag\n\n```bash\nnode bin/alix.js --help\n```\n\n**Expected:**\n- Shows usage text with all commands\n- Mentions plan mode as default\n- Shows `--no-plan`, `--no-stream`, `--mode=` flags\n- Version displayed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.10: --version flag\n\n```bash\nnode bin/alix.js --version\n```\n\n**Expected:**\n- Prints `0.2.0-rc.1`\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite B: Plan Mode (`--plan` default behavior)\n\n### B.1: Research task auto-approves plan\n\n```bash\nnode bin/alix.js run \"research the best caching strategies\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- No plan approval prompt shown\n- No plan output printed\n- Task executes directly\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### B.2: Plan saved to disk\n\nAfter running A.4, check:\n\n```bash\nls -la .alix/plans/\n```\n\n**Expected:**\n- A `.md` file exists in `.alix/plans/`\n- File name matches the session ID from the run\n- File contains the plan content (Summary, Changes, Verification, Risk Assessment)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### B.3: Empty plan rejection\n\nRun a task with a single character:\n\n```bash\nnode bin/alix.js run \"x\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan generated (or task executed directly if classified as read-only)\n- No crash or error\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite C: Configuration\n\n### C.1: Show config\n\n```bash\nnode bin/alix.js config show\n```\n\n**Expected:**\n- Shows current configuration (model, permissions, etc.)\n- Does not crash\n- Shows provider and model name\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### C.2: Set API key\n\n```bash\n# Non-interactive test (just verify the command exists)\nnode bin/alix.js config set-key 2>&1 | head -5\n```\n\n**Expected:**\n- Shows provider selection menu\n- Interactive prompt for selecting a provider\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### C.3: Set default model\n\n```bash\n# Just verify the command launches the menu\nnode bin/alix.js config set-default-model 2>&1 | head -5\n```\n\n**Expected:**\n- Shows provider selection\n- Fetches models from the selected provider API\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### C.4: Set tier model\n\n```bash\nnode bin/alix.js config set-tier 2>&1 | head -5\n```\n\n**Expected:**\n- Shows tier selection menu (thinking, coding, fast)\n- Then provider selection\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite D: Init\n\n### D.1: Init without prompt (bare init)\n\n```bash\nmkdir -p /tmp/alix-test-init && cd /tmp/alix-test-init\nrm -rf .alix .git AGENTS.md\nnode /path/to/alix/bin/alix.js init\n```\n\n**Expected:**\n- `.alix/config.json` created\n- `AGENTS.md` created\n- Git initialized (if not already)\n- `.gitignore` updated with `.alix/` entry\n- Project type detected (Node.js if package.json exists, Generic otherwise)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### D.2: Init with scaffold prompt (requires provider)\n\n```bash\ncd /tmp && mkdir -p alix-test-scaffold && cd alix-test-scaffold\nnode /path/to/alix/bin/alix.js init \"create a Fastify API server with TypeScript\"\n```\n\n**Expected:**\n- `.alix/config.json` created\n- Agent runs the scaffold task\n- Fastify project files created (package.json, tsconfig.json, src/ directory)\n- `Session:` printed at the end\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite E: Chat Mode\n\n### E.1: Start chat session\n\n```bash\necho \"/exit\" | node bin/alix.js chat --session-mode bypass\n```\n\n**Expected:**\n- Chat session starts\n- Prints session ID\n- Shows \"Type /exit or /quit to end\"\n- Exits cleanly when `/exit` typed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### E.2: Ask a question in chat\n\n```bash\nprintf 'what is 2+2?\\n/exit\\n' | node bin/alix.js chat --session-mode bypass\n```\n\n**Expected:**\n- Question sent to model\n- Response printed\n- Session saved\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### E.3: Chat commands\n\n```bash\nprintf '/help\\n/exit\\n' | node bin/alix.js chat\n```\n\n**Expected:**\n- Shows available commands: /exit, /quit, /clear, /context, /model, /remember, /task, /decision\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### E.4: Session listing\n\n```bash\nnode bin/alix.js chat --list\n```\n\n**Expected:**\n- Lists recent sessions with message count and timestamp\n- Or \"No sessions found.\" if none exist\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite F: Memory\n\n### F.1: List memory\n\n```bash\nnode bin/alix.js memory list\n```\n\n**Expected:**\n- Lists all memory entries\n- Or empty list if none\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### F.2: Add memory\n\n```bash\nnode bin/alix.js memory add --name test-note --content \"This is a test memory entry\"\n```\n\n**Expected:**\n- Memory entry added\n- No crash\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### F.3: Memory search\n\n```bash\nnode bin/alix.js memory list --query test\n```\n\n**Expected:**\n- Searches memory entries\n- Returns matching results\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite G: MCP\n\n### G.1: List MCP servers\n\n```bash\nnode bin/alix.js mcp list\n```\n\n**Expected:**\n- Lists connected MCP servers\n- Shows the default `fetch` server\n- Shows tools available from each server\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### G.2: MCP test (fetch)\n\n```bash\nnode bin/alix.js mcp test fetch\n```\n\n**Expected:**\n- Tests the fetch MCP server connection\n- Reports success or failure\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### G.3: Discover MCP package\n\n```bash\nnode bin/alix.js mcp discover mcp-server-fetch\n```\n\n**Expected:**\n- Searches npm for the MCP package\n- Shows package information\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite H: Extensions\n\n### H.1: List extensions\n\n```bash\nnode bin/alix.js extension list\n```\n\n**Expected:**\n- Lists installed extensions (skills, hooks)\n- Shows their names and descriptions\n- Or shows empty list if none installed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### H.2: Search extensions (requires internet)\n\n```bash\nnode bin/alix.js extension search test\n```\n\n**Expected:**\n- Searches extension registry\n- Shows results or \"no results\"\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite I: Inspector (Server)\n\n### I.1: Start inspector server\n\n```bash\ntimeout 5 node bin/alix.js serve 2>&1 || true\n```\n\n**Expected:**\n- Server starts\n- Shows URL: `http://127.0.0.1:4137`\n- Logs incoming connections\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### I.2: Inspector health check\n\n```bash\nnode bin/alix.js serve &\nsleep 2\ncurl http://127.0.0.1:4137/api/health 2>/dev/null || echo \"Health endpoint not found\"\nkill %1 2>/dev/null\n```\n\n**Expected:**\n- Health endpoint returns OK\n- Server responds\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite J: TUI Mode\n\n### J.1: TUI launches\n\n```bash\n# Press q immediately to quit\necho \"q\" | timeout 5 node bin/alix.js tui 2>&1 || true\n```\n\n**Expected:**\n- TUI starts (may need a real terminal)\n- Shows session interface\n- Exits cleanly\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite K: Agent (Subagent)\n\n### K.1: Spawn explorer subagent\n\n```bash\nnode bin/alix.js agent explorer \"list the files in src/\" --session-mode bypass 2>&1\n```\n\n**Expected:**\n- Subagent spawned\n- Returns findings about src/ directory\n- Role is \"explorer\" (read-only)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### K.2: Spawn reviewer subagent\n\n```bash\nnode bin/alix.js agent reviewer \"review src/task-classifier.ts\" --session-mode bypass 2>&1\n```\n\n**Expected:**\n- Subagent spawned\n- Reviews the file\n- Returns feedback\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite L: Error Handling\n\n### L.1: Empty task\n\n```bash\nnode bin/alix.js run \"\"\n```\n\n**Expected:**\n- Error message shown\n- Non-zero exit code\n- Does not crash or hang\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### L.2: Unknown command\n\n```bash\nnode bin/alix.js unknowncommand\n```\n\n**Expected:**\n- Shows help text\n- Does not crash\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### L.3: Invalid flag combination\n\n```bash\nnode bin/alix.js run \"test\" --no-plan --mode=invalid\n```\n\n**Expected:**\n- If mode is invalid, task may still proceed with default session mode\n- No crash or unhandled exception\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### L.4: Interrupt during plan (Ctrl+C)\n\n```bash\n# Run in background, kill after 2 seconds\ntimeout 3 node bin/alix.js run \"add a healthz endpoint\" 2>&1 || true\n```\n\n**Expected:**\n- Plan generated\n- On timeout/SIGINT, process exits cleanly\n- No zombie processes\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite M: Cross-Feature\n\n### M.1: Init then run in same directory\n\n```bash\ncd /tmp && mkdir -p alix-cross-test && cd alix-cross-test\n/path/to/alix/bin/alix.js init\n/path/to/alix/bin/alix.js run \"echo 'hello from alix'\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Init creates `.alix/config.json`\n- Run uses the config to execute the task\n- Output shows `hello from alix`\n- Session logged to `.alix/sessions/`\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### M.2: Session replay via inspector\n\n```bash\n# Run a task first\nnode bin/alix.js run \"ls\" --session-mode bypass --no-stream\n# Note the session ID from output\n# Start inspector\nnode bin/alix.js serve &\nsleep 2\n# Open inspector URL in browser: http://127.0.0.1:4137\n# Find the session in the Sessions list\n# Click to view session replay\nkill %1 2>/dev/null\n```\n\n**Expected:**\n- Inspector shows the session in the list\n- Session events are replayable (tool calls, outputs, decisions)\n- $ cost is calculated and displayed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite N: Web Tools\n\n### N.1: Web search (requires BRAVE_API_KEY)\n\n```bash\nBRAVE_API_KEY=\"your-key-here\" node bin/alix.js run \"search the web for latest AI news\" --session-mode bypass --no-plan --no-stream\n```\n\n**Expected:**\n- Agent uses web_search tool\n- Returns search results\n- Responds based on results\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### N.2: Web fetch (requires BRAVE_API_KEY)\n\n```bash\nBRAVE_API_KEY=\"your-key-here\" node bin/alix.js run \"fetch https://example.com and summarize the page\" --session-mode bypass --no-plan --no-stream\n```\n\n**Expected:**\n- Agent uses web_fetch tool\n- Fetches page content\n- Summarizes the content\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite O: Long-Running / Stability\n\n### O.1: Multiple rapid tasks\n\n```bash\nfor i in 1 2 3 4 5; do\n node bin/alix.js run \"echo task $i\" --session-mode bypass --no-stream\ndone\n```\n\n**Expected:**\n- All 5 tasks complete successfully\n- No memory leaks or crashes\n- Each gets its own session ID\n- Sessions directory has 5 new entries\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### O.2: Session persistence between tasks\n\n```bash\n# Run first task\nnode bin/alix.js run \"echo first task\" --session-mode bypass --no-stream\n# Run second task — verify sessions accumulate\nls .alix/sessions/ | wc -l\n```\n\n**Expected:**\n- Session count increases with each run\n- Events JSONL files are present in each session directory\n- No session files are corrupted\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Regression Checklist\n\nRun before every release:\n\n| # | Test | Status |\n|---|------|--------|\n| A.1 | Hello World (echo) | `[ ]` |\n| A.2 | List files (ls) | `[ ]` |\n| A.4 | Plan + approve | `[ ]` |\n| A.5 | Plan + reject | `[ ]` |\n| A.9 | Help text | `[ ]` |\n| A.10 | Version | `[ ]` |\n| C.1 | Config show | `[ ]` |\n| D.1 | Init | `[ ]` |\n| E.1 | Chat start/exit | `[ ]` |\n| L.1 | Empty task error | `[ ]` |\n| L.2 | Unknown command | `[ ]` |\n| O.1 | Rapid tasks | `[ ]` |\n\n---\n\n## Bug Report Template\n\n```\n## Bug Report\n\n**Test:** <test-id> (e.g., A.3)\n\n**Command:**\n```\n<command that failed>\n```\n\n**Expected:**\n<what should happen>\n\n**Actual:**\n<what actually happened>\n\n**Environment:**\n- OS: <linux/mac/windows>\n- Node version: <node -v>\n- ALiX version: <node bin/alix.js --version>\n- Model provider: <deepseek/google/openai>\n\n**Session ID:** <session-id from output>\n\n**Session Events:**\n```\n<tail -50 .alix/sessions/<id>/events.jsonl>\n```\n```\n"}, "3852dedfa48f35e25e4a1c3f2af58d8f0fdd9ec3": {"t": 1785771868.3096893, "n": 15968, "h": "bcd0277540d93648", "c": "# ALiX Manual Test Suite\n\n> **Purpose:** Manual test scripts for ALiX v0.2.0-rc.1\n> **Tester:** Run each test step-by-step and mark results\n> **Setup:** `cd <project-root> && npm run build` before running tests\n>\n> **Automated tests also available:** See `tests/manual/suite-*.test.ts`. Run all with:\n> ```bash\n> npm run build && node --test dist/tests/manual/suite-*.test.js\n> ```\n\n---\n\n## How to Use\n\n1. Each test has a **Setup**, **Steps**, and **Expected Result**\n2. Run the command listed, observe the output\n3. Mark `[PASS]` or `[FAIL]` after each test\n4. If a test fails, note the actual output in the **Notes** section\n5. Most tests have an equivalent automated version in `suite-*.test.ts`\n\n---\n\n## Test Suite A: Basic CLI (`alix run`)\n\n### A.1: Hello World (read-only, no plan)\n\n```bash\nnode bin/alix.js run \"echo hello\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- No plan generated (no \"## Summary\" or \"## Changes\" output)\n- Output shows `hello`\n- Session ID printed\n- Completes in < 10 seconds\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.2: List files (read-only, no plan)\n\n```bash\nnode bin/alix.js run \"ls\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- No plan generated\n- Lists files in project root (package.json, src/, tests/, etc.)\n- Completes quickly\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.3: Direct execution with `--no-plan`\n\n```bash\nnode bin/alix.js run \"who is the president of Nigeria\" --session-mode bypass --no-plan\n```\n\n**Expected:**\n- Skips plan generation entirely (no plan output)\n- Returns an answer (may not be accurate without web search)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.4: Development task with plan mode (approve)\n\n```bash\necho \"y\" | node bin/alix.js run \"add a healthz endpoint\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan is generated with Summary, Changes, Verification, Risk Assessment sections\n- Plan printed to stdout\n- After approval (via `echo \"y\"`), execution proceeds\n- Tool calls are made (file reads, patch apply)\n- Summary of work printed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.5: Development task with plan rejection\n\n```bash\necho \"n\" | node bin/alix.js run \"add a healthz endpoint\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan generated and printed\n- Rejected with \"Plan rejected. Task cancelled.\"\n- No tool execution happens\n- Session returned immediately\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.6: Plan detail view (d key)\n\n```bash\nprintf 'd\\ny\\n' | node bin/alix.js run \"add a healthz endpoint\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan printed\n- After pressing `d`, expanded details shown (file counts)\n- Plan re-printed\n- Then `y` approves and executes\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.7: Streaming output\n\n```bash\nnode bin/alix.js run \"echo hello\" --session-mode bypass\n```\n\n**Expected:**\n- Output appears character-by-character or line-by-line (streaming)\n- Visible output before the command completes\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.8: Task with scope expansion (denied)\n\nRun in a fresh test directory, or use `--mode=ask`:\n\n```bash\nnode bin/alix.js run \"create a new file called test.txt with content 'hello'\" --session-mode ask --no-stream\n```\n\n**Expected:**\n- Plan generated and printed\n- Plan approved, execution starts\n- When tool wants to create a file, permission prompt appears\n- Type `n` to deny → task continues without creating the file\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.9: --help flag\n\n```bash\nnode bin/alix.js --help\n```\n\n**Expected:**\n- Shows usage text with all commands\n- Mentions plan mode as default\n- Shows `--no-plan`, `--no-stream`, `--mode=` flags\n- Version displayed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### A.10: --version flag\n\n```bash\nnode bin/alix.js --version\n```\n\n**Expected:**\n- Prints `0.2.0-rc.1`\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite B: Plan Mode (`--plan` default behavior)\n\n### B.1: Research task auto-approves plan\n\n```bash\nnode bin/alix.js run \"research the best caching strategies\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- No plan approval prompt shown\n- No plan output printed\n- Task executes directly\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### B.2: Plan saved to disk\n\nAfter running A.4, check:\n\n```bash\nls -la .alix/plans/\n```\n\n**Expected:**\n- A `.md` file exists in `.alix/plans/`\n- File name matches the session ID from the run\n- File contains the plan content (Summary, Changes, Verification, Risk Assessment)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### B.3: Empty plan rejection\n\nRun a task with a single character:\n\n```bash\nnode bin/alix.js run \"x\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Plan generated (or task executed directly if classified as read-only)\n- No crash or error\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite C: Configuration\n\n### C.1: Show config\n\n```bash\nnode bin/alix.js config show\n```\n\n**Expected:**\n- Shows current configuration (model, permissions, etc.)\n- Does not crash\n- Shows provider and model name\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### C.2: Set API key\n\n```bash\n# Non-interactive test (just verify the command exists)\nnode bin/alix.js config set-key 2>&1 | head -5\n```\n\n**Expected:**\n- Shows provider selection menu\n- Interactive prompt for selecting a provider\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### C.3: Set default model\n\n```bash\n# Just verify the command launches the menu\nnode bin/alix.js config set-default-model 2>&1 | head -5\n```\n\n**Expected:**\n- Shows provider selection\n- Fetches models from the selected provider API\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### C.4: Set tier model\n\n```bash\nnode bin/alix.js config set-tier 2>&1 | head -5\n```\n\n**Expected:**\n- Shows tier selection menu (thinking, coding, fast)\n- Then provider selection\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite D: Init\n\n### D.1: Init without prompt (bare init)\n\n```bash\nmkdir -p /tmp/alix-test-init && cd /tmp/alix-test-init\nrm -rf .alix .git AGENTS.md\nnode /path/to/alix/bin/alix.js init\n```\n\n**Expected:**\n- `.alix/config.json` created\n- `AGENTS.md` created\n- Git initialized (if not already)\n- `.gitignore` updated with `.alix/` entry\n- Project type detected (Node.js if package.json exists, Generic otherwise)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### D.2: Init with scaffold prompt (requires provider)\n\n```bash\ncd /tmp && mkdir -p alix-test-scaffold && cd alix-test-scaffold\nnode /path/to/alix/bin/alix.js init \"create a Fastify API server with TypeScript\"\n```\n\n**Expected:**\n- `.alix/config.json` created\n- Agent runs the scaffold task\n- Fastify project files created (package.json, tsconfig.json, src/ directory)\n- `Session:` printed at the end\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite E: Chat Mode\n\n### E.1: Start chat session\n\n```bash\necho \"/exit\" | node bin/alix.js chat --session-mode bypass\n```\n\n**Expected:**\n- Chat session starts\n- Prints session ID\n- Shows \"Type /exit or /quit to end\"\n- Exits cleanly when `/exit` typed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### E.2: Ask a question in chat\n\n```bash\nprintf 'what is 2+2?\\n/exit\\n' | node bin/alix.js chat --session-mode bypass\n```\n\n**Expected:**\n- Question sent to model\n- Response printed\n- Session saved\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### E.3: Chat commands\n\n```bash\nprintf '/help\\n/exit\\n' | node bin/alix.js chat\n```\n\n**Expected:**\n- Shows available commands: /exit, /quit, /clear, /context, /model, /remember, /task, /decision\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### E.4: Session listing\n\n```bash\nnode bin/alix.js chat --list\n```\n\n**Expected:**\n- Lists recent sessions with message count and timestamp\n- Or \"No sessions found.\" if none exist\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite F: Memory\n\n### F.1: List memory\n\n```bash\nnode bin/alix.js memory list\n```\n\n**Expected:**\n- Lists all memory entries\n- Or empty list if none\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### F.2: Add memory\n\n```bash\nnode bin/alix.js memory add --name test-note --content \"This is a test memory entry\"\n```\n\n**Expected:**\n- Memory entry added\n- No crash\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### F.3: Memory search\n\n```bash\nnode bin/alix.js memory list --query test\n```\n\n**Expected:**\n- Searches memory entries\n- Returns matching results\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite G: MCP\n\n### G.1: List MCP servers\n\n```bash\nnode bin/alix.js mcp list\n```\n\n**Expected:**\n- Lists connected MCP servers\n- Shows the default `fetch` server\n- Shows tools available from each server\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### G.2: MCP test (fetch)\n\n```bash\nnode bin/alix.js mcp test fetch\n```\n\n**Expected:**\n- Tests the fetch MCP server connection\n- Reports success or failure\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### G.3: Discover MCP package\n\n```bash\nnode bin/alix.js mcp discover mcp-server-fetch\n```\n\n**Expected:**\n- Searches npm for the MCP package\n- Shows package information\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite H: Extensions\n\n### H.1: List extensions\n\n```bash\nnode bin/alix.js extension list\n```\n\n**Expected:**\n- Lists installed extensions (skills, hooks)\n- Shows their names and descriptions\n- Or shows empty list if none installed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### H.2: Search extensions (requires internet)\n\n```bash\nnode bin/alix.js extension search test\n```\n\n**Expected:**\n- Searches extension registry\n- Shows results or \"no results\"\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite I: Inspector (Server)\n\n### I.1: Start inspector server\n\n```bash\ntimeout 5 node bin/alix.js serve 2>&1 || true\n```\n\n**Expected:**\n- Server starts\n- Shows URL: `http://127.0.0.1:4137`\n- Logs incoming connections\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### I.2: Inspector health check\n\n```bash\nnode bin/alix.js serve &\nsleep 2\ncurl http://127.0.0.1:4137/api/health 2>/dev/null || echo \"Health endpoint not found\"\nkill %1 2>/dev/null\n```\n\n**Expected:**\n- Health endpoint returns OK\n- Server responds\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite J: TUI Mode\n\n### J.1: TUI launches\n\n```bash\n# Press q immediately to quit\necho \"q\" | timeout 5 node bin/alix.js tui 2>&1 || true\n```\n\n**Expected:**\n- TUI starts (may need a real terminal)\n- Shows session interface\n- Exits cleanly\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite K: Agent (Subagent)\n\n### K.1: Spawn explorer subagent\n\n```bash\nnode bin/alix.js agent explorer \"list the files in src/\" --session-mode bypass 2>&1\n```\n\n**Expected:**\n- Subagent spawned\n- Returns findings about src/ directory\n- Role is \"explorer\" (read-only)\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### K.2: Spawn reviewer subagent\n\n```bash\nnode bin/alix.js agent reviewer \"review src/task-classifier.ts\" --session-mode bypass 2>&1\n```\n\n**Expected:**\n- Subagent spawned\n- Reviews the file\n- Returns feedback\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite L: Error Handling\n\n### L.1: Empty task\n\n```bash\nnode bin/alix.js run \"\"\n```\n\n**Expected:**\n- Error message shown\n- Non-zero exit code\n- Does not crash or hang\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### L.2: Unknown command\n\n```bash\nnode bin/alix.js unknowncommand\n```\n\n**Expected:**\n- Shows help text\n- Does not crash\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### L.3: Invalid flag combination\n\n```bash\nnode bin/alix.js run \"test\" --no-plan --mode=invalid\n```\n\n**Expected:**\n- If mode is invalid, task may still proceed with default session mode\n- No crash or unhandled exception\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### L.4: Interrupt during plan (Ctrl+C)\n\n```bash\n# Run in background, kill after 2 seconds\ntimeout 3 node bin/alix.js run \"add a healthz endpoint\" 2>&1 || true\n```\n\n**Expected:**\n- Plan generated\n- On timeout/SIGINT, process exits cleanly\n- No zombie processes\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite M: Cross-Feature\n\n### M.1: Init then run in same directory\n\n```bash\ncd /tmp && mkdir -p alix-cross-test && cd alix-cross-test\n/path/to/alix/bin/alix.js init\n/path/to/alix/bin/alix.js run \"echo 'hello from alix'\" --session-mode bypass --no-stream\n```\n\n**Expected:**\n- Init creates `.alix/config.json`\n- Run uses the config to execute the task\n- Output shows `hello from alix`\n- Session logged to `.alix/sessions/`\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### M.2: Session replay via inspector\n\n```bash\n# Run a task first\nnode bin/alix.js run \"ls\" --session-mode bypass --no-stream\n# Note the session ID from output\n# Start inspector\nnode bin/alix.js serve &\nsleep 2\n# Open inspector URL in browser: http://127.0.0.1:4137\n# Find the session in the Sessions list\n# Click to view session replay\nkill %1 2>/dev/null\n```\n\n**Expected:**\n- Inspector shows the session in the list\n- Session events are replayable (tool calls, outputs, decisions)\n- $ cost is calculated and displayed\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite N: Web Tools\n\n### N.1: Web search (requires BRAVE_API_KEY)\n\n```bash\nBRAVE_API_KEY=\"your-key-here\" node bin/alix.js run \"search the web for latest AI news\" --session-mode bypass --no-plan --no-stream\n```\n\n**Expected:**\n- Agent uses web_search tool\n- Returns search results\n- Responds based on results\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### N.2: Web fetch (requires BRAVE_API_KEY)\n\n```bash\nBRAVE_API_KEY=\"your-key-here\" node bin/alix.js run \"fetch https://example.com and summarize the page\" --session-mode bypass --no-plan --no-stream\n```\n\n**Expected:**\n- Agent uses web_fetch tool\n- Fetches page content\n- Summarizes the content\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Test Suite O: Long-Running / Stability\n\n### O.1: Multiple rapid tasks\n\n```bash\nfor i in 1 2 3 4 5; do\n node bin/alix.js run \"echo task $i\" --session-mode bypass --no-stream\ndone\n```\n\n**Expected:**\n- All 5 tasks complete successfully\n- No memory leaks or crashes\n- Each gets its own session ID\n- Sessions directory has 5 new entries\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n### O.2: Session persistence between tasks\n\n```bash\n# Run first task\nnode bin/alix.js run \"echo first task\" --session-mode bypass --no-stream\n# Run second task — verify sessions accumulate\nls .alix/sessions/ | wc -l\n```\n\n**Expected:**\n- Session count increases with each run\n- Events JSONL files are present in each session directory\n- No session files are corrupted\n\n**Result:** `[ ]` PASS / `[ ]` FAIL\n**Notes:**\n\n---\n\n## Regression Checklist\n\nRun before every release:\n\n| # | Test | Status |\n|---|------|--------|\n| A.1 | Hello World (echo) | `[ ]` |\n| A.2 | List files (ls) | `[ ]` |\n| A.4 | Plan + approve | `[ ]` |\n| A.5 | Plan + reject | `[ ]` |\n| A.9 | Help text | `[ ]` |\n| A.10 | Version | `[ ]` |\n| C.1 | Config show | `[ ]` |\n| D.1 | Init | `[ ]` |\n| E.1 | Chat start/exit | `[ ]` |\n| L.1 | Empty task error | `[ ]` |\n| L.2 | Unknown command | `[ ]` |\n| O.1 | Rapid tasks | `[ ]` |\n\n---\n\n## Bug Report Template\n\n```\n## Bug Report\n\n**Test:** <test-id> (e.g., A.3)\n\n**Command:**\n```\n<command that failed>\n```\n\n**Expected:**\n<what should happen>\n\n**Actual:**\n<what actually happened>\n\n**Environment:**\n- OS: <linux/mac/windows>\n- Node version: <node -v>\n- ALiX version: <node bin/alix.js --version>\n- Model provider: <deepseek/google/openai>\n\n**Session ID:** <session-id from output>\n\n**Session Events:**\n```\n<tail -50 .alix/sessions/<id>/events.jsonl>\n```\n```\n"}}}} |