{"repo": "alihusn3392/claude-code-from-scratch", "files": {"en/docs/14-testing.md": {"46f5ca9cbe7f1772a69222b2059e9cc611ab2459": {"t": 1785809912.0846655, "n": 20690, "h": "e3d7018b85afb8ac", "c": "# 14. Functional Testing Guide\n\n## Chapter Goals\n\nVerify that mini-claude's 19 core features all work correctly. All tests are manual execution + visual verification, all using `--yolo` mode (skip permission confirmations).\n\n```mermaid\ngraph LR\n    Setup[\"bash test/setup.sh\"] --> Build[\"npm run build (TS version)\"]\n    Build --> Test[\"Run tests one by one\"]\n    Test --> Cleanup[\"bash test/cleanup.sh\"]\n\n    style Setup fill:#7c5cfc,color:#fff\n    style Test fill:#e8e0ff\n```\n\n## Why Manual Testing Is Needed\n\nTesting a coding agent is different from testing regular software -- core behavior depends on LLM responses, and output is non-deterministic. Automated unit tests can cover tool functions (file I/O, permission checks), but end-to-end Agent behavior can only be observed manually:\n\n- Did the model choose the right tool?\n- Is parallel execution actually parallel?\n- Is semantic memory recall timing correct?\n- Is the Plan Mode approval workflow interaction smooth?\n\nClaude Code itself uses a similar strategy: core tools have unit tests, but Agent behavior relies on manual QA + evaluation suites (eval suite).\n\n## Preparation\n\n```bash\ncd claude-code-from-scratch\n\n# One-command setup for test environment (MCP, Skills, CLAUDE.md, large file, quote test file, custom Agent)\nbash test/setup.sh\n\n# Build TS version (Python version doesn't need building)\nnpm run build\n```\n\nMake sure `.env` has API keys configured:\n```\nANTHROPIC_API_KEY=sk-xxx\nANTHROPIC_BASE_URL=https://aihubmix.com   # Optional\n```\n\n> **Tip**: If the system environment has both `OPENAI_API_KEY` + `OPENAI_BASE_URL` and `ANTHROPIC_API_KEY`,\n> it will prefer the OpenAI-compatible path. Both paths support all features.\n\n## How to Launch\n\n**TS version**:\n```bash\n# Interactive REPL (recommended; can test skill, plan mode, and REPL commands)\nnode dist/cli.js --yolo\n\n# One-shot mode\nnode dist/cli.js --yolo \"your prompt\"\n```\n\n**Python version**:\n```bash\npython -m mini_claude --yolo\n\n# One-shot mode\npython -m mini_claude --yolo \"your prompt\"\n```\n\n> The following test steps use the TS version in command-line examples. For the Python version, replace `node dist/cli.js` with `python -m mini_claude` -- functionality is identical.\n\n---\n\n## Phase 1: Basic Tools (Test 1-3)\n\n### 1. MCP Tool Invocation\n\n**Test objective**: Verify MCP server connection + tool discovery + transparent routing.\n\n**Expected**: On startup, see `[mcp] Connected to 'test' — 3 tools`\n\n```\nUse the MCP 'add' tool to compute 17+25, then use the 'echo' tool to echo \"hello MCP\", then use the 'timestamp' tool.\n```\n\nPass criteria:\n- add returns `42`\n- echo returns `hello MCP`\n- timestamp returns a Unix timestamp\n- Tool names have the `mcp__test__` prefix\n\n**Design intent**: MCP is the core mechanism for extending Agent capabilities. The three-segment naming `mcp__server__tool` solves both naming conflicts and routing -- you can tell from the name which server to forward to.\n\n---\n\n### 2. WebFetch\n\n**Test objective**: Verify HTTP fetching + HTML cleaning.\n\n```\nFetch the URL https://httpbin.org/json and tell me the slideshow title.\n```\n\nPass criteria: Returns `Sample Slide Show`\n\n```\nFetch https://example.com and tell me what the page is about.\n```\n\nPass criteria: Returns plain text content converted from HTML\n\n---\n\n### 3. Parallel Tool Execution\n\n**Test objective**: Verify that concurrency-safe tools can execute simultaneously (not serially).\n\n```\nRead the files src/frontmatter.ts, src/session.ts, and src/skills.ts at the same time, then tell me each file's line count.\n```\n\nPython version alternative -- read Python files:\n```\nRead the files python/mini_claude/frontmatter.py and python/mini_claude/session.py at the same time, then tell me each file's line count.\n```\n\nPass criteria: Multiple `read_file` calls appear simultaneously (not one after another)\n\n**Design intent**: `CONCURRENCY_SAFE_TOOLS` (read_file, list_files, grep_search, web_fetch) are marked as parallelizable. The Agent starts executing these tools during the streaming output phase without waiting for the model to finish generating.\n\n---\n\n## Phase 2: Memory and Context (Test 4-7)\n\n### 4. Semantic Memory Recall\n\n**Test objective**: Verify memory saving -> semantic recall in a new conversation (async prefetch mechanism).\n\n**Step 1: Save memories**\n```\nSave these memories for me:\n1. type=project, name=\"API migration\", description=\"Moving from REST to GraphQL\", content=\"We are migrating our API from REST to GraphQL. Deadline is end of Q2 2025.\"\n2. type=feedback, name=\"code style\", description=\"Prefers functional programming\", content=\"User prefers functional patterns (map/filter/reduce) over for loops and OOP.\"\n3. type=reference, name=\"staging server\", description=\"Staging environment URL\", content=\"Staging server: https://staging.example.com, credentials in 1Password.\"\n```\n\nPass criteria: Three memory files are written\n\n**Step 2: Exit and start a new conversation**, then enter a query that triggers tool calls:\n\n> **How it works**: Semantic recall uses async prefetch (consistent with Claude Code behavior, zero-wait non-blocking).\n> Prefetch starts when the user message is sent and takes a few seconds to complete. If the model responds\n> with text directly without calling tools, the loop runs only once and finishes before prefetch can be consumed.\n> Therefore, test queries need to trigger tool calls to give prefetch enough time to be injected in the second iteration.\n\n```\nRead the file tsconfig.json, then tell me: where can I deploy to test my changes?\n```\nPass criteria: Recalls the staging server memory, answers `https://staging.example.com`\n\n```\nList the files in the src/ directory, then tell me: what's the deadline for the backend rewrite?\n```\nPass criteria: Recalls the API migration memory, answers `end of Q2 2025`\n\n```\nRead package.json, then tell me: how should I write code for this project?\n```\nPass criteria: Recalls the code style memory, mentions functional programming\n\n---\n\n### 5. @include Directive + Rules Auto-Loading\n\n**Test objective**: Verify CLAUDE.md's `@path` include directive and `.claude/rules/` auto-loading.\n\nsetup.sh has already created:\n- `CLAUDE.md` containing `@./.claude/rules/chinese-greeting.md`\n- Rule content: `When the user greets you, respond in Chinese`\n\n```\nHello! Who are you?\n```\n\nPass criteria: The model responds in **Chinese** (because the rule requires greeting in Chinese)\n\n**Design intent**: The `@include` mechanism supports three formats: `@./relative-path`, `@~/home-path`, and `@/absolute-path`, with circular reference detection and a maximum depth limit (5 levels). All `.md` files in the rules directory are sorted alphabetically and concatenated into the system prompt.\n\n---\n\n### 6. Read-Before-Edit Protection\n\n**Test objective**: Verify the safety check when editing an unread file.\n\n```\nEdit the file package.json and change the version to \"9.9.9\". Do NOT read it first.\n```\n\nPass criteria (either counts as passing):\n- **Best**: Tool layer directly returns `Error: You must read this file before editing`\n- **Acceptable**: The model, prompted by the system prompt, automatically reads before editing\n\nAfter testing, restore:\n```\nNow change it back to \"1.0.0\".\n```\n\n---\n\n### 7. Large Result Persistence\n\n**Test objective**: Verify that oversized tool results are written to disk + preview truncation.\n\n```\nRead the file test/large-file.txt\n```\n\nPass criteria (output includes):\n- `[Result too large (XX.X KB, 1000 lines). Full output saved to ...]`\n- `Preview (first 200 lines):`\n- Only the first 200 lines of preview are shown\n\nThen continue asking:\n```\nWhat does line 500 say?\n```\n\nPass criteria: The model uses grep_search or read_file to find the content of Line 499 from the original file\n\n**Design intent**: Tool results exceeding 30KB are written to `~/.mini-claude/tool-results/`; only a preview is kept in the conversation. This prevents a single large file from blowing up the entire context window. Aligned with Claude Code's `LargeResultPersistence` logic.\n\n---\n\n## Phase 3: Skills and Tool Extensions (Test 8-10)\n\n### 8. Skill Invocation\n\n**Test objective**: Verify skill discovery, inline invocation, and slash commands.\n\n```\n/skills\n```\nPass criteria: Lists the greet and commit skills\n\n```\n/greet Alice\n```\nPass criteria: The model generates a personalized greeting for Alice\n\n```\n/commit\n```\nPass criteria: The model runs git diff/status, then attempts to create a commit\n\n---\n\n### 9. ToolSearch / Lazy-Loaded Tools\n\n**Test objective**: Verify the deferred tool mechanism -- plan mode tools initially don't send schemas and are only activated after being searched.\n\n```\nUse tool_search to find the \"plan mode\" tool.\n```\n\nPass criteria:\n- The model calls `tool_search`\n- Returns the full schema for `enter_plan_mode` and/or `exit_plan_mode`\n- These tools were not in the tool list before and are only activated after being searched\n\n**Design intent**: Deferred tools reduce the size of tool schemas sent with each API call. Claude Code has 60+ tools, but most scenarios only use 5-6. Sending all schemas wastes tokens; lazy loading activates on demand.\n\n---\n\n### 10. REPL Commands\n\n```\n/cost\n```\nPass criteria: Shows token usage and cost\n\n```\n/memory\n```\nPass criteria: Lists saved memories\n\n```\n/compact\n```\nPass criteria: Manually triggers conversation compression\n\n```\n/plan\n```\nPass criteria: Switches to plan mode (enter again to switch back)\n\n---\n\n## Phase 4: Agent Architecture (Test 11-12)\n\n### 11. Sub-Agent System (Agent Tool)\n\n**Test objective**: Verify isolated execution and tool restrictions for the three built-in agent types.\n\n**explore agent** (read-only search):\n```\nUse the agent tool with type \"explore\" to find all files that import from \"./memory.js\" in the src/ directory.\n```\n\nPass criteria:\n- Output shows the `[sub-agent:explore]` marker\n- Returns a list of files referencing `memory.js`\n- Only uses read_file / list_files / grep_search\n\n**plan agent** (structured planning):\n```\nUse the agent tool with type \"plan\" to design a plan for adding a \"help\" REPL command. Identify which files need modification.\n```\n\nPass criteria: Output shows the `[sub-agent:plan]` marker, returns a structured modification plan\n\n**general agent** (full tools):\n```\nUse the agent tool with type \"general\" to create a file called /tmp/mini-claude-agent-test.txt with the content \"agent test passed\", then read it back.\n```\n\nPass criteria:\n- Output shows the `[sub-agent:general]` marker\n- Successfully creates and reads the file\n- Sub-agent token consumption is added to the main agent (visible in `/cost`)\n\n**Design intent**: Sub-agents are Claude Code's \"divide and conquer\" strategy -- breaking large tasks into sub-agents, each with isolated context that doesn't pollute the main conversation. The explore agent is restricted to read-only tools to prevent accidental modifications; the general agent excludes the agent tool to prevent infinite recursion.\n\n---\n\n### 12. Plan Mode (Manual Entry)\n\n**Test objective**: Verify `/plan` toggle + read-only restriction + plan file writing + approval workflow.\n\n**Step 1: Enter plan mode**\n```\n/plan\n```\nPass criteria: Shows that plan mode is enabled\n\n**Step 2: Test read-only restriction**\n```\nRead package.json, then create a plan for changing the project name. Write your plan to the plan file.\n```\n\nPass criteria:\n- The model can read package.json (read tools are always allowed)\n- The model writes to the plan file (the only file allowed for editing)\n- If it tries to edit other files, it's rejected: `Blocked in plan mode`\n\n**Step 3: Approval workflow**\n\nAfter the model calls `exit_plan_mode`, 4 options appear:\n1. Choose `4` (keep-planning), enter feedback: \"Also add a step for updating README\"\n2. After the model revises the plan and calls exit_plan_mode again, choose `1` (clear-and-execute)\n\nPass criteria: After choosing 1, context is cleared and mode switches to execution\n\n**Step 4: Exit plan mode**\n```\n/plan\n```\nPass criteria: Switches back to normal mode\n\n**Design intent**: Plan Mode is Claude Code's \"think before you act\" mechanism. Restricting to read-only + plan file writing prevents the model from making code changes during the planning phase. The four-option approval gives users control over the execution method -- they can keep context and execute (2), or clear context and execute (1), avoiding the plan content itself from consuming the token budget.\n\n---\n\n## Phase 5: Editing and Search (Test 13, 17-18)\n\n### 13. Quote Normalization in Edit\n\n**Test objective**: Verify edit_file's curly quote -> straight quote fallback matching.\n\n```\nRead the file test/quote-test.js\n```\n\nThen request an edit using curly quotes:\n```\nUse edit_file on test/quote-test.js. In the old_string, use curly double quotes (Unicode U+201C and U+201D) around \"Hello World\". Replace with straight quotes saying \"Hi Universe\".\n```\n\nPass criteria:\n- Edit succeeds, output includes `(matched via quote normalization)`\n- File content changes from `\"Hello World\"` to `\"Hi Universe\"`\n\nAfter testing, restore:\n```\nEdit test/quote-test.js, replace \"Hi Universe\" with \"Hello World\"\n```\n\n**Design intent**: LLM output and text copied from documents often contain Unicode curly quotes (`\"\"`, `''`). Claude Code's `normalizeQuotes` function first attempts an exact match, then normalizes both sides to straight quotes on failure, avoiding the common \"can't find content to replace\" error.\n\n---\n\n### 17. Grep Search Tool\n\n**Test objective**: Verify regex search + include file filtering.\n\n```\nUse grep_search to find all lines containing \"import.*chalk\" in the src/ directory\n```\n\nPass criteria: Returns matching lines from `src/agent.ts` and/or `src/ui.ts` in the format `filepath:line_number:matched_content`\n\n```\nUse grep_search to find the pattern \"export function\" in all .ts files under src/\n```\n\nPass criteria: Uses `include: \"*.ts\"` filter, returns locations of all exported functions\n\n```\nUse grep_search to find \"DANGEROUS_PATTERNS\" in the project\n```\n\nPass criteria: Returns the definition location in `src/tools.ts`\n\n---\n\n### 18. Write File (New File + Auto Directory Creation)\n\n**Test objective**: Verify file creation, automatic directory creation, and content preview truncation.\n\n```\nCreate a new file at test/tmp/nested/hello.txt with the content:\nLine 1: Hello from Mini Claude\nLine 2: This is a write test\nLine 3: End of file\n```\n\nPass criteria:\n- Directory `test/tmp/nested/` is automatically created\n- Returns `Successfully wrote to test/tmp/nested/hello.txt (3 lines)` with line number preview\n\n```\nRead the file test/tmp/nested/hello.txt to verify.\n```\nPass criteria: Content is complete\n\nTest long file preview truncation:\n```\nCreate a file test/tmp/long-file.txt with 50 numbered lines like \"Line 1: test data\", etc.\n```\n\nPass criteria: Preview shows only the first 30 lines, with `... (50 lines total)` at the end\n\n---\n\n## Phase 6: Session and CLI (Test 14-16)\n\n### 14. Session Resume (--resume)\n\n**Test objective**: Verify session saving and cross-process restoration.\n\n**First session**:\n```bash\nnode dist/cli.js --yolo          # TS version\npython -m mini_claude --yolo     # Python version\n```\n```\nRemember this: The secret code is BANANA-42. Read package.json and tell me the version.\n```\nThen `exit` to quit.\n\n**Second session (resume)**:\n```bash\nnode dist/cli.js --yolo --resume          # TS version\npython -m mini_claude --yolo --resume     # Python version\n```\n\nPass criteria: On startup, shows session restored information\n\n```\nWhat was the secret code I told you earlier?\n```\n\nPass criteria: The model answers `BANANA-42`\n\n**Comparison (new session)**:\n```bash\nnode dist/cli.js --yolo          # TS version\npython -m mini_claude --yolo     # Python version\n```\n```\nWhat was the secret code I told you earlier?\n```\nPass criteria: The model cannot answer\n\n**Design intent**: Sessions are stored in JSON format in `~/.mini-claude/sessions/`, containing both Anthropic and OpenAI message histories (because the two backends have different message formats). `--resume` automatically finds the most recent session and continues the conversation from there.\n\n---\n\n### 15. One-Shot Mode\n\n**Test objective**: Verify that when a prompt argument is passed, it executes automatically and exits.\n\n```bash\n# TS version\nnode dist/cli.js --yolo \"Read the file package.json and tell me the project name. Only output the name.\"\n# Python version\npython -m mini_claude --yolo \"Read the file package.json and tell me the project name. Only output the name.\"\n```\n\nPass criteria:\n- The model calls read_file, outputs the project name\n- The program **exits automatically** (returns to shell prompt)\n\n```bash\nnode dist/cli.js --yolo \"List all TypeScript files in the src/ directory\"\n```\n\nPass criteria: Outputs the .ts file list, then exits automatically\n\nError scenario:\n```bash\nnode dist/cli.js --yolo \"Read the file /nonexistent/path/file.txt\"\n```\nPass criteria: The tool returns an error message, but the program doesn't crash and exits normally\n\n---\n\n### 16. Budget Control (--max-turns)\n\n**Test objective**: Verify agent loop iteration limits.\n\n```bash\n# TS version\nnode dist/cli.js --yolo --max-turns 2 \"Read these files one by one: package.json, tsconfig.json, src/cli.ts, src/agent.ts, src/tools.ts. Tell me the line count of each.\"\n# Python version\npython -m mini_claude --yolo --max-turns 2 \"Read these files one by one: package.json, tsconfig.json, src/cli.ts, src/agent.ts, src/tools.ts. Tell me the line count of each.\"\n```\n\nPass criteria:\n- The model starts reading files but stops after 2 agentic turns\n- Output includes a budget exceeded message\n- Does **not** finish reading all 5 files\n\n**Design intent**: Budget control has two dimensions -- `--max-cost` (USD limit) and `--max-turns` (loop iteration limit). Each agent loop (one API call + tool execution) counts as one turn. When the limit is exceeded, the model is told \"budget exceeded\" and stops. This prevents the Agent from entering infinite loops and burning money.\n\n---\n\n## Phase 7: Extension System (Test 19)\n\n### 19. Custom Agent (.claude/agents/)\n\n**Test objective**: Verify that user-defined agent types are correctly discovered and used.\n\n```\nWhat agent types are available? List them all.\n```\n\nPass criteria: The list includes explore, plan, general, and **reviewer**\n\n```\nUse the agent tool with type \"reviewer\" to review the file src/frontmatter.ts\n```\n\nPass criteria:\n- Output shows the `[sub-agent:reviewer]` marker\n- The reviewer only uses read_file / list_files / grep_search (restricted by allowed-tools)\n- Returns a code review result\n\n**Design intent**: Custom agents are defined via `.claude/agents/*.md` files, with frontmatter specifying name, description, and allowed tools. This lets users create specialized agents (code review, documentation generation, test writing, etc.) without modifying source code. Claude Code similarly supports both user-level (`~/.claude/agents/`) and project-level (`.claude/agents/`) two-layer override.\n\n---\n\n## Testing Complete\n\n```bash\nbash test/cleanup.sh\n```\n\nCleans up all test-generated files (MCP config, skills, rules, memory files, custom agents, temporary files, etc.).\n\n---\n\n## Quick Reference Table\n\n| # | Feature | Category | TS Pass | PY Pass | Notes |\n|---|---------|----------|:-------:|:-------:|-------|\n| 1 | MCP tool invocation | Basic tools | ☐ | ☐ | 3 tools |\n| 2 | WebFetch | Basic tools | ☐ | ☐ | httpbin.org |\n| 3 | Parallel tool execution | Basic tools | ☐ | ☐ | Multi-file simultaneous read |\n| 4 | Semantic memory recall | Memory & context | ☐ | ☐ | Save -> new conversation -> semantic query |\n| 5 | @include + Rules | Memory & context | ☐ | ☐ | Chinese response |\n| 6 | Read-before-edit | Memory & context | ☐ | ☐ | Code layer or prompt layer |\n| 7 | Large result persistence | Memory & context | ☐ | ☐ | 75KB file |\n| 8 | Skill invocation | Skills & extensions | ☐ | ☐ | /greet /commit |\n| 9 | ToolSearch | Skills & extensions | ☐ | ☐ | Plan mode tools |\n| 10 | REPL commands | Skills & extensions | ☐ | ☐ | /cost /memory /compact /plan |\n| 11 | Sub-agent system | Agent architecture | ☐ | ☐ | explore/plan/general |\n| 12 | Plan Mode | Agent architecture | ☐ | ☐ | /plan manual entry + approval |\n| 13 | Quote normalization | Editing & search | ☐ | ☐ | curly -> straight quotes |\n| 14 | Session Resume | Session & CLI | ☐ | ☐ | --resume restore session |\n| 15 | One-shot mode | Session & CLI | ☐ | ☐ | Pass prompt, auto exit |\n| 16 | Budget control | Session & CLI | ☐ | ☐ | --max-turns limit |\n| 17 | Grep Search | Editing & search | ☐ | ☐ | Regex search + include |\n| 18 | Write File | Editing & search | ☐ | ☐ | New file + auto directory creation |\n| 19 | Custom Agent | Extension system | ☐ | ☐ | .claude/agents/ definition |\n"}}}}