{"repo": "bencan1a/9boxer", "files": {"agent-projects/ai-calibration-summary/DEVELOPER_GUIDE.md": {"7b3b8de931b81f7b4b6b28eb66681275ce2d6755": {"t": 1785596434.6999772, "n": 31170, "h": "e926e04c2ff612c3", "c": "# AI Calibration Summary - Developer Guide\n\n## Table of Contents\n\n1. [Adding New Analyses](#adding-new-analyses)\n2. [Modifying the System Prompt](#modifying-the-system-prompt)\n3. [Testing with Real LLM Calls](#testing-with-real-llm-calls)\n4. [Performance Profiling](#performance-profiling)\n5. [API Reference](#api-reference)\n6. [Frontend Integration](#frontend-integration)\n7. [Troubleshooting](#troubleshooting)\n\n---\n\n## Adding New Analyses\n\nThe analysis registry pattern makes adding new statistical analyses trivial. You can add custom analyses without modifying any core logic.\n\n### Step 1: Create Analysis Function\n\nCreate your analysis function in `backend/src/ninebox/services/intelligence_service.py` (or a new module):\n\n```python\ndef calculate_new_analysis(employees: list[Employee]) -> dict[str, Any]:\n \"\"\"Calculate custom analysis for calibration insights.\n\n Args:\n employees: List of employee records to analyze\n\n Returns:\n Dictionary with analysis results:\n {\n \"status\": \"green\" | \"yellow\" | \"red\",\n \"p_value\": 0.001,\n \"deviations\": [\n {\n \"category\": \"CategoryName\",\n \"z_score\": 3.5,\n \"is_significant\": True,\n \"observed_high_pct\": 45.0,\n \"expected_high_pct\": 20.0,\n \"sample_size\": 30\n }\n ],\n \"interpretation\": \"Human-readable explanation\",\n \"sample_size\": 100\n }\n \"\"\"\n # Your analysis logic here\n # Example: Analyze by department size\n large_depts = [emp for emp in employees if get_dept_size(emp.function) > 10]\n small_depts = [emp for emp in employees if get_dept_size(emp.function) <= 10]\n\n # Run chi-square test\n from scipy.stats import chi2_contingency\n\n contingency_table = build_contingency_table(large_depts, small_depts)\n chi2, p_value, _, _ = chi2_contingency(contingency_table)\n\n # Determine status\n status = \"red\" if p_value < 0.01 else \"yellow\" if p_value < 0.05 else \"green\"\n\n return {\n \"status\": status,\n \"p_value\": p_value,\n \"chi2\": chi2,\n \"interpretation\": f\"Department size shows {'significant' if p_value < 0.05 else 'no'} impact on ratings\",\n \"sample_size\": len(employees),\n \"deviations\": build_deviations(large_depts, small_depts),\n }\n```\n\n**Required Fields:**\n- `status` - \"green\", \"yellow\", or \"red\" (controls UI display)\n- `p_value` - Statistical significance (float between 0 and 1)\n- `sample_size` - Number of employees analyzed\n- `interpretation` - Human-readable summary\n\n**Optional Fields:**\n- `deviations` - List of significant category-level deviations\n- `chi2` - Chi-square statistic (or other test statistic)\n- Any other fields your analysis produces\n\n### Step 2: Register Analysis\n\nAdd your analysis to the registry in `backend/src/ninebox/services/analysis_registry.py`:\n\n```python\nfrom ninebox.services.intelligence_service import (\n calculate_function_analysis,\n calculate_level_analysis,\n calculate_location_analysis,\n calculate_tenure_analysis,\n calculate_new_analysis, # Your new analysis\n)\n\nANALYSIS_REGISTRY: list[tuple[str, AnalysisFunction]] = [\n (\"location\", calculate_location_analysis),\n (\"function\", calculate_function_analysis),\n (\"level\", calculate_level_analysis),\n (\"tenure\", calculate_tenure_analysis),\n (\"new_analysis\", calculate_new_analysis), # Add here\n]\n```\n\nThat's it! Your analysis will now:\n- Run automatically on every calibration summary request\n- Be included in the LLM agent's data package\n- Potentially generate insights in the AI summary\n- Appear in Intelligence tab visualizations\n\n### Step 3: Update System Prompt (Optional)\n\nIf your analysis introduces new categories or types, update the system prompt to help the agent interpret it:\n\n```\n## Issue Categories\n\n- **location**: Geographic location-based patterns\n- **function**: Department/function-based patterns\n- **level**: Job level-based patterns\n- **tenure**: Tenure-based patterns\n- **department_size**: Department size impact patterns <-- Add your category\n- **distribution**: Overall rating distribution patterns\n- **time**: Meeting time allocation\n```\n\n### Step 4: Test\n\nRun the integration test to verify your analysis works:\n\n```bash\ncd backend\npython test_llm_agent_architecture.py\n```\n\nExpected output:\n```\n[OK] Analyses included: ['location', 'function', 'level', 'tenure', 'new_analysis']\n```\n\n---\n\n## Modifying the System Prompt\n\nThe system prompt defines the AI agent's expertise, analysis approach, and output format. Customizing it allows you to tailor insights for your organization.\n\n### Location\n\nFile: `backend/config/calibration_agent_prompt.txt`\n\n**IMPORTANT:** This is a TEXT file, not Python code. Do not add code comments or quotes.\n\n### Structure\n\nThe prompt has 10 main sections:\n\n1. **Persona & Task** - Defines the agent as an HR consultant\n2. **Analysis Focus** - Guides what to look for (root causes, drivers, actionable guidance)\n3. **Clustering Guidelines** - When to group related issues\n4. **Priority Guidelines** - How to assign high/medium/low priority\n5. **Output Format** - Exact JSON schema\n6. **Issue Types** - Taxonomy of issue types\n7. **Issue Categories** - Taxonomy of categories\n8. **Example Output 1** - Clustered issues with clear root cause\n9. **Example Output 2** - Multiple independent high-priority issues\n10. **Constraints & Rules** - What NOT to do\n\n### Common Modifications\n\n#### Change Priority Thresholds\n\n**Original:**\n```\n**High Priority** (requires immediate attention):\n- Significant statistical anomalies with |z-score| > 3.0\n- Critical fairness concerns (major disparities across protected groups)\n- Major distribution problems (>60% in center box, <3% stars, >30% stars)\n```\n\n**Custom (more lenient):**\n```\n**High Priority** (requires immediate attention):\n- Significant statistical anomalies with |z-score| > 2.5 # Lowered from 3.0\n- Critical fairness concerns (major disparities across protected groups)\n- Major distribution problems (>55% in center box, <5% stars, >25% stars) # Adjusted thresholds\n```\n\n#### Adjust Tone\n\n**Original:**\n```\nBe direct and practical. Use business language appropriate for HR leadership.\n```\n\n**Custom (more casual):**\n```\nBe conversational and approachable. Use plain language that managers at all levels can understand.\nAvoid HR jargon unless necessary. Prioritize clarity over formality.\n```\n\n#### Add Industry-Specific Context\n\n**Original:**\n```\nYour role is to analyze calibration data holistically and provide actionable insights\nthat help HR leaders run effective, fair, and efficient calibration meetings.\n```\n\n**Custom (tech industry):**\n```\nYour role is to analyze calibration data holistically and provide actionable insights\nthat help HR leaders run effective, fair, and efficient calibration meetings.\n\nCONTEXT: This organization is a fast-growing tech company where:\n- Engineering talent is critical and often commands premium ratings\n- Remote work may create location-based bias patterns\n- New hire concentration is common due to rapid scaling\n- Retention of Stars (position 9) is a top priority\n```\n\n#### Emphasize Specific Concerns\n\nAdd a new section after \"Analysis Focus\":\n\n```\n## Key Organizational Priorities\n\nWhen generating insights, pay special attention to:\n\n1. **Retention Risk** - Identify Stars and High Performers who may be flight risks\n2. **Succession Planning** - Flag gaps in leadership pipeline (Director+ levels)\n3. **Diversity & Inclusion** - Detect any location/function patterns that could indicate bias\n4. **New Hire Integration** - Track if tenure affects ratings (potential probationary bias)\n```\n\n### Testing Prompt Changes\n\nAfter modifying the prompt, test with the integration script:\n\n```bash\ncd backend\npython test_llm_agent_architecture.py\n```\n\nThis will:\n1. Load the system prompt from `config/calibration_agent_prompt.txt`\n2. Package sample employee data\n3. Call Claude API with your modified prompt\n4. Display the generated summary and issues\n\n**Review the output to verify:**\n- Agent follows your new guidelines\n- Priority assignments match your thresholds\n- Tone is appropriate for your audience\n- Industry-specific context influences recommendations\n\n### Prompt Engineering Best Practices\n\n**DO:**\n- Be explicit about what you want (examples are powerful)\n- Provide clear criteria for priority/classification\n- Include 1-2 detailed examples of desired output\n- Specify what NOT to do (constraints)\n- Use consistent terminology throughout\n\n**DON'T:**\n- Make prompts too long (>4000 words gets diluted)\n- Contradict yourself (e.g., \"be brief\" then \"provide detailed analysis\")\n- Over-specify (let the model apply judgment)\n- Include PII in examples (use anonymized data)\n\n---\n\n## Testing with Real LLM Calls\n\n### Setup\n\n1. **Get Anthropic API Key:**\n - Sign up at https://console.anthropic.com\n - Create an API key\n - Copy the key (starts with `sk-ant-api03-`)\n\n2. **Set Environment Variable:**\n\n **Linux/Mac:**\n ```bash\n export ANTHROPIC_API_KEY='sk-ant-api03-xxxxxxxxxxxxx'\n ```\n\n **Windows (PowerShell):**\n ```powershell\n $env:ANTHROPIC_API_KEY='sk-ant-api03-xxxxxxxxxxxxx'\n ```\n\n **Windows (Command Prompt):**\n ```cmd\n set ANTHROPIC_API_KEY=sk-ant-api03-xxxxxxxxxxxxx\n ```\n\n **Persistent (add to .env file):**\n ```bash\n # backend/.env\n ANTHROPIC_API_KEY=sk-ant-api03-xxxxxxxxxxxxx\n LLM_MODEL=claude-sonnet-4-5-20250929 # Optional: override model\n ```\n\n3. **Install Anthropic SDK:**\n ```bash\n cd backend\n pip install anthropic\n ```\n\n### Integration Test Script\n\n**Location:** `backend/test_llm_agent_architecture.py`\n\n**Run:**\n```bash\ncd backend\npython test_llm_agent_architecture.py\n```\n\n**Expected Output:**\n```\n================================================================================\nLLM Service Agent-First Architecture Test Suite\n================================================================================\n\n================================================================================\nTEST 1: Load System Prompt\n================================================================================\n[OK] System prompt loaded successfully\n[OK] Prompt length: 12500 characters\n[OK] First 200 chars: You are an expert HR consultant and calibration meeting facilitator...\n\n================================================================================\nTEST 2: Data Packaging\n================================================================================\n[OK] Packaged 3 employees\n[OK] Overview: 3 total employees\n[OK] Analyses included: ['location', 'function', 'level', 'tenure', 'per_level_distribution']\n[OK] Employee IDs are anonymized (using 'Employee_1')\n[OK] Business titles are anonymized\n[OK] Package size: 2847 bytes\n\n================================================================================\nTEST 3: Generate Calibration Analysis\n================================================================================\n[OK] LLM service is available\n[OK] Using model: claude-sonnet-4-5-20250929\n[OK] Calling generate_calibration_analysis()...\n[OK] Analysis generated successfully!\n[OK] Summary length: 1523 characters\n[OK] Number of issues: 5\n\nSUMMARY:\n--------------------------------------------------------------------------------\nYour calibration session has a very small population (3 employees), which limits\nstatistical reliability but still reveals some patterns worth noting. The most\nstriking finding is your distribution: 67% Stars (position 9) and 33% High\nPerformers (position 6), with zero employees...\n\nISSUES:\n--------------------------------------------------------------------------------\n1. [HIGH] Grade inflation: 67% rated as Stars\n Type: focus_area, Category: distribution\n Affected: 2 employees\n\n2. [MEDIUM] All Engineering function are high performers\n Type: anomaly, Category: function\n Affected: 2 employees\n\n3. [LOW] Allocate 15 minutes for calibration meeting\n Type: time_allocation, Category: time\n Affected: 3 employees\n\n================================================================================\nTEST SUMMARY\n================================================================================\nPassed: 4/4\n[OK] All tests passed!\n```\n\n### Unit Tests\n\n**Location:** `backend/tests/unit/services/test_llm_service.py`\n\n**Run:**\n```bash\ncd backend\npytest tests/unit/services/test_llm_service.py -v\n```\n\n**Tests:**\n- `test_is_available_when_api_key_set` - Availability check with key\n- `test_is_available_when_no_api_key` - Graceful handling when key missing\n- `test_generate_calibration_analysis_calls_api` - End-to-end API call\n- `test_parse_response_handles_markdown` - JSON extraction from markdown\n- `test_anonymization_removes_pii` - Data privacy verification\n\n### Cost Considerations\n\n**Claude Sonnet 4.5 Pricing (as of Jan 2025):**\n- Input: $3 per million tokens\n- Output: $15 per million tokens\n\n**Typical Request:**\n- Input: ~2,000 tokens (employee data + analyses)\n- Output: ~1,500 tokens (summary + insights)\n- **Cost per request: ~$0.03 (3 cents)**\n\n**Monthly Cost Estimates:**\n- 10 calibration sessions/month: ~$0.30\n- 100 calibration sessions/month: ~$3.00\n- 1,000 calibration sessions/month: ~$30.00\n\n**Optimization Tips:**\n- Cache responses for identical employee datasets\n- Limit to production environment only (not dev/test)\n- Monitor usage via Anthropic console\n\n---\n\n## Performance Profiling\n\nUnderstanding the performance characteristics of the calibration summary feature is essential for setting user expectations and identifying optimization opportunities.\n\n### Baseline Performance Metrics\n\nThe calibration summary feature has been profiled across typical dataset sizes to establish baseline metrics. The table below shows P50 (median) and P95 (95th percentile) timings for each component.\n\n**Context:**\n- Internal desktop tool with max ~2000 employees\n- Users typically analyze director-level subsets (100-500 employees)\n- Performance measured on standard development hardware\n\n**Profiling Results:**\n\n| Employee Count | Analysis (P50) | Packaging (P50) | LLM Call (P50) | Total (P50) | Total (P95) |\n|----------------|----------------|-----------------|----------------|-------------|-------------|\n| 50 | 0.011s | 0.001s | 4.000s | 4.011s | 10.437s |\n| 100 | 0.010s | 0.001s | 8.000s | 8.011s | 8.012s |\n| 250 | 0.011s | 0.003s | 20.000s | 20.014s | 20.020s |\n| 500 | 0.019s | 0.005s | 40.000s | 40.024s | 40.028s |\n\n**Key Insights:**\n\n1. **LLM Dominates Total Time**: The LLM API call accounts for >99% of total execution time\n - For 250 employees: ~20s of the total 20s is LLM processing\n - Analysis registry + data packaging: <1% of total time\n\n2. **Analysis Registry Performance**: Excellent scalability\n - 50 employees: 0.011s\n - 500 employees: 0.019s\n - Near-linear scaling with dataset size\n\n3. **Data Packaging Performance**: Minimal overhead\n - 50 employees: 0.001s\n - 500 employees: 0.005s\n - Optimized data structure keeps packaging fast\n\n4. **LLM Call Latency**: Scales with token count\n - Estimate: ~0.08s per employee for mock timing\n - Real API calls typically 20-30s for 250 employees\n - Network latency + Claude processing time\n\n**User Expectations:**\n\nFor typical use cases:\n- **50 employees**: ~4-6 seconds total (small team calibration)\n- **100 employees**: ~8-10 seconds total (department calibration)\n- **250 employees**: ~20-25 seconds total (director-level calibration)\n- **500 employees**: ~40-45 seconds total (VP-level calibration)\n\n**Setting User Expectations:**\nDisplay a progress indicator with messaging:\n- \"Analyzing calibration data...\" (< 1 second)\n- \"Generating AI insights...\" (20-30 seconds for 250 employees)\n- The LLM call dominates total time, so focus UX messaging on AI processing\n\n### Running Performance Profiling\n\nTo benchmark performance on your environment, use the profiling script:\n\n**Location:** `backend/tests/performance/profile_calibration_summary.py`\n\n**Basic Usage:**\n```bash\ncd backend/tests/performance\npython profile_calibration_summary.py\n```\n\n**Output:**\n```\n================================================================================\nCalibration Summary Performance Profiling\n================================================================================\nDataset sizes: [50, 100, 250, 500]\nRuns per size: 3\nLLM mode: Mock (estimated)\n\n| Employee Count | Analysis (P50) | Packaging (P50) | LLM Call (P50) | Total (P50) |\n|----------------|----------------|-----------------|----------------|-------------|\n| 50 | 0.011s | 0.001s | 4.000s | 4.011s |\n| 100 | 0.010s | 0.001s | 8.000s | 8.011s |\n| 250 | 0.011s | 0.003s | 20.000s | 20.014s |\n| 500 | 0.019s | 0.005s | 40.000s | 40.024s |\n\nDetailed results saved to: profiling_results.csv\n```\n\n**Advanced Options:**\n\n```bash\n# Use real LLM API calls (requires ANTHROPIC_API_KEY)\npython profile_calibration_summary.py --with-llm\n\n# Custom dataset sizes\npython profile_calibration_summary.py --sizes 100,200,300\n\n# More runs for statistical reliability\npython profile_calibration_summary.py --runs 5\n\n# Custom output filename\npython profile_calibration_summary.py --output my_results.csv\n```\n\n**Profiling with Real LLM:**\n\nTo get actual LLM latency (not mock estimates):\n\n1. Set your Anthropic API key:\n ```bash\n export ANTHROPIC_API_KEY='sk-ant-api03-xxxxxxxxxxxxx'\n ```\n\n2. Run with `--with-llm` flag:\n ```bash\n python profile_calibration_summary.py --with-llm --sizes 50,100,250\n ```\n\n**WARNING**: Real LLM calls will incur API costs (~$0.03 per request).\n\n**CSV Output:**\n\nThe script generates a detailed CSV report with P50, P95, mean, min, and max values for each metric:\n\n```csv\nEmployee Count,Metric,P50 (seconds),P95 (seconds),Mean (seconds),Min (seconds),Max (seconds)\n50,analysis_registry,0.011,6.436,2.150,0.004,6.436\n50,data_packaging,0.001,0.001,0.000,0.000,0.001\n50,llm_call,4.000,4.000,4.000,4.000,4.000\n50,total_end_to_end,4.011,10.437,6.151,4.004,10.437\n...\n```\n\nUse this data to:\n- Track performance regressions over time\n- Identify optimization opportunities\n- Set SLA targets for production deployments\n- Inform infrastructure sizing decisions\n\n### Optimization Opportunities\n\nBased on profiling results, optimization efforts should focus on:\n\n1. **LLM Call Optimization** (highest impact):\n - Cache responses for identical employee datasets\n - Implement request batching for multiple simultaneous users\n - Consider streaming responses for perceived performance\n - Use prompt compression techniques to reduce token count\n\n2. **Analysis Registry** (low impact):\n - Already highly optimized (< 20ms for 500 employees)\n - Further optimization yields minimal total time improvement\n\n3. **Data Packaging** (low impact):\n - Already minimal overhead (< 10ms for 500 employees)\n - Current implementation is sufficient\n\n**Recommendation**: Focus on LLM call optimization and UX improvements (progress indicators, caching) for maximum user experience impact.\n\n---\n\n## API Reference\n\n### Calibration Summary Service\n\n#### `calculate_summary(employees, use_agent=True)`\n\nGenerate calibration summary with optional AI analysis.\n\n**Parameters:**\n- `employees` (list[Employee]) - Employee records to analyze\n- `use_agent` (bool, default=True) - Use AI agent for insights + summary\n\n**Returns:**\n```python\nCalibrationSummaryResponse(\n data_overview=DataOverview(...),\n time_allocation=TimeAllocation(...),\n insights=[Insight(...)],\n summary=\"AI-generated summary...\", # Only when use_agent=True\n)\n```\n\n**Example:**\n```python\nfrom ninebox.services.calibration_summary_service import CalibrationSummaryService\n\nservice = CalibrationSummaryService()\nsummary = service.calculate_summary(employees, use_agent=True)\n\nprint(f\"Total employees: {summary['data_overview']['total_employees']}\")\nprint(f\"Stars: {summary['data_overview']['stars_percentage']}%\")\nprint(f\"AI Summary: {summary.get('summary', 'Not available')}\")\n```\n\n### LLM Service\n\n#### `generate_calibration_analysis(package)`\n\nGenerate AI-powered calibration analysis from packaged data.\n\n**Parameters:**\n- `package` (dict) - Output from `package_for_llm(employees, analyses)`\n\n**Returns:**\n```python\n{\n \"summary\": str, # Executive summary (2-3 paragraphs)\n \"issues\": [\n {\n \"type\": str, # anomaly | focus_area | recommendation | time_allocation\n \"category\": str, # location | function | level | tenure | distribution | time\n \"priority\": str, # high | medium | low\n \"title\": str,\n \"description\": str,\n \"affected_count\": int,\n \"source_data\": dict,\n \"cluster_id\": str | None,\n \"cluster_title\": str | None,\n }\n ],\n}\n```\n\n**Example:**\n```python\nfrom ninebox.services.llm_service import LLMService\nfrom ninebox.services.analysis_registry import run_all_analyses\nfrom ninebox.services.data_packaging_service import package_for_llm\n\n# Run analyses\nanalyses = run_all_analyses(employees)\n\n# Package for LLM (anonymized)\npackage = package_for_llm(employees, analyses)\n\n# Generate analysis\nllm = LLMService()\nresult = llm.generate_calibration_analysis(package)\n\nprint(result[\"summary\"])\nfor issue in result[\"issues\"]:\n print(f\"[{issue['priority'].upper()}] {issue['title']}\")\n```\n\n### Analysis Registry\n\n#### `run_all_analyses(employees)`\n\nRun all registered analyses and return results.\n\n**Parameters:**\n- `employees` (list[Employee]) - Employee records to analyze\n\n**Returns:**\n```python\n{\n \"location\": {\"status\": \"green\" | \"yellow\" | \"red\", \"p_value\": float, \"deviations\": [...]},\n \"function\": {...},\n \"level\": {...},\n \"tenure\": {...},\n \"per_level_distribution\": {...},\n}\n```\n\n**Example:**\n```python\nfrom ninebox.services.analysis_registry import run_all_analyses\n\nanalyses = run_all_analyses(employees)\n\nif analyses[\"location\"][\"status\"] == \"red\":\n print(f\"Location anomaly detected! p-value: {analyses['location']['p_value']}\")\n```\n\n### Data Packaging Service\n\n#### `package_for_llm(employees, analyses, org_data=None)`\n\nPackage calibration data for LLM analysis (strict anonymization).\n\n**Parameters:**\n- `employees` (list[Employee]) - Employee records\n- `analyses` (dict) - Results from `run_all_analyses()`\n- `org_data` (dict | None) - Optional pre-built org data\n\n**Returns:**\n```python\n{\n \"employees\": [\n {\n \"id\": \"Employee_1\", # Anonymized\n \"level\": \"MT5\",\n \"function\": \"Engineering\",\n \"grid_position\": 9,\n ...\n }\n ],\n \"organization\": {...},\n \"analyses\": {...},\n \"overview\": {...}\n}\n```\n\n**Example:**\n```python\nfrom ninebox.services.data_packaging_service import package_for_llm\n\npackage = package_for_llm(employees, analyses)\n\n# Verify anonymization\nassert \"employee_id\" not in package[\"employees\"][0]\nassert package[\"employees\"][0][\"id\"] == \"Employee_1\"\n```\n\n#### `package_for_ui(employees, analyses, org_data=None)`\n\nPackage calibration data for UI display (includes PII).\n\n**Same signature as `package_for_llm()`, but includes:**\n- `employee_id` (real ID)\n- `business_title`\n- `job_title`\n- `manager_id` (real manager ID)\n\n**Example:**\n```python\nfrom ninebox.services.data_packaging_service import package_for_ui\n\nui_package = package_for_ui(employees, analyses)\n\n# Has real employee IDs for frontend display\nassert \"employee_id\" in ui_package[\"employees\"][0]\n```\n\n---\n\n## Frontend Integration\n\n### Using the Hook\n\n**Location:** `frontend/src/hooks/useCalibrationSummary.ts`\n\n**Basic Usage:**\n```typescript\nimport { useCalibrationSummary } from '../hooks/useCalibrationSummary';\n\nfunction CalibrationPage() {\n const {\n data,\n isLoading,\n error,\n summary,\n refetch\n } = useCalibrationSummary({ useAgent: true });\n\n if (isLoading) return
Loading...
;\n if (error) return
Error: {error.message}
;\n if (!data) return null;\n\n return (\n
\n
Calibration Summary
\n\n {/* Data Overview */}\n \n
Overview
\n
Total Employees: {data.data_overview.total_employees}
\n
Stars: {data.data_overview.stars_percentage}%
\n
Center Box: {data.data_overview.center_box_percentage}%
\n );\n}\n```\n\n### Real-time Updates\n\nThe hook automatically refetches when employee data changes:\n\n```typescript\nimport { useSessionStore } from '../store/sessionStore';\n\nfunction CalibrationPage() {\n const { employees } = useSessionStore();\n const { data } = useCalibrationSummary(); // Auto-refetches when employees change\n\n useEffect(() => {\n console.log(`Summary updated for ${employees.length} employees`);\n }, [data, employees]);\n\n // ...\n}\n```\n\n### Direct API Calls\n\nIf you need more control, use the API client directly:\n\n```typescript\nimport { apiClient } from '../services/api';\n\nasync function fetchSummary() {\n try {\n // Agent-first approach (default)\n const summary = await apiClient.getCalibrationSummary({ useAgent: true });\n console.log(summary.summary); // AI-generated summary\n\n // Legacy approach (no AI summary)\n const legacySummary = await apiClient.getCalibrationSummary({ useAgent: false });\n console.log(legacySummary.summary); // null\n\n // Check LLM availability\n const availability = await apiClient.checkLLMAvailability();\n if (!availability.available) {\n console.warn(`LLM unavailable: ${availability.reason}`);\n }\n } catch (error) {\n console.error('Failed to fetch calibration summary:', error);\n }\n}\n```\n\n---\n\n## Troubleshooting\n\n### \"Analysis failed\" in Results\n\n**Symptom:** One or more analyses show `\"status\": \"error\"` in response\n\n**Diagnosis:**\n```python\n# Check backend logs for traceback\n# Look for: \"Analysis 'XXX' failed: ...\"\n```\n\n**Common Causes:**\n1. **Empty employee list** - Need at least 2 employees for chi-square tests\n2. **Missing attributes** - Employee objects missing required fields (location, function, etc.)\n3. **Division by zero** - All employees in same category (no variance)\n\n**Solution:**\n```python\n# Add defensive checks in your analysis function\ndef calculate_new_analysis(employees: list[Employee]) -> dict[str, Any]:\n if len(employees) < 2:\n return {\n \"status\": \"green\",\n \"p_value\": 1.0,\n \"sample_size\": len(employees),\n \"interpretation\": \"Not enough employees for analysis\",\n }\n\n # Ensure required fields exist\n if not all(hasattr(emp, \"custom_field\") for emp in employees):\n return {\"status\": \"error\", \"error\": \"Missing custom_field attribute\", \"sample_size\": 0}\n\n # ... rest of analysis\n```\n\n### Agent Returns Generic Insights\n\n**Symptom:** AI summary is vague and doesn't mention specific categories/levels\n\n**Diagnosis:** Check if your data package includes sufficient detail:\n\n```python\n# Verify package contents\npackage = package_for_llm(employees, analyses)\n\n# Should have specific categories with counts\nassert len(package[\"overview\"][\"by_level\"]) > 1\nassert len(package[\"analyses\"][\"location\"][\"deviations\"]) > 0\n```\n\n**Solution:**\n1. Ensure analyses are finding significant deviations (p_value < 0.05)\n2. Check if employee distribution allows for meaningful comparisons\n3. Review system prompt - may need to emphasize specific insights\n\n### Rate Limiting / API Errors\n\n**Symptom:** 500 error: \"Failed to call Claude API: rate_limit_error\"\n\n**Diagnosis:**\n- Check Anthropic console for rate limit tier\n- Review usage patterns (too many concurrent requests)\n\n**Solution:**\n```python\n# Add retry logic with exponential backoff\nfrom tenacity import retry, stop_after_attempt, wait_exponential\n\n\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))\ndef call_claude_with_retry(self, prompt: str):\n return self._client.messages.create(...)\n```\n\n### Inconsistent Clustering\n\n**Symptom:** Same issues sometimes clustered, sometimes not\n\n**Diagnosis:** Agent is applying judgment about whether issues are related\n\n**Solution:**\n1. Make clustering criteria more explicit in system prompt\n2. Provide more examples of when to cluster\n3. Accept that clustering is judgment-based (some variance is OK)\n4. Consider post-processing to enforce clustering rules\n\n### Frontend Not Showing Summary\n\n**Symptom:** `data.summary` is null even though `useAgent: true`\n\n**Diagnosis:**\n```typescript\n// Check LLM availability first\nconst availability = await apiClient.checkLLMAvailability();\nconsole.log(availability); // { available: false, reason: \"...\" }\n```\n\n**Common Causes:**\n1. ANTHROPIC_API_KEY not set in backend\n2. LLM API call failed (check backend logs)\n3. Response parsing error (invalid JSON from Claude)\n\n**Solution:**\n- Set API key in backend environment\n- Check backend logs for detailed error messages\n- Verify network connectivity to Anthropic API\n\n---\n\n## Additional Resources\n\n- **Anthropic API Docs:** https://docs.anthropic.com/claude/reference\n- **Claude Model Specs:** https://docs.anthropic.com/claude/docs/models-overview\n- **Prompt Engineering Guide:** https://docs.anthropic.com/claude/docs/prompt-engineering\n- **9Boxer Intelligence Service:** `backend/src/ninebox/services/intelligence_service.py`\n\n## Support\n\nFor questions or issues:\n1. Check this guide first\n2. Review backend logs for detailed error messages\n3. Test with `test_llm_agent_architecture.py` to isolate issues\n4. Contact the 9Boxer development team\n"}}}}