{"repo": "aaeekaayyyyyy/boston-tea-party", "files": {"eval/README.md": {"b87ba326fc1c6d75db9dad1902f432995a3c88e2": {"t": 1785846896.6344762, "n": 2181, "h": "7fa4ed8c309f823e", "c": "# Eval Harness\n\nEvaluation framework for Boston Tea Party 2.0.\n\n## Quick start\n\n```bash\npip install -r requirements.txt\n\n# Local metric smoke check (LettuceDetect + legacy HHEM, no API key needed)\npython -m eval.smoke_test --hhem-only\n\n# Full smoke test (needs both GEMINI_API_KEY for generation and OPENAI_API_KEY for grading)\nexport GEMINI_API_KEY=your-key\nexport OPENAI_API_KEY=your-openai-key\npython -m eval.smoke_test\n\n# Standalone rule benchmark (separate from answer-generation eval)\npython -m eval.constraint_benchmark\n\n# Run eval (uses LettuceDetect for hallucination)\npython -m eval.harness\n\n# Run the real planning-system eval (constraint engine + planning agent + hybrid retrieval)\npython -m eval.planning_system_eval\n\n# Legacy compatibility flag: skip faithfulness/API calls\npython -m eval.harness --hhem-only\n\n# Dry run (load scenarios only)\npython -m eval.harness --dry-run\n```\n\nThe harness now uses LettuceDetect as its hallucination metric. The retained\n`--hhem-only` flag is only a compatibility shortcut for running the local-only\nsubset without faithfulness/API calls.\n\n`python -m eval.constraint_benchmark` is a separate standalone rule-benchmark\npath for scoring the YAML rule benchmarks under `src/benchmarks/`. It does not\nrun the answer-generation harness.\n\n## Environment variables\n\nAdd to `.env` in repo root:\n\n```\nGEMINI_API_KEY=AIza...\nOPENAI_API_KEY=sk-...\n```\n\n## Switching providers\n\nEverything uses the OpenAI client format. Swap = change config.py:\n\n- Gemini (free): model=gemini-2.5-flash, url=generativelanguage.googleapis.com/v1beta/openai/\n- DeepSeek ($0.28/M): model=deepseek-chat, url=api.deepseek.com\n- Qwen ($0.26/M): model=qwen3.5-plus, url=dashscope-intl.aliyuncs.com/compatible-mode/v1\n- Ollama (free, local): model=qwen3:8b, url=localhost:11434/v1\n- Cluster vLLM (free): model=Qwen/Qwen2.5-72B-Instruct, url=cluster-node:8000/v1\n\n## Adding scenarios\n\nPut JSON files in benchmarks/. Minimum fields per scenario:\n\n```json\n{\n  \"id\": \"TC-001\",\n  \"question\": \"...\",\n  \"answer\": \"...\",\n  \"required_citations\": [\"...\"],\n  \"constraint_result\": { \"eligible\": true, \"conditions_met\": [\"...\"] }\n}\n```\n\nSee docs/benchmark_schema.md for full spec.\n"}}}}