1 line
26 KiB
JSON
1 line
26 KiB
JSON
{"repo": "alya71/medask-updated", "files": {"triage_bench/README.md": {"5b5e98a1a07048d4d24e583c6aad555ce7cc79ae": {"t": 1785746464.017426, "n": 8515, "h": "9bf3beffbf5e358e", "c": "# Triage Benchmark\n\nA comprehensive benchmark for evaluating Large Language Model (LLM) performance on medical triage classification tasks.\n\n## Overview\n\nThis benchmark evaluates how accurately LLMs can classify medical vignettes into appropriate triage categories:\n\n- **Emergency (em)**: Call 911 or go directly to the emergency room\n- **Non-Emergency (ne)**: See a doctor, but symptoms don't require immediate ER attention (within a week)\n- **Self-care (sc)**: Let the health issue resolve on its own and reassess in a few days\n\n## Features\n\n- **Multiple LLM Support**: OpenAI models (GPT-4o, GPT-4.5, O1, O3, etc.), DeepSeek models, and open-source models via Ollama (Llama, Qwen, DeepSeek R1, Mistral, etc.)\n- **Vignettes**: Semigran clinical vignettes\n- **Statistical Analysis**: Paired model comparison with McNemar tests\n- **Safety Metrics**: Tracks over-triage and under-triage tendencies\n- **Reproducible Results**: Supports multiple runs for statistical significance\n\n## Installation\n\nFrom the project root directory:\n\n```bash\n# Install dependencies\npip install -r requirements/development.txt\npip install -e .\n\n# Set API keys (for cloud models)\nexport KEY_OPENAI=\"sk-...\" # For OpenAI models\nexport KEY_DEEPSEEK=\"...\" # For DeepSeek models\n\n# Install Ollama (for open-source models)\n# Platform-specific installation:\n# Mac: Download from https://ollama.com/download or: brew install ollama\n# Linux: curl -fsSL https://ollama.com/install.sh | sh\n# Windows: Download installer from https://ollama.com/download\n# After installation, verify: ollama --version\n\n# Pull open-source models (examples)\nollama pull llama3.1\nollama pull qwen2.5\nollama pull deepseek-r1\nollama pull mistral\n```\n<｜tool▁call▁begin｜>\nread_file\n\n\n\n## Usage\n\n### Web UI (Recommended)\n\nLaunch the interactive Gradio web interface:\n\n```bash\ncd triage_bench\npython3 app.py\n```\n\nThe web UI will open in your browser at `http://localhost:7860`.\n\n#### Web UI Features:\n- **Live Progress**: Real-time progress updates showing vignettes completed / total\n- **Model Selection**: Easy dropdown selection for OpenAI, DeepSeek, and Ollama models\n- **Custom Models**: Support for custom model names or URLs\n- **Configuration**: Adjust runs (1-5), vignette count, and vignette sets\n- **Results Display**: Formatted summary metrics with accuracy breakdowns\n- **Downloadable Results**: JSON results available for download\n- **API Key Management**: Enter API keys directly in the UI (or use environment variables)\n\n#### Web UI Usage:\n1. Select model provider (OpenAI, DeepSeek, Ollama, or Custom)\n2. Choose specific model from dropdown (or enter custom model name)\n3. Configure benchmark settings (runs, vignette set, vignette count)\n4. Enter API keys if using cloud models (optional if set as environment variables)\n5. Click \"Run Benchmark\" to start evaluation\n6. Watch live progress updates as vignettes are processed\n7. View final summary metrics and download results as JSON\n\n### Command Line Interface\n\n```bash\ncd triage_bench\n\n# Run with default settings (DeepSeek chat, Semigran vignettes, 1 run)\npython3 main.py\n\n# Run with GPT-4o\npython3 main.py --model gpt-4o\n\n# Run with multiple passes for statistical robustness\npython3 main.py --model gpt-4o --runs 5\n\npython3 main.py --model gpt-4o --vignette_set semigran\n```\n\n### Available Models\n\n**OpenAI Models:**\n- `gpt-4o`\n- `gpt-4.5-preview` \n- `o1`, `o1-mini`\n- `o3`, `o3-mini`\n- `o4-mini`\n\n**DeepSeek Models:**\n- `deepseek-chat` (default)\n- `deepseek-reasoner`\n\n**Open-Source Models (via Ollama):**\n- **Llama**: `llama3.1`, `llama3`, `llama2`\n- **Qwen**: `qwen2.5`, `qwen2`, `qwen2.5:72b`\n- **DeepSeek R1**: `deepseek-r1`, `deepseek-r1-distill`\n- **Mistral**: `mistral`, `mixtral`, `mistral-small`\n\n> **Note**: Any model available in Ollama can be used. The benchmark auto-detects common model names (llama*, qwen*, deepseek-r1*, mistral*). You can also use the explicit format: `ollama+<model-name>`.\n\n**Prerequisites for Ollama models:**\n1. Install Ollama: `curl -fsSL https://ollama.com/install.sh | sh`\n2. Pull the model: `ollama pull <model-name>`\n3. Ensure Ollama is running: `ollama serve` (usually runs automatically)\n\n### Available Vignette Sets\n\n- `semigran`: 45 clinical vignettes from Semigran et al.\n\n### Example Commands\n\n```bash\n# Compare GPT-4o vs GPT-4.5 with 3 runs each\npython3 main.py --model gpt-4o --runs 3\npython3 main.py --model gpt-4.5-preview --runs 3\n\n# Test DeepSeek reasoning model\npython3 main.py --model deepseek-reasoner --vignette_set semigran --runs 2\n\n# Test open-source models via Ollama\npython3 main.py --model llama3.1 --runs 1\npython3 main.py --model qwen2.5 --runs 1\npython3 main.py --model deepseek-r1 --runs 1\npython3 main.py --model mistral --runs 1\n\n# Explicit Ollama format (optional)\npython3 main.py --model ollama+llama3.1 --runs 1\n```\n\n## Statistical Analysis\n\n### Paired Model Comparison\n\nCompare two models statistically using McNemar's test:\n\n```bash\npython3 paired_analysis.py results/model1_results.jsonl results/model2_results.jsonl\n```\n\n**Example Output:**\n```\n=== MODEL A ===\nOverall accuracy: 80.00% (36/45)\nAccuracy by triage level:\n em: 93.33% (14/15)\n ne: 60.00% (9/15)\n sc: 86.67% (13/15)\nSafety (at‑or‑above correct urgency): 88.89% (40/45)\nOver‑triage inclination (among incorrect): 44.44% (4/9)\n\n=== MODEL B ===\nOverall accuracy: 77.78% (35/45)\nSafety (at‑or‑above correct urgency): 95.56% (43/45)\n\n=== McNemar paired test ===\ndiscordant pairs: A‑right/B‑wrong = 5, A‑wrong/B‑right = 4\np‑value: 1.0 (exact=True)\nAccuracy difference (B − A): -2.22%\n```\n\n## Output Format\n\n### Results Files\n\nResults are saved as timestamped JSONL files in the `results/` directory:\n\n```\nresults/20250703T134639_gpt-4o_semigran_triage.jsonl\n```\n\nEach line contains:\n```json\n{\n \"run_id\": 1,\n \"case_id\": 1,\n \"true_urgency\": \"em\",\n \"llm_output\": \"em\",\n \"correct\": true,\n \"model\": \"gpt-4o\"\n}\n```\n\n### Evaluation Metrics\n\n- **Overall Accuracy**: Percentage of correct triage classifications\n- **Per-Level Accuracy**: Accuracy for each triage category (em/ne/sc)\n- **Safety Rate**: Percentage of predictions at or above correct urgency level\n- **Over-triage Rate**: Among incorrect predictions, percentage that over-triaged\n\n## Vignette Format\n\nVignettes are stored as JSONL files with the following structure:\n\n```json\n{\n \"urgency_level\": \"em\",\n \"correct_diagnosis\": \"Acute liver failure\",\n \"case_description\": \"A 48-year-old woman with altered mental status...\"\n}\n```\n\n## Safety Considerations\n\nThe benchmark uses a conservative safety framework:\n- **Under-triage** (predicting lower urgency than correct) is considered more dangerous\n- **Over-triage** (predicting higher urgency than correct) is safer but may strain resources\n- Safety metrics help evaluate model reliability for real-world deployment\n\n## Research Applications\n\nThis benchmark is useful for:\n- Comparing LLM performance on medical triage tasks\n- Evaluating safety vs. efficiency trade-offs\n- Testing prompt engineering approaches for medical AI\n- Validating model reliability before clinical deployment\n\n## File Structure\n\n```\ntriage_bench/\n├── README.md # This file\n├── main.py # Main evaluation script\n├── paired_analysis.py # Statistical comparison tool\n├── vignettes/ # Clinical vignette datasets\n│ └── semigran_vignettes.jsonl\n└── results/ # Output directory for results\n ├── medask_results_jul25/ # Benchmarking results from July 2025 study\n └── [timestamped results files]\n```\n\n## Published Results\n\nThe `results/medask_results_jul25/` folder contains comprehensive benchmarking results from our July 2025 evaluation study, where MedAsk was compared against leading LLMs including OpenAI's O3 and GPT-4.5. These results are described in detail in our blog post:\n\n**[Medical AI Triage Accuracy 2025: MedAsk Beats OpenAI's O3 & GPT-4.5](https://medask.tech/blogs/medical-ai-triage-accuracy-2025-medask-beats-openais-o3-gpt-4-5/)**\n\nThis study demonstrates MedAsk's superior performance in medical triage accuracy across multiple evaluation metrics.\n\n## Contributing\n\nWhen adding new vignette sets or models:\n1. Follow the existing JSONL format for vignettes\n2. Add model support in the client factory section of `main.py`\n3. Update this README with new options\n4. Test with multiple runs to ensure reproducibility\n\n## Citation\n\nIf you use this benchmark in research, please cite the associated publication and link to the MedAsk blog posts mentioned in the main project README."}, "47a87dc1cd87127666da54861d390eb96aa421fc": {"t": 1785764595.279757, "n": 8515, "h": "9bf3beffbf5e358e", "c": "# Triage Benchmark\n\nA comprehensive benchmark for evaluating Large Language Model (LLM) performance on medical triage classification tasks.\n\n## Overview\n\nThis benchmark evaluates how accurately LLMs can classify medical vignettes into appropriate triage categories:\n\n- **Emergency (em)**: Call 911 or go directly to the emergency room\n- **Non-Emergency (ne)**: See a doctor, but symptoms don't require immediate ER attention (within a week)\n- **Self-care (sc)**: Let the health issue resolve on its own and reassess in a few days\n\n## Features\n\n- **Multiple LLM Support**: OpenAI models (GPT-4o, GPT-4.5, O1, O3, etc.), DeepSeek models, and open-source models via Ollama (Llama, Qwen, DeepSeek R1, Mistral, etc.)\n- **Vignettes**: Semigran clinical vignettes\n- **Statistical Analysis**: Paired model comparison with McNemar tests\n- **Safety Metrics**: Tracks over-triage and under-triage tendencies\n- **Reproducible Results**: Supports multiple runs for statistical significance\n\n## Installation\n\nFrom the project root directory:\n\n```bash\n# Install dependencies\npip install -r requirements/development.txt\npip install -e .\n\n# Set API keys (for cloud models)\nexport KEY_OPENAI=\"sk-...\" # For OpenAI models\nexport KEY_DEEPSEEK=\"...\" # For DeepSeek models\n\n# Install Ollama (for open-source models)\n# Platform-specific installation:\n# Mac: Download from https://ollama.com/download or: brew install ollama\n# Linux: curl -fsSL https://ollama.com/install.sh | sh\n# Windows: Download installer from https://ollama.com/download\n# After installation, verify: ollama --version\n\n# Pull open-source models (examples)\nollama pull llama3.1\nollama pull qwen2.5\nollama pull deepseek-r1\nollama pull mistral\n```\n<｜tool▁call▁begin｜>\nread_file\n\n\n\n## Usage\n\n### Web UI (Recommended)\n\nLaunch the interactive Gradio web interface:\n\n```bash\ncd triage_bench\npython3 app.py\n```\n\nThe web UI will open in your browser at `http://localhost:7860`.\n\n#### Web UI Features:\n- **Live Progress**: Real-time progress updates showing vignettes completed / total\n- **Model Selection**: Easy dropdown selection for OpenAI, DeepSeek, and Ollama models\n- **Custom Models**: Support for custom model names or URLs\n- **Configuration**: Adjust runs (1-5), vignette count, and vignette sets\n- **Results Display**: Formatted summary metrics with accuracy breakdowns\n- **Downloadable Results**: JSON results available for download\n- **API Key Management**: Enter API keys directly in the UI (or use environment variables)\n\n#### Web UI Usage:\n1. Select model provider (OpenAI, DeepSeek, Ollama, or Custom)\n2. Choose specific model from dropdown (or enter custom model name)\n3. Configure benchmark settings (runs, vignette set, vignette count)\n4. Enter API keys if using cloud models (optional if set as environment variables)\n5. Click \"Run Benchmark\" to start evaluation\n6. Watch live progress updates as vignettes are processed\n7. View final summary metrics and download results as JSON\n\n### Command Line Interface\n\n```bash\ncd triage_bench\n\n# Run with default settings (DeepSeek chat, Semigran vignettes, 1 run)\npython3 main.py\n\n# Run with GPT-4o\npython3 main.py --model gpt-4o\n\n# Run with multiple passes for statistical robustness\npython3 main.py --model gpt-4o --runs 5\n\npython3 main.py --model gpt-4o --vignette_set semigran\n```\n\n### Available Models\n\n**OpenAI Models:**\n- `gpt-4o`\n- `gpt-4.5-preview` \n- `o1`, `o1-mini`\n- `o3`, `o3-mini`\n- `o4-mini`\n\n**DeepSeek Models:**\n- `deepseek-chat` (default)\n- `deepseek-reasoner`\n\n**Open-Source Models (via Ollama):**\n- **Llama**: `llama3.1`, `llama3`, `llama2`\n- **Qwen**: `qwen2.5`, `qwen2`, `qwen2.5:72b`\n- **DeepSeek R1**: `deepseek-r1`, `deepseek-r1-distill`\n- **Mistral**: `mistral`, `mixtral`, `mistral-small`\n\n> **Note**: Any model available in Ollama can be used. The benchmark auto-detects common model names (llama*, qwen*, deepseek-r1*, mistral*). You can also use the explicit format: `ollama+<model-name>`.\n\n**Prerequisites for Ollama models:**\n1. Install Ollama: `curl -fsSL https://ollama.com/install.sh | sh`\n2. Pull the model: `ollama pull <model-name>`\n3. Ensure Ollama is running: `ollama serve` (usually runs automatically)\n\n### Available Vignette Sets\n\n- `semigran`: 45 clinical vignettes from Semigran et al.\n\n### Example Commands\n\n```bash\n# Compare GPT-4o vs GPT-4.5 with 3 runs each\npython3 main.py --model gpt-4o --runs 3\npython3 main.py --model gpt-4.5-preview --runs 3\n\n# Test DeepSeek reasoning model\npython3 main.py --model deepseek-reasoner --vignette_set semigran --runs 2\n\n# Test open-source models via Ollama\npython3 main.py --model llama3.1 --runs 1\npython3 main.py --model qwen2.5 --runs 1\npython3 main.py --model deepseek-r1 --runs 1\npython3 main.py --model mistral --runs 1\n\n# Explicit Ollama format (optional)\npython3 main.py --model ollama+llama3.1 --runs 1\n```\n\n## Statistical Analysis\n\n### Paired Model Comparison\n\nCompare two models statistically using McNemar's test:\n\n```bash\npython3 paired_analysis.py results/model1_results.jsonl results/model2_results.jsonl\n```\n\n**Example Output:**\n```\n=== MODEL A ===\nOverall accuracy: 80.00% (36/45)\nAccuracy by triage level:\n em: 93.33% (14/15)\n ne: 60.00% (9/15)\n sc: 86.67% (13/15)\nSafety (at‑or‑above correct urgency): 88.89% (40/45)\nOver‑triage inclination (among incorrect): 44.44% (4/9)\n\n=== MODEL B ===\nOverall accuracy: 77.78% (35/45)\nSafety (at‑or‑above correct urgency): 95.56% (43/45)\n\n=== McNemar paired test ===\ndiscordant pairs: A‑right/B‑wrong = 5, A‑wrong/B‑right = 4\np‑value: 1.0 (exact=True)\nAccuracy difference (B − A): -2.22%\n```\n\n## Output Format\n\n### Results Files\n\nResults are saved as timestamped JSONL files in the `results/` directory:\n\n```\nresults/20250703T134639_gpt-4o_semigran_triage.jsonl\n```\n\nEach line contains:\n```json\n{\n \"run_id\": 1,\n \"case_id\": 1,\n \"true_urgency\": \"em\",\n \"llm_output\": \"em\",\n \"correct\": true,\n \"model\": \"gpt-4o\"\n}\n```\n\n### Evaluation Metrics\n\n- **Overall Accuracy**: Percentage of correct triage classifications\n- **Per-Level Accuracy**: Accuracy for each triage category (em/ne/sc)\n- **Safety Rate**: Percentage of predictions at or above correct urgency level\n- **Over-triage Rate**: Among incorrect predictions, percentage that over-triaged\n\n## Vignette Format\n\nVignettes are stored as JSONL files with the following structure:\n\n```json\n{\n \"urgency_level\": \"em\",\n \"correct_diagnosis\": \"Acute liver failure\",\n \"case_description\": \"A 48-year-old woman with altered mental status...\"\n}\n```\n\n## Safety Considerations\n\nThe benchmark uses a conservative safety framework:\n- **Under-triage** (predicting lower urgency than correct) is considered more dangerous\n- **Over-triage** (predicting higher urgency than correct) is safer but may strain resources\n- Safety metrics help evaluate model reliability for real-world deployment\n\n## Research Applications\n\nThis benchmark is useful for:\n- Comparing LLM performance on medical triage tasks\n- Evaluating safety vs. efficiency trade-offs\n- Testing prompt engineering approaches for medical AI\n- Validating model reliability before clinical deployment\n\n## File Structure\n\n```\ntriage_bench/\n├── README.md # This file\n├── main.py # Main evaluation script\n├── paired_analysis.py # Statistical comparison tool\n├── vignettes/ # Clinical vignette datasets\n│ └── semigran_vignettes.jsonl\n└── results/ # Output directory for results\n ├── medask_results_jul25/ # Benchmarking results from July 2025 study\n └── [timestamped results files]\n```\n\n## Published Results\n\nThe `results/medask_results_jul25/` folder contains comprehensive benchmarking results from our July 2025 evaluation study, where MedAsk was compared against leading LLMs including OpenAI's O3 and GPT-4.5. These results are described in detail in our blog post:\n\n**[Medical AI Triage Accuracy 2025: MedAsk Beats OpenAI's O3 & GPT-4.5](https://medask.tech/blogs/medical-ai-triage-accuracy-2025-medask-beats-openais-o3-gpt-4-5/)**\n\nThis study demonstrates MedAsk's superior performance in medical triage accuracy across multiple evaluation metrics.\n\n## Contributing\n\nWhen adding new vignette sets or models:\n1. Follow the existing JSONL format for vignettes\n2. Add model support in the client factory section of `main.py`\n3. Update this README with new options\n4. Test with multiple runs to ensure reproducibility\n\n## Citation\n\nIf you use this benchmark in research, please cite the associated publication and link to the MedAsk blog posts mentioned in the main project README."}, "d3e700b33c81644a1d5baa6be1a103a6a720f988": {"t": 1785764596.7328382, "n": 8251, "h": "0544057299bb29a5", "c": "# Triage Benchmark\n\nA comprehensive benchmark for evaluating Large Language Model (LLM) performance on medical triage classification tasks.\n\n## Overview\n\nThis benchmark evaluates how accurately LLMs can classify medical vignettes into appropriate triage categories:\n\n- **Emergency (em)**: Call 911 or go directly to the emergency room\n- **Non-Emergency (ne)**: See a doctor, but symptoms don't require immediate ER attention (within a week)\n- **Self-care (sc)**: Let the health issue resolve on its own and reassess in a few days\n\n## Features\n\n- **Multiple LLM Support**: OpenAI models (GPT-4o, GPT-4.5, O1, O3, etc.), DeepSeek models, and open-source models via Ollama (Llama, Qwen, DeepSeek R1, Mistral, etc.)\n- **Vignettes**: Semigran clinical vignettes\n- **Statistical Analysis**: Paired model comparison with McNemar tests\n- **Safety Metrics**: Tracks over-triage and under-triage tendencies\n- **Reproducible Results**: Supports multiple runs for statistical significance\n\n## Installation\n\nFrom the project root directory:\n\n```bash\n# Install dependencies\npip install -r requirements/development.txt\npip install -e .\n\n# Set API keys (for cloud models)\nexport KEY_OPENAI=\"sk-...\" # For OpenAI models\nexport KEY_DEEPSEEK=\"...\" # For DeepSeek models\n\n# Install Ollama (for open-source models)\ncurl -fsSL https://ollama.com/install.sh | sh\n\n# Pull open-source models (examples)\nollama pull llama3.1\nollama pull qwen2.5\nollama pull deepseek-r1\nollama pull mistral\n```\n\n## Usage\n\n### Web UI (Recommended)\n\nLaunch the interactive Gradio web interface:\n\n```bash\ncd triage_bench\npython3 app.py\n```\n\nThe web UI will open in your browser at `http://localhost:7860`.\n\n#### Web UI Features:\n- **Live Progress**: Real-time progress updates showing vignettes completed / total\n- **Model Selection**: Easy dropdown selection for OpenAI, DeepSeek, and Ollama models\n- **Custom Models**: Support for custom model names or URLs\n- **Configuration**: Adjust runs (1-5), vignette count, and vignette sets\n- **Results Display**: Formatted summary metrics with accuracy breakdowns\n- **Downloadable Results**: JSON results available for download\n- **API Key Management**: Enter API keys directly in the UI (or use environment variables)\n\n#### Web UI Usage:\n1. Select model provider (OpenAI, DeepSeek, Ollama, or Custom)\n2. Choose specific model from dropdown (or enter custom model name)\n3. Configure benchmark settings (runs, vignette set, vignette count)\n4. Enter API keys if using cloud models (optional if set as environment variables)\n5. Click \"Run Benchmark\" to start evaluation\n6. Watch live progress updates as vignettes are processed\n7. View final summary metrics and download results as JSON\n\n### Command Line Interface\n\n```bash\ncd triage_bench\n\n# Run with default settings (DeepSeek chat, Semigran vignettes, 1 run)\npython3 main.py\n\n# Run with GPT-4o\npython3 main.py --model gpt-4o\n\n# Run with multiple passes for statistical robustness\npython3 main.py --model gpt-4o --runs 5\n\npython3 main.py --model gpt-4o --vignette_set semigran\n```\n\n### Available Models\n\n**OpenAI Models:**\n- `gpt-4o`\n- `gpt-4.5-preview` \n- `o1`, `o1-mini`\n- `o3`, `o3-mini`\n- `o4-mini`\n\n**DeepSeek Models:**\n- `deepseek-chat` (default)\n- `deepseek-reasoner`\n\n**Open-Source Models (via Ollama):**\n- **Llama**: `llama3.1`, `llama3`, `llama2`\n- **Qwen**: `qwen2.5`, `qwen2`, `qwen2.5:72b`\n- **DeepSeek R1**: `deepseek-r1`, `deepseek-r1-distill`\n- **Mistral**: `mistral`, `mixtral`, `mistral-small`\n\n> **Note**: Any model available in Ollama can be used. The benchmark auto-detects common model names (llama*, qwen*, deepseek-r1*, mistral*). You can also use the explicit format: `ollama+<model-name>`.\n\n**Prerequisites for Ollama models:**\n1. Install Ollama: `curl -fsSL https://ollama.com/install.sh | sh`\n2. Pull the model: `ollama pull <model-name>`\n3. Ensure Ollama is running: `ollama serve` (usually runs automatically)\n\n### Available Vignette Sets\n\n- `semigran`: 45 clinical vignettes from Semigran et al.\n\n### Example Commands\n\n```bash\n# Compare GPT-4o vs GPT-4.5 with 3 runs each\npython3 main.py --model gpt-4o --runs 3\npython3 main.py --model gpt-4.5-preview --runs 3\n\n# Test DeepSeek reasoning model\npython3 main.py --model deepseek-reasoner --vignette_set semigran --runs 2\n\n# Test open-source models via Ollama\npython3 main.py --model llama3.1 --runs 1\npython3 main.py --model qwen2.5 --runs 1\npython3 main.py --model deepseek-r1 --runs 1\npython3 main.py --model mistral --runs 1\n\n# Explicit Ollama format (optional)\npython3 main.py --model ollama+llama3.1 --runs 1\n```\n\n## Statistical Analysis\n\n### Paired Model Comparison\n\nCompare two models statistically using McNemar's test:\n\n```bash\npython3 paired_analysis.py results/model1_results.jsonl results/model2_results.jsonl\n```\n\n**Example Output:**\n```\n=== MODEL A ===\nOverall accuracy: 80.00% (36/45)\nAccuracy by triage level:\n em: 93.33% (14/15)\n ne: 60.00% (9/15)\n sc: 86.67% (13/15)\nSafety (at‑or‑above correct urgency): 88.89% (40/45)\nOver‑triage inclination (among incorrect): 44.44% (4/9)\n\n=== MODEL B ===\nOverall accuracy: 77.78% (35/45)\nSafety (at‑or‑above correct urgency): 95.56% (43/45)\n\n=== McNemar paired test ===\ndiscordant pairs: A‑right/B‑wrong = 5, A‑wrong/B‑right = 4\np‑value: 1.0 (exact=True)\nAccuracy difference (B − A): -2.22%\n```\n\n## Output Format\n\n### Results Files\n\nResults are saved as timestamped JSONL files in the `results/` directory:\n\n```\nresults/20250703T134639_gpt-4o_semigran_triage.jsonl\n```\n\nEach line contains:\n```json\n{\n \"run_id\": 1,\n \"case_id\": 1,\n \"true_urgency\": \"em\",\n \"llm_output\": \"em\",\n \"correct\": true,\n \"model\": \"gpt-4o\"\n}\n```\n\n### Evaluation Metrics\n\n- **Overall Accuracy**: Percentage of correct triage classifications\n- **Per-Level Accuracy**: Accuracy for each triage category (em/ne/sc)\n- **Safety Rate**: Percentage of predictions at or above correct urgency level\n- **Over-triage Rate**: Among incorrect predictions, percentage that over-triaged\n\n## Vignette Format\n\nVignettes are stored as JSONL files with the following structure:\n\n```json\n{\n \"urgency_level\": \"em\",\n \"correct_diagnosis\": \"Acute liver failure\",\n \"case_description\": \"A 48-year-old woman with altered mental status...\"\n}\n```\n\n## Safety Considerations\n\nThe benchmark uses a conservative safety framework:\n- **Under-triage** (predicting lower urgency than correct) is considered more dangerous\n- **Over-triage** (predicting higher urgency than correct) is safer but may strain resources\n- Safety metrics help evaluate model reliability for real-world deployment\n\n## Research Applications\n\nThis benchmark is useful for:\n- Comparing LLM performance on medical triage tasks\n- Evaluating safety vs. efficiency trade-offs\n- Testing prompt engineering approaches for medical AI\n- Validating model reliability before clinical deployment\n\n## File Structure\n\n```\ntriage_bench/\n├── README.md # This file\n├── main.py # Main evaluation script\n├── paired_analysis.py # Statistical comparison tool\n├── vignettes/ # Clinical vignette datasets\n│ └── semigran_vignettes.jsonl\n└── results/ # Output directory for results\n ├── medask_results_jul25/ # Benchmarking results from July 2025 study\n └── [timestamped results files]\n```\n\n## Published Results\n\nThe `results/medask_results_jul25/` folder contains comprehensive benchmarking results from our July 2025 evaluation study, where MedAsk was compared against leading LLMs including OpenAI's O3 and GPT-4.5. These results are described in detail in our blog post:\n\n**[Medical AI Triage Accuracy 2025: MedAsk Beats OpenAI's O3 & GPT-4.5](https://medask.tech/blogs/medical-ai-triage-accuracy-2025-medask-beats-openais-o3-gpt-4-5/)**\n\nThis study demonstrates MedAsk's superior performance in medical triage accuracy across multiple evaluation metrics.\n\n## Contributing\n\nWhen adding new vignette sets or models:\n1. Follow the existing JSONL format for vignettes\n2. Add model support in the client factory section of `main.py`\n3. Update this README with new options\n4. Test with multiple runs to ensure reproducibility\n\n## Citation\n\nIf you use this benchmark in research, please cite the associated publication and link to the MedAsk blog posts mentioned in the main project README."}}}} |