1 line
232 KiB
JSON
1 line
232 KiB
JSON
{"repo": "aq-medai/medicalaibencheval", "files": {"README.md": {"38d096413a0be30020cd15fa6db322372ae7d24a": {"t": 1785746653.480065, "n": 38909, "h": "53d8e02a4a74d233", "c": "# Medical AI Evaluation Framework: Clinical Benchmark Dataset and Automated Assessment Pipeline\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n - [Installation](#installation)\n - [Basic Usage](#basic-usage)\n- [System Architecture](#system-architecture)\n- [Data Format](#data-format)\n - [Input Format](#input-format)\n - [Output Format](#output-format)\n- [Configuration](#configuration)\n - [Basic Configuration](#basic-configuration)\n - [Advanced Configuration](#advanced-configuration)\n- [Usage Guide](#usage-guide)\n - [Command Line Interface](#command-line-interface)\n - [Python API](#python-api)\n - [Examples](#examples)\n- [Evaluation Framework](#evaluation-framework)\n - [Scoring System](#scoring-system)\n - [Evaluation Metrics](#evaluation-metrics)\n - [Human Expert Comparison Evaluation](#human-expert-comparison-evaluation)\n- [Dataset](#dataset)\n - [Thoracic Surgery Dataset](#thoracic-surgery-dataset)\n - [Data Quality](#data-quality)\n- [Advanced Features](#advanced-features)\n - [Parallel Processing](#parallel-processing)\n - [Voting Strategies](#voting-strategies)\n - [Custom Models](#custom-models)\n- [Development](#development)\n - [Technical Documentation](#technical-documentation)\n - [API Reference](#api-reference)\n - [Troubleshooting](#troubleshooting)\n- [Community](#community)\n - [Contributing](#contributing)\n - [Citation](#citation)\n- [License](#license)\n\n## Overview\n\nThis Medical AI Evaluation Framework provides a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this framework includes both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.\n\nThe framework addresses the critical need for standardized evaluation of AI clinical decision-making by providing:\n- **Clinically Grounded Assessment**: Evaluation criteria based on real medical guidelines and expert knowledge\n- **Automated Pipeline**: Streamlined processing from raw responses to detailed performance metrics\n- **Multi-Model Support**: Simultaneous evaluation of multiple AI models with comparative analysis\n- **Scalable Architecture**: Efficient processing of large datasets with parallel execution capabilities\n\n## Key Features\n\n- **🏥 Medical-Specific Evaluation**: Specialized rubrics for clinical scenarios with positive/negative scoring\n- **🔄 Parallel Processing**: Simultaneous execution of Met/Not Met review and irrelevant content detection\n- **📊 Comprehensive Analytics**: Detailed statistical analysis with visualization reports\n- **🎯 Multi-Model Assessment**: Support for evaluating multiple AI models simultaneously\n- **⚙️ Flexible Configuration**: Customizable models, voting strategies, and evaluation parameters\n- **📈 Rich Visualization**: Automated generation of performance charts and comparative analysis\n- **🔧 Modular Design**: Independent modules for different evaluation stages\n- **📋 Standardized Output**: Consistent Excel-based reporting with detailed metrics\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd medical-ai-bench-eval\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Basic Usage\n\n```bash\n# Run complete evaluation pipeline\npython medical_evaluation_pipeline.py input_data.xlsx\n\n# With custom output file\npython medical_evaluation_pipeline.py input_data.xlsx -o results/evaluation_results.xlsx\n\n# Enable verbose logging\npython medical_evaluation_pipeline.py input_data.xlsx -v\n```\n\n## System Architecture\n\nThe system processes medical AI responses through a sophisticated pipeline that includes:\n\n1. **Input Processing**: Reads Excel files containing medical questions, evaluation rubrics, and AI model responses\n2. **Parallel Evaluation**: Simultaneously executes Met/Not Met review and irrelevant content detection\n3. **Intelligent Scoring**: Calculates comprehensive scores based on clinical evaluation criteria\n4. **Analysis & Reporting**: Generates detailed statistical reports and visualizations\n\n## System Architecture\n\n```\nInput Excel File\n↓\n┌─────────────────────────────────────────┐\n│ Parallel Processing Phase │\n├─────────────────┬─────────────────────┤\n│ Step 1: Met Review │ Step 2: Irrelevant Content │\n│ - Multi-model Review │ - Content Extraction │\n│ - Voting Decision │ - Level Assessment │\n│ - Result Summary │ - Voting Classification │\n└─────────────────┴─────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Result Merging │\n│ - Intelligent merging of parallel results │\n│ - Data integrity check │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 3: Score Calculation │\n│ - Multi-dimensional score statistics │\n│ - Irrelevant content deduction │\n│ - Normalization processing │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 4: Data Analysis & Visualization │\n│ - Statistical analysis reports │\n│ - Diverse charts │\n│ - CSV data export │\n└─────────────────────────────────────────┘\n↓\nOutput Excel File + Analysis Report\n```\n\n## Data Format\n\n### Input Format\n\nThe system accepts Excel files with the following structure:\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n#### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n### Output Format\n\nThe system generates multiple output files:\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n#### Scoring Metrics\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Configuration\n\n### Basic Configuration\n\n### Environment Requirements\n- **Python**: 3.12+\n- **Operating System**: Windows, macOS, Linux\n- **Core Dependencies**: pandas, numpy, matplotlib, seaborn, asyncio, langchain, openpyxl, xlsxwriter\n\n### Installation Steps\n\n1. **Clone the repository**:\n```bash\ngit clone <repository-url>\ncd medical-ai-bench-eval\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Configure API Keys** (if using external AI model services):\n\n ### API Configuration Guide\n\n #### Environment Variables Setup\n\n **Linux/Mac:**\n ```bash\n # OpenAI\n export OPENAI_API_KEY=\"sk-your-actual-key\"\n export OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n\n # Claude (Anthropic)\n export ANTHROPIC_API_KEY=\"sk-ant-your-actual-key\"\n export ANTHROPIC_BASE_URL=\"https://api.anthropic.com\"\n\n # Google Gemini\n export GOOGLE_API_KEY=\"your-google-api-key\"\n export GOOGLE_BASE_URL=\"https://generativelanguage.googleapis.com/v1beta\"\n\n # Moonshot Kimi\n export MOONSHOT_API_KEY=\"sk-your-moonshot-key\"\n export MOONSHOT_BASE_URL=\"https://api.moonshot.cn/v1\"\n\n # Alibaba Qwen\n export DASHSCOPE_API_KEY=\"your-dashscope-key\"\n export DASHSCOPE_BASE_URL=\"https://dashscope.aliyuncs.com/api/v1\"\n\n # Baichuan\n export BAICHUAN_API_KEY=\"your-baichuan-key\"\n export BAICHUAN_BASE_URL=\"https://api.baichuan-ai.com/v1\"\n\n # DeepSeek\n export DEEPSEEK_API_KEY=\"sk-your-deepseek-key\"\n export DEEPSEEK_BASE_URL=\"https://api.deepseek.com/v1\"\n\n # Zhipu ChatGLM\n export ZHIPU_API_KEY=\"your-zhipu-key\"\n export ZHIPU_BASE_URL=\"https://open.bigmodel.cn/api/paas/v4\"\n ```\n\n **Windows PowerShell:**\n ```powershell\n $Env:OPENAI_API_KEY=\"sk-your-actual-key\"\n $Env:OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n ```\n\n #### Model Configuration\n\n **⚠️ IMPORTANT: OpenAI-Compatible API Only**\n \n This system currently supports **ONLY OpenAI-compatible API interfaces**. All models must provide OpenAI-compatible endpoints, regardless of the actual provider.\n\n **🌐 Third-Party Multi-Model Platforms (Recommended)**\n \n For models that don't natively support OpenAI-compatible APIs (like Claude, Gemini), you can use third-party platforms that provide unified OpenAI-compatible access:\n \n - **[OpenRouter](https://openrouter.ai/)** - Access Claude, Gemini, GPT, and many other models via OpenAI-compatible API\n - **[Together AI](https://together.ai/)** - Various open-source and commercial models\n - **[Anyscale](https://anyscale.com/)** - Multiple LLMs via OpenAI-compatible endpoints\n\n Edit `config_model.yaml` to configure your preferred models:\n\n ```yaml\n models:\n m1: # OpenAI GPT-4 (Native)\n api_key_env: \"OPENAI_API_KEY\"\n api_base_env: \"OPENAI_BASE_URL\"\n model: \"gpt-4-turbo-preview\"\n temperature: 0.2\n max_tokens: 4000\n\n m2: # Kimi (OpenAI-Compatible)\n api_key_env: \"MOONSHOT_API_KEY\"\n api_base_env: \"MOONSHOT_BASE_URL\"\n model: \"kimi-latest\"\n temperature: 0.2\n max_tokens: 4000\n\n m3: # Qwen (OpenAI-Compatible)\n api_key_env: \"DASHSCOPE_API_KEY\"\n api_base_env: \"DASHSCOPE_BASE_URL\"\n model: \"qwen-max\"\n temperature: 0.1\n max_tokens: 4000\n\n m4: # DeepSeek (OpenAI-Compatible)\n api_key_env: \"DEEPSEEK_API_KEY\"\n api_base_env: \"DEEPSEEK_BASE_URL\"\n model: \"deepseek-chat\"\n temperature: 0.2\n max_tokens: 4000\n\n m5: # Local Model (via OpenAI-Compatible Server)\n api_key_env: \"LOCAL_API_KEY\"\n api_base_env: \"LOCAL_BASE_URL\"\n model: \"llama3-medical\"\n temperature: 0.2\n max_tokens: 4000\n\n # OpenRouter Examples (uncomment to use)\n # m6: # Claude via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"anthropic/claude-3-5-sonnet-20241022\"\n # temperature: 0.2\n # max_tokens: 4000\n\n # m7: # Gemini via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"google/gemini-pro-1.5\"\n # temperature: 0.1\n # max_tokens: 4000\n ```\n\n #### Supported vs Unsupported Providers\n\n | Provider | Status | API Type | Notes |\n |----------|--------|----------|-------|\n | **OpenAI** | ✅ Supported | Native OpenAI | Direct support |\n | **OpenRouter** | ✅ Supported | Multi-Model Platform | Access Claude, Gemini, GPT via OpenAI-compatible API |\n | **Together AI** | ✅ Supported | Multi-Model Platform | Various models via OpenAI-compatible API |\n | **Anyscale** | ✅ Supported | Multi-Model Platform | Multiple LLMs via OpenAI-compatible API |\n | **Kimi/Moonshot** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Qwen/DashScope** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **DeepSeek** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Baichuan** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Zhipu** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Claude/Anthropic** | ⚠️ Via OpenRouter | Native Anthropic API | Use OpenRouter for OpenAI-compatible access |\n | **Gemini/Google** | ⚠️ Via OpenRouter | Native Google API | Use OpenRouter for OpenAI-compatible access |\n\n #### Medical Model Recommendations (OpenAI-Compatible Only)\n\n | Use Case | Recommended Models | Temperature | Notes |\n |----------|-------------------|-------------|--------|\n | **Diagnostic Tasks** | gpt-4-turbo, deepseek-chat | 0.1-0.2 | High accuracy |\n | **Chinese Medical** | qwen-max, baichuan-medical | 0.2-0.3 | Better CJK support |\n | **Cost-effective** | deepseek-chat, kimi | 0.2-0.3 | Good balance |\n | **Real-time** | kimi, deepseek-chat | 0.1-0.2 | Fast response |\n | **Privacy-focused** | local-llama3-medical | 0.2-0.3 | On-premise via OpenAI-compatible server |\n\n #### Quick Model Setup Commands\n\n **For GPT-4 testing:**\n ```bash\n export OPENAI_API_KEY=\"your-key-here\"\n # Run with GPT-4\n python medical_evaluation_pipeline.py input.xlsx --judge-models m1\n ```\n\n **For Chinese medical models:**\n ```bash\n export DASHSCOPE_API_KEY=\"your-key-here\"\n # Run with Qwen\n python medical_evaluation_pipeline.py input.xlsx --judge-models m4\n ```\n\n **For mixed model evaluation:**\n ```bash\n export OPENAI_API_KEY=\"your-key\"\n export DASHSCOPE_API_KEY=\"your-key\"\n # Use GPT-4 and Qwen for evaluation\n python medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m4 \\\n --extract-model m5\n\n### Basic Usage\n\n1. **Prepare input data**:\n - Excel file format\n - Contains columns for questions, scoring criteria (JSON format), model responses, etc.\n - Default column names can be configured in `config.yaml`\n\n2. **Run the complete evaluation pipeline**:\n```bash\npython medical_evaluation_pipeline.py input.xlsx -o output.xlsx\n```\n\n3. **View results**:\n - Output Excel file contains detailed review results and scores\n - Data analysis reports are saved in the `data/output/analysis/` directory\n\n## ⚙️ System Configuration\n\n### Configuration File (config.yaml)\n\n```yaml\n# File path configuration\nfiles:\n input_file: \"data/medical_evaluation_result.xlsx\"\n output_file: \"data/output/processed_result.xlsx\"\n\n# Process control\npipeline:\n disable_met_nomet: false # Whether to disable Met review\n disable_irrelevant_extraction: false # Whether to disable irrelevant content detection\n exclude_irrelevant_scoring: false # Whether to exclude irrelevant content from scoring\n disable_data_analysis: false # Whether to disable data analysis\n analysis_config_file: \"config_visualization.yaml\"\n\n# Model configuration\nmodels:\n judge_models: [\"m1\", \"m2\", \"m3\"] # Met review models\n extract_model: \"m5\" # Irrelevant content extraction model\n grade_models: [\"m1\", \"m2\", \"m3\"] # Irrelevant content grading models\n voting_strategy: \"conservative\" # Voting strategy\n\n# Column name mapping\ncolumns:\n question_col: \"question\"\n rubric_col: \"final_merged_json\"\n answer_columns:\n - \"gpt_5_answer\"\n - \"gemini_2_5_pro_answer\"\n - \"claude_opus_4_answer\"\n\n# Scoring system\npoint:\n A1: 5\n A2: 3\n A3: 1\n S1: -1\n S2: -2\n S3: -3\n S4: -4\n```\n\n### Command Line Arguments\n\n```bash\n# Process control\n--disable-met-nomet # Skip Met review step\n--disable-irrelevant # Skip irrelevant content detection\n--exclude-irrelevant-scoring # Exclude irrelevant content deduction from scoring\n--disable-data-analysis # Skip data analysis\n\n# Model configuration\n--judge-models m1 m2 # Specify Met review models\n--extract-model m3 # Specify irrelevant content extraction model\n--grade-models m1 m2 m4 # Specify irrelevant content grading models\n--voting-strategy majority # Voting strategy: conservative/majority/average\n\n# Data column configuration\n--question-col \"Question\" # Question column name\n--rubric-col \"Evaluation Points\" # Evaluation points column name\n--answer-columns \"Response1\" \"Response2\" \"Response3\" # Model response column names\n```\n\n## Evaluation Framework\n\n### Scoring System\n\nThe GAPS evaluation framework employs a multi-dimensional scoring system that combines positive and negative points for comprehensive assessment:\n\n**Positive Points (A-class)**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n\n**Negative Points (S-class)**:\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Evaluation Metrics\n\nThe system provides various evaluation metrics to comprehensively measure AI model performance:\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n### Human Expert Comparison Evaluation\n\nThis evaluation aims to examine the consistency performance of multiple large language models with expert annotations in specialized domain text key point extraction tasks. Through comprehensive comparison of three mainstream models and their fusion systems, combined with independent annotation results from five domain experts, we have reached the following key conclusions:\n\n#### 1. Excellent Overall AI System Performance\nThe fusion model achieved an overall consistency rate of **90.00%** with expert consensus, with a Cohen's Kappa coefficient of **0.77**, entering the \"substantial agreement\" range (Landis & Koch standard) and approaching the threshold of \"near perfect\" agreement. This level falls within the consistency range among the five experts (88.50%–92.00%), indicating that current AI systems possess professional reliability approaching that of human experts in clinical judgment capabilities.\n\n## 📊 Thoracic Surgery Evaluation Dataset\n\n### Dataset Overview\n\nWe provide a specialized **Thoracic Surgery Evaluation Dataset** (`data/input/GAPS-NSCLC-preview.xlsx`) containing 92 carefully curated clinical cases for evaluating AI models' performance in thoracic surgery scenarios. This dataset focuses on complex clinical decision-making in chest surgery, particularly for non-small cell lung cancer (NSCLC) staging and treatment planning.\n\n### Dataset Structure\n\n| Column Name | Description | Data Type |\n|-------------|-------------|-----------|\n| **question** | Clinical questions covering thoracic surgery scenarios | Text |\n| **分类** (Category) | Medical specialty classification | Text |\n| **rubrics** | Evaluation criteria in JSON format with scoring levels | JSON Array |\n| **gpt_5_answer** | GPT-4 model responses to clinical questions | Text |\n| **gemini_2_5_pro_answer** | Gemini 2.5 Pro model responses | Text |\n| **claude_opus_4_answer** | Claude Opus model responses | Text |\n\n### Clinical Focus Areas\n\nThe dataset covers critical aspects of thoracic surgery, including:\n\n- **Pre-operative Evaluation**: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)\n- **Diagnostic Procedures**: EBUS-TBNA, mediastinoscopy, PET-CT interpretation\n- **Staging Assessment**: TNM staging, mediastinal lymph node evaluation\n- **Treatment Planning**: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions\n- **Risk Assessment**: Pulmonary function evaluation, cardiac risk stratification\n- **Molecular Diagnostics**: EGFR, ALK, PD-L1 testing strategies\n\n### Evaluation Rubrics Format\n\nEach question includes detailed evaluation criteria in JSON format:\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Should perform enhanced chest CT covering thoracic inlet to upper abdomen\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"PET-CT is essential for accurate N-staging and excluding distant metastases\",\n \"level\": \"A2\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Incorrectly generalizing upper sulcus tumor MRI requirements to all IIB/IIIA patients\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n]\n```\n\n**Scoring Levels**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Sample Clinical Question\n\n**Question**: \"For NSCLC patients with IIB-IIIA staging, what comprehensive pre-treatment evaluation should be performed?\"\n\n**Key Evaluation Points**:\n- Pathological diagnosis and molecular characterization\n- Accurate clinical staging (T, N, M assessment)\n- Invasive mediastinal staging for N2 evaluation\n- Pulmonary and cardiac function assessment\n- Multidisciplinary team (MDT) treatment planning\n\n### Usage with GAPS Framework\n\nThis dataset is designed to work seamlessly with the GAPS evaluation pipeline:\n\n```bash\n# Evaluate thoracic surgery dataset\npython medical_evaluation_pipeline.py data/input/GAPS-NSCLC-preview.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -o results/thoracic_surgery_evaluation.xlsx\n```\n\n### Clinical Validation\n\nThe dataset has been validated by:\n- **Board-certified thoracic surgeons**\n- **Pulmonary oncologists** \n- **Medical imaging specialists**\n- **Pathologists specializing in lung cancer**\n\nAll evaluation criteria are based on current clinical guidelines including:\n- NCCN Guidelines for Non-Small Cell Lung Cancer\n- ESTS Guidelines for Intraoperative Lymph Node Staging\n- IASLC Staging Manual in Thoracic Oncology\n\n### Research Applications\n\nThis dataset enables research in:\n- **AI Clinical Decision Support**: Evaluating AI models' ability to provide accurate clinical recommendations\n- **Medical Education**: Training and assessment of clinical reasoning skills\n- **Quality Assurance**: Benchmarking AI systems against established clinical standards\n- **Comparative Analysis**: Cross-model performance evaluation in specialized medical domains\n\n### Data Quality Metrics\n\n- **Total Cases**: 92 clinical scenarios\n- **Completeness**: 100% data coverage across all columns\n- **Clinical Diversity**: Covers full spectrum of IIB-IIIA NSCLC presentations\n- **Expert Validation**: All cases reviewed by multidisciplinary clinical team\n\n---\n\n## Input Data Format\n\n### Excel File Structure\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n## Output Results\n\n### Main Output Files\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n### Scoring Metrics Description\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Advanced Features\n\n### 1. Parallel Processing Optimization\n\nThe system supports parallel execution of Step 1 (Met Review) and Step 2 (Irrelevant Content Detection):\n\n```bash\n# Enable all steps (auto parallel)\npython medical_evaluation_pipeline.py input.xlsx\n\n# Enable only Met Review\npython medical_evaluation_pipeline.py input.xlsx --disable-irrelevant\n\n# Enable only Irrelevant Content Detection\npython medical_evaluation_pipeline.py input.xlsx --disable-met-nomet\n```\n\n### 2. Voting Strategy\n\nSupports three voting strategies for processing multi-model results:\n- **conservative**: Conservative strategy, prioritizes more severe ratings\n- **majority**: Majority voting, selects the result with the most votes\n- **average**: Average strategy, calculates based on numerical average\n\n```bash\npython medical_evaluation_pipeline.py input.xlsx --voting-strategy conservative\n```\n\n### 3. Custom Model Configuration\n\n```bash\n# Use specific review model combinations\npython medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m2 \\\n --extract-model m5 \\\n --grade-models m1 m3 m4\n```\n\n### 4. Data Analysis Customization\n\nCreate `config_visualization.yaml` for custom analysis configuration:\n\n```yaml\nanalysis:\n input_file: \"data/processed_result.xlsx\"\n columns:\n model_scores:\n GPT-5: \"gpt_5_answer_judged_json_scores\"\n Gemini-2.5-Pro: \"gemini_2_5_pro_answer_judged_json_scores\"\n Claude-Opus-4: \"claude_opus_4_answer_judged_json_scores\"\n\nvisualization:\n charts:\n bar_overall: true\n boxplot_distribution: true\n heatmap_category: true\n radar_chart: true\n style:\n figure_size: [24, 16]\n dpi: 300\n```\n\n## 🔧 Independent Module Usage\n\n### 1. Run Met Review Separately\n\n```python\nimport asyncio\nfrom clinical_answer_judge import MultiModelMergedJsonJudge\n\nasync def run_judge():\n judge = MultiModelMergedJsonJudge(\n judge_models=['m1', 'm2', 'm3'],\n question_col='question',\n rubric_col='rubrics'\n )\n result = await judge.process_excel_file('input.xlsx', 'output.xlsx')\n return result\n\n# Run the async function\nresult = asyncio.run(run_judge())\n```\n\n### 2. Run Score Calculation Separately\n\n```python\nfrom clinical_scoring_calculator import process_excel_data\n\nresult = process_excel_data(\n input_file='judged_data.xlsx',\n output_file='scored_data.xlsx',\n model_columns=['gpt_5_answer_judged_json']\n)\n```\n\n### 3. Run Data Analysis Separately\n\n```python\nfrom enhanced_analyzer import EnhancedMedicalAnalyzer\n\nanalyzer = EnhancedMedicalAnalyzer('config_visualization.yaml')\nanalyzer.run_analysis()\n```\n\n## Usage Examples\n\n### Example 1: Complete Process Handling\n\n```bash\n# Process complete medical evaluation data\npython medical_evaluation_pipeline.py medical_data.xlsx \\\n -o results/final_report.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -v\n```\n\n### Example 2: Quick Testing\n\n```bash\n# Process only first 10 rows of data\npython medical_evaluation_pipeline.py test_data.xlsx \\\n --max_rows 10 \\\n --disable-data-analysis\n```\n\n### Example 3: Custom Column Names\n\n```bash\n# Use Chinese column names\npython medical_evaluation_pipeline.py data.xlsx \\\n --question-col \"Clinical Question\" \\\n --rubric-col \"Scoring Criteria\" \\\n --answer-columns \"GPT Response\" \"Claude Response\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **File Not Found Error**\n - Solution: Check if the input file path is correct\n\n2. **Column Name Mismatch**\n - Solution: Use `--question-col`, `--rubric-col` and other parameters to specify correct column names\n\n3. **Insufficient Memory**\n - Solution: Use `--max_rows` parameter to limit the number of rows processed\n\n4. **API Call Failure**\n - Solution: Check model configuration and network connection\n\n### Debug Mode\n\n```bash\n# Enable verbose logging to view processing details\npython medical_evaluation_pipeline.py input.xlsx -v\n\n# Keep temporary files for debugging\n# Modify the cleanup() call in the code\n```\n\n## Technical Documentation\n\n### Core Modules\n\n- `medical_evaluation_pipeline.py` - Main pipeline controller\n- `clinical_answer_judge.py` - Met/Not Met review module\n- `clinical_scoring_calculator.py` - Score calculation module\n- `enhanced_analyzer.py` - Data analysis and visualization module\n- `general_negative_points.py` - Irrelevant content processing module\n\n### File Structure\n\n```\nMedicalAiBenchEval/\n├── clinical_answer_judge.py # Multi-model review processor\n├── clinical_scoring_calculator.py # Score calculator\n├── medical_evaluation_pipeline.py # Main pipeline controller\n├── enhanced_analyzer.py # Data analyzer\n├── judge_engine.py # Review engine\n├── irrelevant_content_grading.py # Irrelevant content scoring\n├── extract_irrelevant_content.py # Irrelevant content extraction\n├── general_negative_points.py # Negative point processing\n├── config.yaml # Main configuration file\n├── config_model.yaml # Model configuration file\n├── config_visualization.yaml # Visualization configuration\n├── requirements.txt # Dependency list\n└── data/ # Data directory\n ├── input/ # Input files\n │ ├── GAPS-NSCLC-preview.xlsx # Thoracic surgery evaluation dataset (92 cases)\n │ └── community_contributions.xlsx # Community contributed cases\n └── output/ # Output files\n └── analysis/ # Analysis reports\n```\n\n### Data Flow\n\n1. **Input Phase**: Read Excel file, validate data format\n2. **Parallel Processing**: Simultaneously execute Met review and irrelevant content detection\n3. **Result Merging**: Intelligently merge outputs from both parallel steps\n4. **Score Calculation**: Calculate multi-dimensional scores based on merged results\n5. **Analysis Output**: Generate statistical reports and visualization charts\n\n### Performance Optimization\n\n- Asynchronous parallel processing improves efficiency\n- Batch processing reduces API call frequency\n- Intelligent retry mechanism handles network exceptions\n- Memory optimization handles large files\n\n## Contribution Guidelines\n\nWelcome to submit Issues and Pull Requests to improve system functionality.\n\n### Development Environment Setup\n\n```bash\ngit clone <repository>\ncd medical-ai-bench-eval\npip install -r requirements.txt\n```\n\n### Code Standards\n\n- Follow PEP 8 coding standards\n- Add detailed docstrings\n- Include unit tests\n- Update README documentation\n\n## Citation\n\nIf you use this system in your research, please cite our paper:\n\n**English:**\n> Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Jun Wang, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan Yang (2025). GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians. *arXiv preprint* arXiv:2510.13734. https://arxiv.org/abs/2510.13734\n\n```bibtex\n@article{chen2025gaps,\n title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},\n author={Chen, Xiuyuan and Sun, Tao and Su, Dexin and Yu, Ailing and Liu, Junwei and Chen, Zhe and Jin, Gangzeng and Wang, Xin and Liu, Jingnan and Xiao, Hansong and Zhou, Hualei and Tao, Dongjie and Guo, Chunxiao and Yang, Minghui and Xia, Yuan and Zhao, Jing and Fan, Qianrui and Wang, Yanyun and Zhen, Shuai and Chen, Kezhong and Wang, Jun and Sun, Zewen and Zhao, Heng and Guan, Tian and Wang, Shaodong and Chang, Geyun and Deng, Jiaming and Chen, Hongchengcheng and Feng, Kexin and Li, Ruzhen and Geng, Jiayi and Zhao, Changtai and Wang, Jun and Lin, Guihu and Li, Peihao and Liu, Liqi and Wei, Peng and Wang, Jian and Gu, Jinjie and Wang, Ping and Yang, Fan},\n journal={arXiv preprint arXiv:2510.13734},\n year={2025},\n url={https://arxiv.org/abs/2510.13734}\n}\n```\n\n## 🤝 Community Contribution / 共创呼吁\n\nWe welcome contributions from the medical AI community to expand and improve our evaluation dataset! Your expertise can help build a more comprehensive benchmark for medical AI systems.\n\n### 📋 How to Contribute\n\n#### Option 1: Submit an Issue\nCreate a new issue with the label `data-contribution` using the following template:\n\n```markdown\n**Medical Question**\n[Provide your clinical question here]\n\n**Evaluation Rubrics (JSON Format)**\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Your evaluation criterion description\",\n \"level\": \"A1|A2|A3|S1|S2|S3|S4\",\n \"type\": \"positive|negative\"\n },\n {\n \"id\": 2,\n \"claim\": \"Another evaluation criterion\",\n \"level\": \"A2\",\n \"type\": \"positive\"\n }\n]\n```\n\n**Medical Specialty**\n[e.g., Cardiology, Pulmonology, Oncology, etc.]\n\n**Question Type**\n[e.g., Diagnosis, Treatment, Risk Assessment, etc.]\n\n**Additional Context**\n[Any relevant clinical context or references]\n```\n\n#### Option 2: Submit a Pull Request\n1. Fork this repository\n2. Add your data to `data/input/community_contributions.xlsx` following the existing format:\n - **question**: Your medical question\n - **rubrics**: JSON array of evaluation criteria\n - **specialty**: Medical specialty area\n - **difficulty**: Easy/Medium/Hard\n - **source**: Your name/institution (optional)\n\n3. Ensure your contribution follows these guidelines:\n - **Clinical Relevance**: Questions should reflect real-world medical scenarios\n - **Evidence-Based**: Evaluation criteria should be based on established medical guidelines\n - **Clear Scoring**: Use our scoring system appropriately:\n - **Positive Points**: A1 (5 pts), A2 (3 pts), A3 (1 pt)\n - **Negative Points**: S1 (-1 pt), S2 (-2 pts), S3 (-3 pts), S4 (-4 pts)\n\n### 📝 Contribution Guidelines\n\n#### Evaluation Criteria Design\n- **A1 (Critical, 5 points)**: Essential medical knowledge that could impact patient safety\n- **A2 (Important, 3 points)**: Significant clinical considerations\n- **A3 (Helpful, 1 point)**: Additional relevant information\n- **S1 (Minor Error, -1 point)**: Small inaccuracies that don't affect core treatment\n- **S2 (Moderate Error, -2 points)**: Incorrect information that could mislead\n- **S3 (Major Error, -3 points)**: Serious medical errors\n- **S4 (Critical Error, -4 points)**: Dangerous misinformation that could harm patients\n\n#### Quality Standards\n- Questions should be **clinically realistic** and based on actual medical scenarios\n- Evaluation criteria must be **evidence-based** and reference current medical guidelines\n- Each question should have **3-8 evaluation points** for balanced assessment\n- Include both **positive criteria** (what should be mentioned) and **negative criteria** (what should be avoided)\n\n### 🎯 Priority Areas\nWe especially welcome contributions in:\n- **Rare Diseases**: Uncommon conditions that challenge AI systems\n- **Multidisciplinary Cases**: Complex cases requiring multiple specialties\n- **Cultural Considerations**: Cases reflecting diverse patient populations\n- **Emergency Medicine**: Time-critical decision-making scenarios\n- **Pediatric Medicine**: Age-specific medical considerations\n\n### 📊 Data Format Example\n```json\n{\n \"question\": \"A 65-year-old patient presents with chest pain and shortness of breath. How would you approach the initial evaluation?\",\n \"rubrics\": [\n {\n \"id\": 1,\n \"claim\": \"Should obtain detailed history including pain characteristics, timing, and associated symptoms\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"Must perform ECG and chest X-ray as initial diagnostic tests\",\n \"level\": \"A1\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Should not delay evaluation with unnecessary tests in acute presentation\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n ],\n \"specialty\": \"Emergency Medicine\",\n \"difficulty\": \"Medium\"\n}\n```\n\n### 🔄 Review Process\n1. **Community Review**: Contributions will be reviewed by medical professionals\n2. **Quality Check**: Ensure clinical accuracy and appropriate difficulty\n3. **Integration**: Approved contributions will be merged into the main dataset\n4. **Attribution**: Contributors will be acknowledged in our contributor list\n\n### 📞 Contact\nFor questions about contributing, please:\n- Open an issue with the `question` label\n- Join our discussion in the GitHub Discussions section\n\nThank you for helping build a better benchmark for medical AI evaluation! 🙏\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n🤗 <a href=\"https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview\" target=\"_blank\">HuggingFace Dataset</a> ｜\n📄 <a href=\"https://arxiv.org/abs/2510.13734\" target=\"_blank\">arXiv</a> ｜\n🌐 <a href=\"README_CN.md\">中文</a> ｜\n"}, "915b76a5b626ac4e352cc31268ab8b212122830d": {"t": 1785766588.3536913, "n": 36638, "h": "09a982f74272c152", "c": "# Medical AI Evaluation Framework: Clinical Benchmark Dataset and Automated Assessment Pipeline\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n - [Installation](#installation)\n - [Basic Usage](#basic-usage)\n- [System Architecture](#system-architecture)\n- [Data Format](#data-format)\n - [Input Format](#input-format)\n - [Output Format](#output-format)\n- [Configuration](#configuration)\n - [Basic Configuration](#basic-configuration)\n - [Advanced Configuration](#advanced-configuration)\n- [Usage Guide](#usage-guide)\n - [Command Line Interface](#command-line-interface)\n - [Python API](#python-api)\n - [Examples](#examples)\n- [Evaluation Framework](#evaluation-framework)\n - [Scoring System](#scoring-system)\n - [Evaluation Metrics](#evaluation-metrics)\n- [Dataset](#dataset)\n - [Thoracic Surgery Dataset](#thoracic-surgery-dataset)\n - [Data Quality](#data-quality)\n- [Advanced Features](#advanced-features)\n - [Parallel Processing](#parallel-processing)\n - [Voting Strategies](#voting-strategies)\n - [Custom Models](#custom-models)\n- [Development](#development)\n - [Technical Documentation](#technical-documentation)\n - [API Reference](#api-reference)\n - [Troubleshooting](#troubleshooting)\n- [Community](#community)\n - [Contributing](#contributing)\n - [Citation](#citation)\n- [License](#license)\n\n## Overview\n\nThis Medical AI Evaluation Framework provides a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this framework includes both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.\n\nThe framework addresses the critical need for standardized evaluation of AI clinical decision-making by providing:\n- **Clinically Grounded Assessment**: Evaluation criteria based on real medical guidelines and expert knowledge\n- **Automated Pipeline**: Streamlined processing from raw responses to detailed performance metrics\n- **Multi-Model Support**: Simultaneous evaluation of multiple AI models with comparative analysis\n- **Scalable Architecture**: Efficient processing of large datasets with parallel execution capabilities\n\n## Key Features\n\n- **🏥 Medical-Specific Evaluation**: Specialized rubrics for clinical scenarios with positive/negative scoring\n- **🔄 Parallel Processing**: Simultaneous execution of Met/Not Met review and irrelevant content detection\n- **📊 Comprehensive Analytics**: Detailed statistical analysis with visualization reports\n- **🎯 Multi-Model Assessment**: Support for evaluating multiple AI models simultaneously\n- **⚙️ Flexible Configuration**: Customizable models, voting strategies, and evaluation parameters\n- **📈 Rich Visualization**: Automated generation of performance charts and comparative analysis\n- **🔧 Modular Design**: Independent modules for different evaluation stages\n- **📋 Standardized Output**: Consistent Excel-based reporting with detailed metrics\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd medical-ai-bench-eval\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Basic Usage\n\n```bash\n# Run complete evaluation pipeline\npython medical_evaluation_pipeline.py input_data.xlsx\n\n# With custom output file\npython medical_evaluation_pipeline.py input_data.xlsx -o results/evaluation_results.xlsx\n\n# Enable verbose logging\npython medical_evaluation_pipeline.py input_data.xlsx -v\n```\n\n## System Architecture\n\nThe system processes medical AI responses through a sophisticated pipeline that includes:\n\n1. **Input Processing**: Reads Excel files containing medical questions, evaluation rubrics, and AI model responses\n2. **Parallel Evaluation**: Simultaneously executes Met/Not Met review and irrelevant content detection\n3. **Intelligent Scoring**: Calculates comprehensive scores based on clinical evaluation criteria\n4. **Analysis & Reporting**: Generates detailed statistical reports and visualizations\n\n## System Architecture\n\n```\nInput Excel File\n↓\n┌─────────────────────────────────────────┐\n│ Parallel Processing Phase │\n├─────────────────┬─────────────────────┤\n│ Step 1: Met Review │ Step 2: Irrelevant Content │\n│ - Multi-model Review │ - Content Extraction │\n│ - Voting Decision │ - Level Assessment │\n│ - Result Summary │ - Voting Classification │\n└─────────────────┴─────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Result Merging │\n│ - Intelligent merging of parallel results │\n│ - Data integrity check │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 3: Score Calculation │\n│ - Multi-dimensional score statistics │\n│ - Irrelevant content deduction │\n│ - Normalization processing │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 4: Data Analysis & Visualization │\n│ - Statistical analysis reports │\n│ - Diverse charts │\n│ - CSV data export │\n└─────────────────────────────────────────┘\n↓\nOutput Excel File + Analysis Report\n```\n\n## Data Format\n\n### Input Format\n\nThe system accepts Excel files with the following structure:\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n#### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n### Output Format\n\nThe system generates multiple output files:\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n#### Scoring Metrics\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Configuration\n\n### Basic Configuration\n\n### Environment Requirements\n- **Python**: 3.12+\n- **Operating System**: Windows, macOS, Linux\n- **Core Dependencies**: pandas, numpy, matplotlib, seaborn, asyncio, langchain, openpyxl, xlsxwriter\n\n### Installation Steps\n\n1. **Clone the repository**:\n```bash\ngit clone <repository-url>\ncd medical-ai-bench-eval\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Configure API Keys** (if using external AI model services):\n\n ### API Configuration Guide\n\n #### Environment Variables Setup\n\n **Linux/Mac:**\n ```bash\n # OpenAI\n export OPENAI_API_KEY=\"sk-your-actual-key\"\n export OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n\n # Claude (Anthropic)\n export ANTHROPIC_API_KEY=\"sk-ant-your-actual-key\"\n export ANTHROPIC_BASE_URL=\"https://api.anthropic.com\"\n\n # Google Gemini\n export GOOGLE_API_KEY=\"your-google-api-key\"\n export GOOGLE_BASE_URL=\"https://generativelanguage.googleapis.com/v1beta\"\n\n # Moonshot Kimi\n export MOONSHOT_API_KEY=\"sk-your-moonshot-key\"\n export MOONSHOT_BASE_URL=\"https://api.moonshot.cn/v1\"\n\n # Alibaba Qwen\n export DASHSCOPE_API_KEY=\"your-dashscope-key\"\n export DASHSCOPE_BASE_URL=\"https://dashscope.aliyuncs.com/api/v1\"\n\n # Baichuan\n export BAICHUAN_API_KEY=\"your-baichuan-key\"\n export BAICHUAN_BASE_URL=\"https://api.baichuan-ai.com/v1\"\n\n # DeepSeek\n export DEEPSEEK_API_KEY=\"sk-your-deepseek-key\"\n export DEEPSEEK_BASE_URL=\"https://api.deepseek.com/v1\"\n\n # Zhipu ChatGLM\n export ZHIPU_API_KEY=\"your-zhipu-key\"\n export ZHIPU_BASE_URL=\"https://open.bigmodel.cn/api/paas/v4\"\n ```\n\n **Windows PowerShell:**\n ```powershell\n $Env:OPENAI_API_KEY=\"sk-your-actual-key\"\n $Env:OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n ```\n\n #### Model Configuration\n\n **⚠️ IMPORTANT: OpenAI-Compatible API Only**\n \n This system currently supports **ONLY OpenAI-compatible API interfaces**. All models must provide OpenAI-compatible endpoints, regardless of the actual provider.\n\n **🌐 Third-Party Multi-Model Platforms (Recommended)**\n \n For models that don't natively support OpenAI-compatible APIs (like Claude, Gemini), you can use third-party platforms that provide unified OpenAI-compatible access:\n \n - **[OpenRouter](https://openrouter.ai/)** - Access Claude, Gemini, GPT, and many other models via OpenAI-compatible API\n - **[Together AI](https://together.ai/)** - Various open-source and commercial models\n - **[Anyscale](https://anyscale.com/)** - Multiple LLMs via OpenAI-compatible endpoints\n\n Edit `config_model.yaml` to configure your preferred models:\n\n ```yaml\n models:\n m1: # OpenAI GPT-4 (Native)\n api_key_env: \"OPENAI_API_KEY\"\n api_base_env: \"OPENAI_BASE_URL\"\n model: \"gpt-4-turbo-preview\"\n temperature: 0.2\n max_tokens: 4000\n\n m2: # Kimi (OpenAI-Compatible)\n api_key_env: \"MOONSHOT_API_KEY\"\n api_base_env: \"MOONSHOT_BASE_URL\"\n model: \"kimi-latest\"\n temperature: 0.2\n max_tokens: 4000\n\n m3: # Qwen (OpenAI-Compatible)\n api_key_env: \"DASHSCOPE_API_KEY\"\n api_base_env: \"DASHSCOPE_BASE_URL\"\n model: \"qwen-max\"\n temperature: 0.1\n max_tokens: 4000\n\n m4: # DeepSeek (OpenAI-Compatible)\n api_key_env: \"DEEPSEEK_API_KEY\"\n api_base_env: \"DEEPSEEK_BASE_URL\"\n model: \"deepseek-chat\"\n temperature: 0.2\n max_tokens: 4000\n\n m5: # Local Model (via OpenAI-Compatible Server)\n api_key_env: \"LOCAL_API_KEY\"\n api_base_env: \"LOCAL_BASE_URL\"\n model: \"llama3-medical\"\n temperature: 0.2\n max_tokens: 4000\n\n # OpenRouter Examples (uncomment to use)\n # m6: # Claude via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"anthropic/claude-3-5-sonnet-20241022\"\n # temperature: 0.2\n # max_tokens: 4000\n\n # m7: # Gemini via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"google/gemini-pro-1.5\"\n # temperature: 0.1\n # max_tokens: 4000\n ```\n\n #### Supported vs Unsupported Providers\n\n | Provider | Status | API Type | Notes |\n |----------|--------|----------|-------|\n | **OpenAI** | ✅ Supported | Native OpenAI | Direct support |\n | **OpenRouter** | ✅ Supported | Multi-Model Platform | Access Claude, Gemini, GPT via OpenAI-compatible API |\n | **Together AI** | ✅ Supported | Multi-Model Platform | Various models via OpenAI-compatible API |\n | **Anyscale** | ✅ Supported | Multi-Model Platform | Multiple LLMs via OpenAI-compatible API |\n | **Kimi/Moonshot** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Qwen/DashScope** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **DeepSeek** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Baichuan** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Zhipu** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Claude/Anthropic** | ⚠️ Via OpenRouter | Native Anthropic API | Use OpenRouter for OpenAI-compatible access |\n | **Gemini/Google** | ⚠️ Via OpenRouter | Native Google API | Use OpenRouter for OpenAI-compatible access |\n\n #### Medical Model Recommendations (OpenAI-Compatible Only)\n\n | Use Case | Recommended Models | Temperature | Notes |\n |----------|-------------------|-------------|--------|\n | **Diagnostic Tasks** | gpt-4-turbo, deepseek-chat | 0.1-0.2 | High accuracy |\n | **Chinese Medical** | qwen-max, baichuan-medical | 0.2-0.3 | Better CJK support |\n | **Cost-effective** | deepseek-chat, kimi | 0.2-0.3 | Good balance |\n | **Real-time** | kimi, deepseek-chat | 0.1-0.2 | Fast response |\n | **Privacy-focused** | local-llama3-medical | 0.2-0.3 | On-premise via OpenAI-compatible server |\n\n #### Quick Model Setup Commands\n\n **For GPT-4 testing:**\n ```bash\n export OPENAI_API_KEY=\"your-key-here\"\n # Run with GPT-4\n python medical_evaluation_pipeline.py input.xlsx --judge-models m1\n ```\n\n **For Chinese medical models:**\n ```bash\n export DASHSCOPE_API_KEY=\"your-key-here\"\n # Run with Qwen\n python medical_evaluation_pipeline.py input.xlsx --judge-models m4\n ```\n\n **For mixed model evaluation:**\n ```bash\n export OPENAI_API_KEY=\"your-key\"\n export DASHSCOPE_API_KEY=\"your-key\"\n # Use GPT-4 and Qwen for evaluation\n python medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m4 \\\n --extract-model m5\n\n### Basic Usage\n\n1. **Prepare input data**:\n - Excel file format\n - Contains columns for questions, scoring criteria (JSON format), model responses, etc.\n - Default column names can be configured in `config.yaml`\n\n2. **Run the complete evaluation pipeline**:\n```bash\npython medical_evaluation_pipeline.py input.xlsx -o output.xlsx\n```\n\n3. **View results**:\n - Output Excel file contains detailed review results and scores\n - Data analysis reports are saved in the `data/output/analysis/` directory\n\n## ⚙️ System Configuration\n\n### Configuration File (config.yaml)\n\n```yaml\n# File path configuration\nfiles:\n input_file: \"data/medical_evaluation_result.xlsx\"\n output_file: \"data/output/processed_result.xlsx\"\n\n# Process control\npipeline:\n disable_met_nomet: false # Whether to disable Met review\n disable_irrelevant_extraction: false # Whether to disable irrelevant content detection\n exclude_irrelevant_scoring: false # Whether to exclude irrelevant content from scoring\n disable_data_analysis: false # Whether to disable data analysis\n analysis_config_file: \"config_visualization.yaml\"\n\n# Model configuration\nmodels:\n judge_models: [\"m1\", \"m2\", \"m3\"] # Met review models\n extract_model: \"m5\" # Irrelevant content extraction model\n grade_models: [\"m1\", \"m2\", \"m3\"] # Irrelevant content grading models\n voting_strategy: \"conservative\" # Voting strategy\n\n# Column name mapping\ncolumns:\n question_col: \"question\"\n rubric_col: \"final_merged_json\"\n answer_columns:\n - \"gpt_5_answer\"\n - \"gemini_2_5_pro_answer\"\n - \"claude_opus_4_answer\"\n\n# Scoring system\npoint:\n A1: 5\n A2: 3\n A3: 1\n S1: -1\n S2: -2\n S3: -3\n S4: -4\n```\n\n### Command Line Arguments\n\n```bash\n# Process control\n--disable-met-nomet # Skip Met review step\n--disable-irrelevant # Skip irrelevant content detection\n--exclude-irrelevant-scoring # Exclude irrelevant content deduction from scoring\n--disable-data-analysis # Skip data analysis\n\n# Model configuration\n--judge-models m1 m2 # Specify Met review models\n--extract-model m3 # Specify irrelevant content extraction model\n--grade-models m1 m2 m4 # Specify irrelevant content grading models\n--voting-strategy majority # Voting strategy: conservative/majority/average\n\n# Data column configuration\n--question-col \"Question\" # Question column name\n--rubric-col \"Evaluation Points\" # Evaluation points column name\n--answer-columns \"Response1\" \"Response2\" \"Response3\" # Model response column names\n```\n\n## 📊 Thoracic Surgery Evaluation Dataset\n\n### Dataset Overview\n\nWe provide a specialized **Thoracic Surgery Evaluation Dataset** (`data/input/GAPS-NSCLC-preview.xlsx`) containing 92 carefully curated clinical cases for evaluating AI models' performance in thoracic surgery scenarios. This dataset focuses on complex clinical decision-making in chest surgery, particularly for non-small cell lung cancer (NSCLC) staging and treatment planning.\n\n### Dataset Structure\n\n| Column Name | Description | Data Type |\n|-------------|-------------|-----------|\n| **question** | Clinical questions covering thoracic surgery scenarios | Text |\n| **分类** (Category) | Medical specialty classification | Text |\n| **rubrics** | Evaluation criteria in JSON format with scoring levels | JSON Array |\n| **gpt_5_answer** | GPT-4 model responses to clinical questions | Text |\n| **gemini_2_5_pro_answer** | Gemini 2.5 Pro model responses | Text |\n| **claude_opus_4_answer** | Claude Opus model responses | Text |\n\n### Clinical Focus Areas\n\nThe dataset covers critical aspects of thoracic surgery, including:\n\n- **Pre-operative Evaluation**: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)\n- **Diagnostic Procedures**: EBUS-TBNA, mediastinoscopy, PET-CT interpretation\n- **Staging Assessment**: TNM staging, mediastinal lymph node evaluation\n- **Treatment Planning**: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions\n- **Risk Assessment**: Pulmonary function evaluation, cardiac risk stratification\n- **Molecular Diagnostics**: EGFR, ALK, PD-L1 testing strategies\n\n### Evaluation Rubrics Format\n\nEach question includes detailed evaluation criteria in JSON format:\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Should perform enhanced chest CT covering thoracic inlet to upper abdomen\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"PET-CT is essential for accurate N-staging and excluding distant metastases\",\n \"level\": \"A2\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Incorrectly generalizing upper sulcus tumor MRI requirements to all IIB/IIIA patients\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n]\n```\n\n**Scoring Levels**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Sample Clinical Question\n\n**Question**: \"For NSCLC patients with IIB-IIIA staging, what comprehensive pre-treatment evaluation should be performed?\"\n\n**Key Evaluation Points**:\n- Pathological diagnosis and molecular characterization\n- Accurate clinical staging (T, N, M assessment)\n- Invasive mediastinal staging for N2 evaluation\n- Pulmonary and cardiac function assessment\n- Multidisciplinary team (MDT) treatment planning\n\n### Usage with GAPS Framework\n\nThis dataset is designed to work seamlessly with the GAPS evaluation pipeline:\n\n```bash\n# Evaluate thoracic surgery dataset\npython medical_evaluation_pipeline.py data/input/GAPS-NSCLC-preview.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -o results/thoracic_surgery_evaluation.xlsx\n```\n\n### Clinical Validation\n\nThe dataset has been validated by:\n- **Board-certified thoracic surgeons**\n- **Pulmonary oncologists** \n- **Medical imaging specialists**\n- **Pathologists specializing in lung cancer**\n\nAll evaluation criteria are based on current clinical guidelines including:\n- NCCN Guidelines for Non-Small Cell Lung Cancer\n- ESTS Guidelines for Intraoperative Lymph Node Staging\n- IASLC Staging Manual in Thoracic Oncology\n\n### Research Applications\n\nThis dataset enables research in:\n- **AI Clinical Decision Support**: Evaluating AI models' ability to provide accurate clinical recommendations\n- **Medical Education**: Training and assessment of clinical reasoning skills\n- **Quality Assurance**: Benchmarking AI systems against established clinical standards\n- **Comparative Analysis**: Cross-model performance evaluation in specialized medical domains\n\n### Data Quality Metrics\n\n- **Total Cases**: 92 clinical scenarios\n- **Completeness**: 100% data coverage across all columns\n- **Clinical Diversity**: Covers full spectrum of IIB-IIIA NSCLC presentations\n- **Expert Validation**: All cases reviewed by multidisciplinary clinical team\n\n---\n\n## Input Data Format\n\n### Excel File Structure\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n## Output Results\n\n### Main Output Files\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n### Scoring Metrics Description\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Advanced Features\n\n### 1. Parallel Processing Optimization\n\nThe system supports parallel execution of Step 1 (Met Review) and Step 2 (Irrelevant Content Detection):\n\n```bash\n# Enable all steps (auto parallel)\npython medical_evaluation_pipeline.py input.xlsx\n\n# Enable only Met Review\npython medical_evaluation_pipeline.py input.xlsx --disable-irrelevant\n\n# Enable only Irrelevant Content Detection\npython medical_evaluation_pipeline.py input.xlsx --disable-met-nomet\n```\n\n### 2. Voting Strategy\n\nSupports three voting strategies for processing multi-model results:\n- **conservative**: Conservative strategy, prioritizes more severe ratings\n- **majority**: Majority voting, selects the result with the most votes\n- **average**: Average strategy, calculates based on numerical average\n\n```bash\npython medical_evaluation_pipeline.py input.xlsx --voting-strategy conservative\n```\n\n### 3. Custom Model Configuration\n\n```bash\n# Use specific review model combinations\npython medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m2 \\\n --extract-model m5 \\\n --grade-models m1 m3 m4\n```\n\n### 4. Data Analysis Customization\n\nCreate `config_visualization.yaml` for custom analysis configuration:\n\n```yaml\nanalysis:\n input_file: \"data/processed_result.xlsx\"\n columns:\n model_scores:\n GPT-5: \"gpt_5_answer_judged_json_scores\"\n Gemini-2.5-Pro: \"gemini_2_5_pro_answer_judged_json_scores\"\n Claude-Opus-4: \"claude_opus_4_answer_judged_json_scores\"\n\nvisualization:\n charts:\n bar_overall: true\n boxplot_distribution: true\n heatmap_category: true\n radar_chart: true\n style:\n figure_size: [24, 16]\n dpi: 300\n```\n\n## 🔧 Independent Module Usage\n\n### 1. Run Met Review Separately\n\n```python\nimport asyncio\nfrom clinical_answer_judge import MultiModelMergedJsonJudge\n\nasync def run_judge():\n judge = MultiModelMergedJsonJudge(\n judge_models=['m1', 'm2', 'm3'],\n question_col='question',\n rubric_col='rubrics'\n )\n result = await judge.process_excel_file('input.xlsx', 'output.xlsx')\n return result\n\n# Run the async function\nresult = asyncio.run(run_judge())\n```\n\n### 2. Run Score Calculation Separately\n\n```python\nfrom clinical_scoring_calculator import process_excel_data\n\nresult = process_excel_data(\n input_file='judged_data.xlsx',\n output_file='scored_data.xlsx',\n model_columns=['gpt_5_answer_judged_json']\n)\n```\n\n### 3. Run Data Analysis Separately\n\n```python\nfrom enhanced_analyzer import EnhancedMedicalAnalyzer\n\nanalyzer = EnhancedMedicalAnalyzer('config_visualization.yaml')\nanalyzer.run_analysis()\n```\n\n## Usage Examples\n\n### Example 1: Complete Process Handling\n\n```bash\n# Process complete medical evaluation data\npython medical_evaluation_pipeline.py medical_data.xlsx \\\n -o results/final_report.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -v\n```\n\n### Example 2: Quick Testing\n\n```bash\n# Process only first 10 rows of data\npython medical_evaluation_pipeline.py test_data.xlsx \\\n --max_rows 10 \\\n --disable-data-analysis\n```\n\n### Example 3: Custom Column Names\n\n```bash\n# Use Chinese column names\npython medical_evaluation_pipeline.py data.xlsx \\\n --question-col \"Clinical Question\" \\\n --rubric-col \"Scoring Criteria\" \\\n --answer-columns \"GPT Response\" \"Claude Response\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **File Not Found Error**\n - Solution: Check if the input file path is correct\n\n2. **Column Name Mismatch**\n - Solution: Use `--question-col`, `--rubric-col` and other parameters to specify correct column names\n\n3. **Insufficient Memory**\n - Solution: Use `--max_rows` parameter to limit the number of rows processed\n\n4. **API Call Failure**\n - Solution: Check model configuration and network connection\n\n### Debug Mode\n\n```bash\n# Enable verbose logging to view processing details\npython medical_evaluation_pipeline.py input.xlsx -v\n\n# Keep temporary files for debugging\n# Modify the cleanup() call in the code\n```\n\n## Technical Documentation\n\n### Core Modules\n\n- `medical_evaluation_pipeline.py` - Main pipeline controller\n- `clinical_answer_judge.py` - Met/Not Met review module\n- `clinical_scoring_calculator.py` - Score calculation module\n- `enhanced_analyzer.py` - Data analysis and visualization module\n- `general_negative_points.py` - Irrelevant content processing module\n\n### File Structure\n\n```\nMedicalAiBenchEval/\n├── clinical_answer_judge.py # Multi-model review processor\n├── clinical_scoring_calculator.py # Score calculator\n├── medical_evaluation_pipeline.py # Main pipeline controller\n├── enhanced_analyzer.py # Data analyzer\n├── judge_engine.py # Review engine\n├── irrelevant_content_grading.py # Irrelevant content scoring\n├── extract_irrelevant_content.py # Irrelevant content extraction\n├── general_negative_points.py # Negative point processing\n├── config.yaml # Main configuration file\n├── config_model.yaml # Model configuration file\n├── config_visualization.yaml # Visualization configuration\n├── requirements.txt # Dependency list\n└── data/ # Data directory\n ├── input/ # Input files\n │ ├── GAPS-NSCLC-preview.xlsx # Thoracic surgery evaluation dataset (92 cases)\n │ └── community_contributions.xlsx # Community contributed cases\n └── output/ # Output files\n └── analysis/ # Analysis reports\n```\n\n### Data Flow\n\n1. **Input Phase**: Read Excel file, validate data format\n2. **Parallel Processing**: Simultaneously execute Met review and irrelevant content detection\n3. **Result Merging**: Intelligently merge outputs from both parallel steps\n4. **Score Calculation**: Calculate multi-dimensional scores based on merged results\n5. **Analysis Output**: Generate statistical reports and visualization charts\n\n### Performance Optimization\n\n- Asynchronous parallel processing improves efficiency\n- Batch processing reduces API call frequency\n- Intelligent retry mechanism handles network exceptions\n- Memory optimization handles large files\n\n## Contribution Guidelines\n\nWelcome to submit Issues and Pull Requests to improve system functionality.\n\n### Development Environment Setup\n\n```bash\ngit clone <repository>\ncd medical-ai-bench-eval\npip install -r requirements.txt\n```\n\n### Code Standards\n\n- Follow PEP 8 coding standards\n- Add detailed docstrings\n- Include unit tests\n- Update README documentation\n\n## Citation\n\nIf you use this system in your research, please cite our paper:\n\n**English:**\n> Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Jun Wang, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan Yang (2025). GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians. *arXiv preprint* arXiv:2510.13734. https://arxiv.org/abs/2510.13734\n\n```bibtex\n@article{chen2025gaps,\n title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},\n author={Chen, Xiuyuan and Sun, Tao and Su, Dexin and Yu, Ailing and Liu, Junwei and Chen, Zhe and Jin, Gangzeng and Wang, Xin and Liu, Jingnan and Xiao, Hansong and Zhou, Hualei and Tao, Dongjie and Guo, Chunxiao and Yang, Minghui and Xia, Yuan and Zhao, Jing and Fan, Qianrui and Wang, Yanyun and Zhen, Shuai and Chen, Kezhong and Wang, Jun and Sun, Zewen and Zhao, Heng and Guan, Tian and Wang, Shaodong and Chang, Geyun and Deng, Jiaming and Chen, Hongchengcheng and Feng, Kexin and Li, Ruzhen and Geng, Jiayi and Zhao, Changtai and Wang, Jun and Lin, Guihu and Li, Peihao and Liu, Liqi and Wei, Peng and Wang, Jian and Gu, Jinjie and Wang, Ping and Yang, Fan},\n journal={arXiv preprint arXiv:2510.13734},\n year={2025},\n url={https://arxiv.org/abs/2510.13734}\n}\n```\n\n## 🤝 Community Contribution / 共创呼吁\n\nWe welcome contributions from the medical AI community to expand and improve our evaluation dataset! Your expertise can help build a more comprehensive benchmark for medical AI systems.\n\n### 📋 How to Contribute\n\n#### Option 1: Submit an Issue\nCreate a new issue with the label `data-contribution` using the following template:\n\n```markdown\n**Medical Question**\n[Provide your clinical question here]\n\n**Evaluation Rubrics (JSON Format)**\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Your evaluation criterion description\",\n \"level\": \"A1|A2|A3|S1|S2|S3|S4\",\n \"type\": \"positive|negative\"\n },\n {\n \"id\": 2,\n \"claim\": \"Another evaluation criterion\",\n \"level\": \"A2\",\n \"type\": \"positive\"\n }\n]\n```\n\n**Medical Specialty**\n[e.g., Cardiology, Pulmonology, Oncology, etc.]\n\n**Question Type**\n[e.g., Diagnosis, Treatment, Risk Assessment, etc.]\n\n**Additional Context**\n[Any relevant clinical context or references]\n```\n\n#### Option 2: Submit a Pull Request\n1. Fork this repository\n2. Add your data to `data/input/community_contributions.xlsx` following the existing format:\n - **question**: Your medical question\n - **rubrics**: JSON array of evaluation criteria\n - **specialty**: Medical specialty area\n - **difficulty**: Easy/Medium/Hard\n - **source**: Your name/institution (optional)\n\n3. Ensure your contribution follows these guidelines:\n - **Clinical Relevance**: Questions should reflect real-world medical scenarios\n - **Evidence-Based**: Evaluation criteria should be based on established medical guidelines\n - **Clear Scoring**: Use our scoring system appropriately:\n - **Positive Points**: A1 (5 pts), A2 (3 pts), A3 (1 pt)\n - **Negative Points**: S1 (-1 pt), S2 (-2 pts), S3 (-3 pts), S4 (-4 pts)\n\n### 📝 Contribution Guidelines\n\n#### Evaluation Criteria Design\n- **A1 (Critical, 5 points)**: Essential medical knowledge that could impact patient safety\n- **A2 (Important, 3 points)**: Significant clinical considerations\n- **A3 (Helpful, 1 point)**: Additional relevant information\n- **S1 (Minor Error, -1 point)**: Small inaccuracies that don't affect core treatment\n- **S2 (Moderate Error, -2 points)**: Incorrect information that could mislead\n- **S3 (Major Error, -3 points)**: Serious medical errors\n- **S4 (Critical Error, -4 points)**: Dangerous misinformation that could harm patients\n\n#### Quality Standards\n- Questions should be **clinically realistic** and based on actual medical scenarios\n- Evaluation criteria must be **evidence-based** and reference current medical guidelines\n- Each question should have **3-8 evaluation points** for balanced assessment\n- Include both **positive criteria** (what should be mentioned) and **negative criteria** (what should be avoided)\n\n### 🎯 Priority Areas\nWe especially welcome contributions in:\n- **Rare Diseases**: Uncommon conditions that challenge AI systems\n- **Multidisciplinary Cases**: Complex cases requiring multiple specialties\n- **Cultural Considerations**: Cases reflecting diverse patient populations\n- **Emergency Medicine**: Time-critical decision-making scenarios\n- **Pediatric Medicine**: Age-specific medical considerations\n\n### 📊 Data Format Example\n```json\n{\n \"question\": \"A 65-year-old patient presents with chest pain and shortness of breath. How would you approach the initial evaluation?\",\n \"rubrics\": [\n {\n \"id\": 1,\n \"claim\": \"Should obtain detailed history including pain characteristics, timing, and associated symptoms\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"Must perform ECG and chest X-ray as initial diagnostic tests\",\n \"level\": \"A1\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Should not delay evaluation with unnecessary tests in acute presentation\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n ],\n \"specialty\": \"Emergency Medicine\",\n \"difficulty\": \"Medium\"\n}\n```\n\n### 🔄 Review Process\n1. **Community Review**: Contributions will be reviewed by medical professionals\n2. **Quality Check**: Ensure clinical accuracy and appropriate difficulty\n3. **Integration**: Approved contributions will be merged into the main dataset\n4. **Attribution**: Contributors will be acknowledged in our contributor list\n\n### 📞 Contact\nFor questions about contributing, please:\n- Open an issue with the `question` label\n- Join our discussion in the GitHub Discussions section\n\nThank you for helping build a better benchmark for medical AI evaluation! 🙏\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n🤗 <a href=\"https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview\" target=\"_blank\">HuggingFace Dataset</a> ｜\n📄 <a href=\"https://arxiv.org/abs/2510.13734\" target=\"_blank\">arXiv</a> ｜\n🌐 <a href=\"README_CN.md\">中文</a> ｜\n"}, "1130d19e319df59549ad7c799ad5d34e618ffa62": {"t": 1785766590.0352232, "n": 36638, "h": "3d5d410059f5f71c", "c": "# Medical AI Evaluation Framework: Clinical Benchmark Dataset and Automated Assessment Pipeline\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n - [Installation](#installation)\n - [Basic Usage](#basic-usage)\n- [System Architecture](#system-architecture)\n- [Data Format](#data-format)\n - [Input Format](#input-format)\n - [Output Format](#output-format)\n- [Configuration](#configuration)\n - [Basic Configuration](#basic-configuration)\n - [Advanced Configuration](#advanced-configuration)\n- [Usage Guide](#usage-guide)\n - [Command Line Interface](#command-line-interface)\n - [Python API](#python-api)\n - [Examples](#examples)\n- [Evaluation Framework](#evaluation-framework)\n - [Scoring System](#scoring-system)\n - [Evaluation Metrics](#evaluation-metrics)\n- [Dataset](#dataset)\n - [Thoracic Surgery Dataset](#thoracic-surgery-dataset)\n - [Data Quality](#data-quality)\n- [Advanced Features](#advanced-features)\n - [Parallel Processing](#parallel-processing)\n - [Voting Strategies](#voting-strategies)\n - [Custom Models](#custom-models)\n- [Development](#development)\n - [Technical Documentation](#technical-documentation)\n - [API Reference](#api-reference)\n - [Troubleshooting](#troubleshooting)\n- [Community](#community)\n - [Contributing](#contributing)\n - [Citation](#citation)\n- [License](#license)\n\n## Overview\n\nThis Medical AI Evaluation Framework provides a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this framework includes both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.\n\nThe framework addresses the critical need for standardized evaluation of AI clinical decision-making by providing:\n- **Clinically Grounded Assessment**: Evaluation criteria based on real medical guidelines and expert knowledge\n- **Automated Pipeline**: Streamlined processing from raw responses to detailed performance metrics\n- **Multi-Model Support**: Simultaneous evaluation of multiple AI models with comparative analysis\n- **Scalable Architecture**: Efficient processing of large datasets with parallel execution capabilities\n\n## Key Features\n\n- **🏥 Medical-Specific Evaluation**: Specialized rubrics for clinical scenarios with positive/negative scoring\n- **🔄 Parallel Processing**: Simultaneous execution of Met/Not Met review and irrelevant content detection\n- **📊 Comprehensive Analytics**: Detailed statistical analysis with visualization reports\n- **🎯 Multi-Model Assessment**: Support for evaluating multiple AI models simultaneously\n- **⚙️ Flexible Configuration**: Customizable models, voting strategies, and evaluation parameters\n- **📈 Rich Visualization**: Automated generation of performance charts and comparative analysis\n- **🔧 Modular Design**: Independent modules for different evaluation stages\n- **📋 Standardized Output**: Consistent Excel-based reporting with detailed metrics\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd medical-ai-bench-eval\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Basic Usage\n\n```bash\n# Run complete evaluation pipeline\npython medical_evaluation_pipeline.py input_data.xlsx\n\n# With custom output file\npython medical_evaluation_pipeline.py input_data.xlsx -o results/evaluation_results.xlsx\n\n# Enable verbose logging\npython medical_evaluation_pipeline.py input_data.xlsx -v\n```\n\n## System Architecture\n\nThe system processes medical AI responses through a sophisticated pipeline that includes:\n\n1. **Input Processing**: Reads Excel files containing medical questions, evaluation rubrics, and AI model responses\n2. **Parallel Evaluation**: Simultaneously executes Met/Not Met review and irrelevant content detection\n3. **Intelligent Scoring**: Calculates comprehensive scores based on clinical evaluation criteria\n4. **Analysis & Reporting**: Generates detailed statistical reports and visualizations\n\n## System Architecture\n\n```\nInput Excel File\n↓\n┌─────────────────────────────────────────┐\n│ Parallel Processing Phase │\n├─────────────────┬─────────────────────┤\n│ Step 1: Met Review │ Step 2: Irrelevant Content │\n│ - Multi-model Review │ - Content Extraction │\n│ - Voting Decision │ - Level Assessment │\n│ - Result Summary │ - Voting Classification │\n└─────────────────┴─────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Result Merging │\n│ - Intelligent merging of parallel results │\n│ - Data integrity check │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 3: Score Calculation │\n│ - Multi-dimensional score statistics │\n│ - Irrelevant content deduction │\n│ - Normalization processing │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 4: Data Analysis & Visualization │\n│ - Statistical analysis reports │\n│ - Diverse charts │\n│ - CSV data export │\n└─────────────────────────────────────────┘\n↓\nOutput Excel File + Analysis Report\n```\n\n## Data Format\n\n### Input Format\n\nThe system accepts Excel files with the following structure:\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n#### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n### Output Format\n\nThe system generates multiple output files:\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n#### Scoring Metrics\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Configuration\n\n### Basic Configuration\n\n### Environment Requirements\n- **Python**: 3.12+\n- **Operating System**: Windows, macOS, Linux\n- **Core Dependencies**: pandas, numpy, matplotlib, seaborn, asyncio, langchain, openpyxl, xlsxwriter\n\n### Installation Steps\n\n1. **Clone the repository**:\n```bash\ngit clone <repository-url>\ncd medical-ai-bench-eval\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Configure API Keys** (if using external AI model services):\n\n ### API Configuration Guide\n\n #### Environment Variables Setup\n\n **Linux/Mac:**\n ```bash\n # OpenAI\n export OPENAI_API_KEY=\"sk-your-actual-key\"\n export OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n\n # Claude (Anthropic)\n export ANTHROPIC_API_KEY=\"sk-ant-your-actual-key\"\n export ANTHROPIC_BASE_URL=\"https://api.anthropic.com\"\n\n # Google Gemini\n export GOOGLE_API_KEY=\"your-google-api-key\"\n export GOOGLE_BASE_URL=\"https://generativelanguage.googleapis.com/v1beta\"\n\n # Moonshot Kimi\n export MOONSHOT_API_KEY=\"sk-your-moonshot-key\"\n export MOONSHOT_BASE_URL=\"https://api.moonshot.cn/v1\"\n\n # Alibaba Qwen\n export DASHSCOPE_API_KEY=\"your-dashscope-key\"\n export DASHSCOPE_BASE_URL=\"https://dashscope.aliyuncs.com/api/v1\"\n\n # Baichuan\n export BAICHUAN_API_KEY=\"your-baichuan-key\"\n export BAICHUAN_BASE_URL=\"https://api.baichuan-ai.com/v1\"\n\n # DeepSeek\n export DEEPSEEK_API_KEY=\"sk-your-deepseek-key\"\n export DEEPSEEK_BASE_URL=\"https://api.deepseek.com/v1\"\n\n # Zhipu ChatGLM\n export ZHIPU_API_KEY=\"your-zhipu-key\"\n export ZHIPU_BASE_URL=\"https://open.bigmodel.cn/api/paas/v4\"\n ```\n\n **Windows PowerShell:**\n ```powershell\n $Env:OPENAI_API_KEY=\"sk-your-actual-key\"\n $Env:OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n ```\n\n #### Model Configuration\n\n **⚠️ IMPORTANT: OpenAI-Compatible API Only**\n \n This system currently supports **ONLY OpenAI-compatible API interfaces**. All models must provide OpenAI-compatible endpoints, regardless of the actual provider.\n\n **🌐 Third-Party Multi-Model Platforms (Recommended)**\n \n For models that don't natively support OpenAI-compatible APIs (like Claude, Gemini), you can use third-party platforms that provide unified OpenAI-compatible access:\n \n - **[OpenRouter](https://openrouter.ai/)** - Access Claude, Gemini, GPT, and many other models via OpenAI-compatible API\n - **[Together AI](https://together.ai/)** - Various open-source and commercial models\n - **[Anyscale](https://anyscale.com/)** - Multiple LLMs via OpenAI-compatible endpoints\n\n Edit `config_model.yaml` to configure your preferred models:\n\n ```yaml\n models:\n m1: # OpenAI GPT-4 (Native)\n api_key_env: \"OPENAI_API_KEY\"\n api_base_env: \"OPENAI_BASE_URL\"\n model: \"gpt-4-turbo-preview\"\n temperature: 0.2\n max_tokens: 4000\n\n m2: # Kimi (OpenAI-Compatible)\n api_key_env: \"MOONSHOT_API_KEY\"\n api_base_env: \"MOONSHOT_BASE_URL\"\n model: \"kimi-latest\"\n temperature: 0.2\n max_tokens: 4000\n\n m3: # Qwen (OpenAI-Compatible)\n api_key_env: \"DASHSCOPE_API_KEY\"\n api_base_env: \"DASHSCOPE_BASE_URL\"\n model: \"qwen-max\"\n temperature: 0.1\n max_tokens: 4000\n\n m4: # DeepSeek (OpenAI-Compatible)\n api_key_env: \"DEEPSEEK_API_KEY\"\n api_base_env: \"DEEPSEEK_BASE_URL\"\n model: \"deepseek-chat\"\n temperature: 0.2\n max_tokens: 4000\n\n m5: # Local Model (via OpenAI-Compatible Server)\n api_key_env: \"LOCAL_API_KEY\"\n api_base_env: \"LOCAL_BASE_URL\"\n model: \"llama3-medical\"\n temperature: 0.2\n max_tokens: 4000\n\n # OpenRouter Examples (uncomment to use)\n # m6: # Claude via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"anthropic/claude-3-5-sonnet-20241022\"\n # temperature: 0.2\n # max_tokens: 4000\n\n # m7: # Gemini via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"google/gemini-pro-1.5\"\n # temperature: 0.1\n # max_tokens: 4000\n ```\n\n #### Supported vs Unsupported Providers\n\n | Provider | Status | API Type | Notes |\n |----------|--------|----------|-------|\n | **OpenAI** | ✅ Supported | Native OpenAI | Direct support |\n | **OpenRouter** | ✅ Supported | Multi-Model Platform | Access Claude, Gemini, GPT via OpenAI-compatible API |\n | **Together AI** | ✅ Supported | Multi-Model Platform | Various models via OpenAI-compatible API |\n | **Anyscale** | ✅ Supported | Multi-Model Platform | Multiple LLMs via OpenAI-compatible API |\n | **Kimi/Moonshot** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Qwen/DashScope** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **DeepSeek** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Baichuan** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Zhipu** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Claude/Anthropic** | ⚠️ Via OpenRouter | Native Anthropic API | Use OpenRouter for OpenAI-compatible access |\n | **Gemini/Google** | ⚠️ Via OpenRouter | Native Google API | Use OpenRouter for OpenAI-compatible access |\n\n #### Medical Model Recommendations (OpenAI-Compatible Only)\n\n | Use Case | Recommended Models | Temperature | Notes |\n |----------|-------------------|-------------|--------|\n | **Diagnostic Tasks** | gpt-4-turbo, deepseek-chat | 0.1-0.2 | High accuracy |\n | **Chinese Medical** | qwen-max, baichuan-medical | 0.2-0.3 | Better CJK support |\n | **Cost-effective** | deepseek-chat, kimi | 0.2-0.3 | Good balance |\n | **Real-time** | kimi, deepseek-chat | 0.1-0.2 | Fast response |\n | **Privacy-focused** | local-llama3-medical | 0.2-0.3 | On-premise via OpenAI-compatible server |\n\n #### Quick Model Setup Commands\n\n **For GPT-4 testing:**\n ```bash\n export OPENAI_API_KEY=\"your-key-here\"\n # Run with GPT-4\n python medical_evaluation_pipeline.py input.xlsx --judge-models m1\n ```\n\n **For Chinese medical models:**\n ```bash\n export DASHSCOPE_API_KEY=\"your-key-here\"\n # Run with Qwen\n python medical_evaluation_pipeline.py input.xlsx --judge-models m4\n ```\n\n **For mixed model evaluation:**\n ```bash\n export OPENAI_API_KEY=\"your-key\"\n export DASHSCOPE_API_KEY=\"your-key\"\n # Use GPT-4 and Qwen for evaluation\n python medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m4 \\\n --extract-model m5\n\n### Basic Usage\n\n1. **Prepare input data**:\n - Excel file format\n - Contains columns for questions, scoring criteria (JSON format), model responses, etc.\n - Default column names can be configured in `config.yaml`\n\n2. **Run the complete evaluation pipeline**:\n```bash\npython medical_evaluation_pipeline.py input.xlsx -o output.xlsx\n```\n\n3. **View results**:\n - Output Excel file contains detailed review results and scores\n - Data analysis reports are saved in the `data/output/analysis/` directory\n\n## ⚙️ System Configuration\n\n### Configuration File (config.yaml)\n\n```yaml\n# File path configuration\nfiles:\n input_file: \"data/medical_evaluation_result.xlsx\"\n output_file: \"data/output/processed_result.xlsx\"\n\n# Process control\npipeline:\n disable_met_nomet: false # Whether to disable Met review\n disable_irrelevant_extraction: false # Whether to disable irrelevant content detection\n exclude_irrelevant_scoring: false # Whether to exclude irrelevant content from scoring\n disable_data_analysis: false # Whether to disable data analysis\n analysis_config_file: \"config_visualization.yaml\"\n\n# Model configuration\nmodels:\n judge_models: [\"m1\", \"m2\", \"m3\"] # Met review models\n extract_model: \"m5\" # Irrelevant content extraction model\n grade_models: [\"m1\", \"m2\", \"m3\"] # Irrelevant content grading models\n voting_strategy: \"conservative\" # Voting strategy\n\n# Column name mapping\ncolumns:\n question_col: \"question\"\n rubric_col: \"final_merged_json\"\n answer_columns:\n - \"gpt_5_answer\"\n - \"gemini_2_5_pro_answer\"\n - \"claude_opus_4_answer\"\n\n# Scoring system\npoint:\n A1: 5\n A2: 3\n A3: 1\n S1: -1\n S2: -2\n S3: -3\n S4: -4\n```\n\n### Command Line Arguments\n\n```bash\n# Process control\n--disable-met-nomet # Skip Met review step\n--disable-irrelevant # Skip irrelevant content detection\n--exclude-irrelevant-scoring # Exclude irrelevant content deduction from scoring\n--disable-data-analysis # Skip data analysis\n\n# Model configuration\n--judge-models m1 m2 # Specify Met review models\n--extract-model m3 # Specify irrelevant content extraction model\n--grade-models m1 m2 m4 # Specify irrelevant content grading models\n--voting-strategy majority # Voting strategy: conservative/majority/average\n\n# Data column configuration\n--question-col \"Question\" # Question column name\n--rubric-col \"Evaluation Points\" # Evaluation points column name\n--answer-columns \"Response1\" \"Response2\" \"Response3\" # Model response column names\n```\n\n## 📊 Thoracic Surgery Evaluation Dataset\n\n### Dataset Overview\n\nWe provide a specialized **Thoracic Surgery Evaluation Dataset** (`data/input/GAPS-NSCLC-preview.xlsx`) containing 92 carefully curated clinical cases for evaluating AI models' performance in thoracic surgery scenarios. This dataset focuses on complex clinical decision-making in chest surgery, particularly for non-small cell lung cancer (NSCLC) staging and treatment planning.\n\n### Dataset Structure\n\n| Column Name | Description | Data Type |\n|-------------|-------------|-----------|\n| **question** | Clinical questions covering thoracic surgery scenarios | Text |\n| **分类** (Category) | Medical specialty classification | Text |\n| **rubrics** | Evaluation criteria in JSON format with scoring levels | JSON Array |\n| **gpt_5_answer** | GPT-4 model responses to clinical questions | Text |\n| **gemini_2_5_pro_answer** | Gemini 2.5 Pro model responses | Text |\n| **claude_opus_4_answer** | Claude Opus model responses | Text |\n\n### Clinical Focus Areas\n\nThe dataset covers critical aspects of thoracic surgery, including:\n\n- **Pre-operative Evaluation**: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)\n- **Diagnostic Procedures**: EBUS-TBNA, mediastinoscopy, PET-CT interpretation\n- **Staging Assessment**: TNM staging, mediastinal lymph node evaluation\n- **Treatment Planning**: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions\n- **Risk Assessment**: Pulmonary function evaluation, cardiac risk stratification\n- **Molecular Diagnostics**: EGFR, ALK, PD-L1 testing strategies\n\n### Evaluation Rubrics Format\n\nEach question includes detailed evaluation criteria in JSON format:\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Should perform enhanced chest CT covering thoracic inlet to upper abdomen\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"PET-CT is essential for accurate N-staging and excluding distant metastases\",\n \"level\": \"A2\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Incorrectly generalizing upper sulcus tumor MRI requirements to all IIB/IIIA patients\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n]\n```\n\n**Scoring Levels**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Sample Clinical Question\n\n**Question**: \"For NSCLC patients with IIB-IIIA staging, what comprehensive pre-treatment evaluation should be performed?\"\n\n**Key Evaluation Points**:\n- Pathological diagnosis and molecular characterization\n- Accurate clinical staging (T, N, M assessment)\n- Invasive mediastinal staging for N2 evaluation\n- Pulmonary and cardiac function assessment\n- Multidisciplinary team (MDT) treatment planning\n\n### Usage with GAPS Framework\n\nThis dataset is designed to work seamlessly with the GAPS evaluation pipeline:\n\n```bash\n# Evaluate thoracic surgery dataset\npython medical_evaluation_pipeline.py data/input/GAPS-NSCLC-preview.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -o results/thoracic_surgery_evaluation.xlsx\n```\n\n### Clinical Validation\n\nThe dataset has been validated by:\n- **Board-certified thoracic surgeons**\n- **Pulmonary oncologists** \n- **Medical imaging specialists**\n- **Pathologists specializing in lung cancer**\n\nAll evaluation criteria are based on current clinical guidelines including:\n- NCCN Guidelines for Non-Small Cell Lung Cancer\n- ESTS Guidelines for Intraoperative Lymph Node Staging\n- IASLC Staging Manual in Thoracic Oncology\n\n### Research Applications\n\nThis dataset enables research in:\n- **AI Clinical Decision Support**: Evaluating AI models' ability to provide accurate clinical recommendations\n- **Medical Education**: Training and assessment of clinical reasoning skills\n- **Quality Assurance**: Benchmarking AI systems against established clinical standards\n- **Comparative Analysis**: Cross-model performance evaluation in specialized medical domains\n\n### Data Quality Metrics\n\n- **Total Cases**: 92 clinical scenarios\n- **Completeness**: 100% data coverage across all columns\n- **Clinical Diversity**: Covers full spectrum of IIB-IIIA NSCLC presentations\n- **Expert Validation**: All cases reviewed by multidisciplinary clinical team\n\n---\n\n## Input Data Format\n\n### Excel File Structure\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n## Output Results\n\n### Main Output Files\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n### Scoring Metrics Description\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Advanced Features\n\n### 1. Parallel Processing Optimization\n\nThe system supports parallel execution of Step 1 (Met Review) and Step 2 (Irrelevant Content Detection):\n\n```bash\n# Enable all steps (auto parallel)\npython medical_evaluation_pipeline.py input.xlsx\n\n# Enable only Met Review\npython medical_evaluation_pipeline.py input.xlsx --disable-irrelevant\n\n# Enable only Irrelevant Content Detection\npython medical_evaluation_pipeline.py input.xlsx --disable-met-nomet\n```\n\n### 2. Voting Strategy\n\nSupports three voting strategies for processing multi-model results:\n- **conservative**: Conservative strategy, prioritizes more severe ratings\n- **majority**: Majority voting, selects the result with the most votes\n- **average**: Average strategy, calculates based on numerical average\n\n```bash\npython medical_evaluation_pipeline.py input.xlsx --voting-strategy conservative\n```\n\n### 3. Custom Model Configuration\n\n```bash\n# Use specific review model combinations\npython medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m2 \\\n --extract-model m5 \\\n --grade-models m1 m3 m4\n```\n\n### 4. Data Analysis Customization\n\nCreate `config_visualization.yaml` for custom analysis configuration:\n\n```yaml\nanalysis:\n input_file: \"data/processed_result.xlsx\"\n columns:\n model_scores:\n GPT-5: \"gpt_5_answer_judged_json_scores\"\n Gemini-2.5-Pro: \"gemini_2_5_pro_answer_judged_json_scores\"\n Claude-Opus-4: \"claude_opus_4_answer_judged_json_scores\"\n\nvisualization:\n charts:\n bar_overall: true\n boxplot_distribution: true\n heatmap_category: true\n radar_chart: true\n style:\n figure_size: [24, 16]\n dpi: 300\n```\n\n## 🔧 Independent Module Usage\n\n### 1. Run Met Review Separately\n\n```python\nimport asyncio\nfrom clinical_answer_judge import MultiModelMergedJsonJudge\n\nasync def run_judge():\n judge = MultiModelMergedJsonJudge(\n judge_models=['m1', 'm2', 'm3'],\n question_col='question',\n rubric_col='rubrics'\n )\n result = await judge.process_excel_file('input.xlsx', 'output.xlsx')\n return result\n\n# Run the async function\nresult = asyncio.run(run_judge())\n```\n\n### 2. Run Score Calculation Separately\n\n```python\nfrom clinical_scoring_calculator import process_excel_data\n\nresult = process_excel_data(\n input_file='judged_data.xlsx',\n output_file='scored_data.xlsx',\n model_columns=['gpt_5_answer_judged_json']\n)\n```\n\n### 3. Run Data Analysis Separately\n\n```python\nfrom enhanced_analyzer import EnhancedMedicalAnalyzer\n\nanalyzer = EnhancedMedicalAnalyzer('config_visualization.yaml')\nanalyzer.run_analysis()\n```\n\n## Usage Examples\n\n### Example 1: Complete Process Handling\n\n```bash\n# Process complete medical evaluation data\npython medical_evaluation_pipeline.py medical_data.xlsx \\\n -o results/final_report.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -v\n```\n\n### Example 2: Quick Testing\n\n```bash\n# Process only first 10 rows of data\npython medical_evaluation_pipeline.py test_data.xlsx \\\n --max_rows 10 \\\n --disable-data-analysis\n```\n\n### Example 3: Custom Column Names\n\n```bash\n# Use Chinese column names\npython medical_evaluation_pipeline.py data.xlsx \\\n --question-col \"Clinical Question\" \\\n --rubric-col \"Scoring Criteria\" \\\n --answer-columns \"GPT Response\" \"Claude Response\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **File Not Found Error**\n - Solution: Check if the input file path is correct\n\n2. **Column Name Mismatch**\n - Solution: Use `--question-col`, `--rubric-col` and other parameters to specify correct column names\n\n3. **Insufficient Memory**\n - Solution: Use `--max_rows` parameter to limit the number of rows processed\n\n4. **API Call Failure**\n - Solution: Check model configuration and network connection\n\n### Debug Mode\n\n```bash\n# Enable verbose logging to view processing details\npython medical_evaluation_pipeline.py input.xlsx -v\n\n# Keep temporary files for debugging\n# Modify the cleanup() call in the code\n```\n\n## Technical Documentation\n\n### Core Modules\n\n- `medical_evaluation_pipeline.py` - Main pipeline controller\n- `clinical_answer_judge.py` - Met/Not Met review module\n- `clinical_scoring_calculator.py` - Score calculation module\n- `enhanced_analyzer.py` - Data analysis and visualization module\n- `general_negative_points.py` - Irrelevant content processing module\n\n### File Structure\n\n```\nMedicalAiBenchEval/\n├── clinical_answer_judge.py # Multi-model review processor\n├── clinical_scoring_calculator.py # Score calculator\n├── medical_evaluation_pipeline.py # Main pipeline controller\n├── enhanced_analyzer.py # Data analyzer\n├── judge_engine.py # Review engine\n├── irrelevant_content_grading.py # Irrelevant content scoring\n├── extract_irrelevant_content.py # Irrelevant content extraction\n├── general_negative_points.py # Negative point processing\n├── config.yaml # Main configuration file\n├── config_model.yaml # Model configuration file\n├── config_visualization.yaml # Visualization configuration\n├── requirements.txt # Dependency list\n└── data/ # Data directory\n ├── input/ # Input files\n │ ├── GAPS-NSCLC-preview.xlsx # Thoracic surgery evaluation dataset (92 cases)\n │ └── community_contributions.xlsx # Community contributed cases\n └── output/ # Output files\n └── analysis/ # Analysis reports\n```\n\n### Data Flow\n\n1. **Input Phase**: Read Excel file, validate data format\n2. **Parallel Processing**: Simultaneously execute Met review and irrelevant content detection\n3. **Result Merging**: Intelligently merge outputs from both parallel steps\n4. **Score Calculation**: Calculate multi-dimensional scores based on merged results\n5. **Analysis Output**: Generate statistical reports and visualization charts\n\n### Performance Optimization\n\n- Asynchronous parallel processing improves efficiency\n- Batch processing reduces API call frequency\n- Intelligent retry mechanism handles network exceptions\n- Memory optimization handles large files\n\n## Contribution Guidelines\n\nWelcome to submit Issues and Pull Requests to improve system functionality.\n\n### Development Environment Setup\n\n```bash\ngit clone <repository>\ncd medical-ai-bench-eval\npip install -r requirements.txt\n```\n\n### Code Standards\n\n- Follow PEP 8 coding standards\n- Add detailed docstrings\n- Include unit tests\n- Update README documentation\n\n## Citation\n\nIf you use this system in your research, please cite our paper:\n\n**English:**\n> Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Jun Wang, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan Yang (2025). GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians. *arXiv preprint* arXiv:2510.13734. https://arxiv.org/abs/2510.13734\n\n```bibtex\n@article{chen2025gaps,\n title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},\n author={Chen, Xiuyuan and Sun, Tao and Su, Dexin and Yu, Ailing and Liu, Junwei and Chen, Zhe and Jin, Gangzeng and Wang, Xin and Liu, Jingnan and Xiao, Hansong and Zhou, Hualei and Tao, Dongjie and Guo, Chunxiao and Yang, Minghui and Xia, Yuan and Zhao, Jing and Fan, Qianrui and Wang, Yanyun and Zhen, Shuai and Chen, Kezhong and Wang, Jun and Sun, Zewen and Zhao, Heng and Guan, Tian and Wang, Shaodong and Chang, Geyun and Deng, Jiaming and Chen, Hongchengcheng and Feng, Kexin and Li, Ruzhen and Geng, Jiayi and Zhao, Changtai and Wang, Jun and Lin, Guihu and Li, Peihao and Liu, Liqi and Wei, Peng and Wang, Jian and Gu, Jinjie and Wang, Ping and Yang, Fan},\n journal={arXiv preprint arXiv:2510.13734},\n year={2025},\n url={https://arxiv.org/abs/2510.13734}\n}\n```\n\n## 🤝 Community Contribution / 共创呼吁\n\nWe welcome contributions from the medical AI community to expand and improve our evaluation dataset! Your expertise can help build a more comprehensive benchmark for medical AI systems.\n\n### 📋 How to Contribute\n\n#### Option 1: Submit an Issue\nCreate a new issue with the label `data-contribution` using the following template:\n\n```markdown\n**Medical Question**\n[Provide your clinical question here]\n\n**Evaluation Rubrics (JSON Format)**\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Your evaluation criterion description\",\n \"level\": \"A1|A2|A3|S1|S2|S3|S4\",\n \"type\": \"positive|negative\"\n },\n {\n \"id\": 2,\n \"claim\": \"Another evaluation criterion\",\n \"level\": \"A2\",\n \"type\": \"positive\"\n }\n]\n```\n\n**Medical Specialty**\n[e.g., Cardiology, Pulmonology, Oncology, etc.]\n\n**Question Type**\n[e.g., Diagnosis, Treatment, Risk Assessment, etc.]\n\n**Additional Context**\n[Any relevant clinical context or references]\n```\n\n#### Option 2: Submit a Pull Request\n1. Fork this repository\n2. Add your data to `data/input/community_contributions.xlsx` following the existing format:\n - **question**: Your medical question\n - **rubrics**: JSON array of evaluation criteria\n - **specialty**: Medical specialty area\n - **difficulty**: Easy/Medium/Hard\n - **source**: Your name/institution (optional)\n\n3. Ensure your contribution follows these guidelines:\n - **Clinical Relevance**: Questions should reflect real-world medical scenarios\n - **Evidence-Based**: Evaluation criteria should be based on established medical guidelines\n - **Clear Scoring**: Use our scoring system appropriately:\n - **Positive Points**: A1 (5 pts), A2 (3 pts), A3 (1 pt)\n - **Negative Points**: S1 (-1 pt), S2 (-2 pts), S3 (-3 pts), S4 (-4 pts)\n\n### 📝 Contribution Guidelines\n\n#### Evaluation Criteria Design\n- **A1 (Critical, 5 points)**: Essential medical knowledge that could impact patient safety\n- **A2 (Important, 3 points)**: Significant clinical considerations\n- **A3 (Helpful, 1 point)**: Additional relevant information\n- **S1 (Minor Error, -1 point)**: Small inaccuracies that don't affect core treatment\n- **S2 (Moderate Error, -2 points)**: Incorrect information that could mislead\n- **S3 (Major Error, -3 points)**: Serious medical errors\n- **S4 (Critical Error, -4 points)**: Dangerous misinformation that could harm patients\n\n#### Quality Standards\n- Questions should be **clinically realistic** and based on actual medical scenarios\n- Evaluation criteria must be **evidence-based** and reference current medical guidelines\n- Each question should have **3-8 evaluation points** for balanced assessment\n- Include both **positive criteria** (what should be mentioned) and **negative criteria** (what should be avoided)\n\n### 🎯 Priority Areas\nWe especially welcome contributions in:\n- **Rare Diseases**: Uncommon conditions that challenge AI systems\n- **Multidisciplinary Cases**: Complex cases requiring multiple specialties\n- **Cultural Considerations**: Cases reflecting diverse patient populations\n- **Emergency Medicine**: Time-critical decision-making scenarios\n- **Pediatric Medicine**: Age-specific medical considerations\n\n### 📊 Data Format Example\n```json\n{\n \"question\": \"A 65-year-old patient presents with chest pain and shortness of breath. How would you approach the initial evaluation?\",\n \"rubrics\": [\n {\n \"id\": 1,\n \"claim\": \"Should obtain detailed history including pain characteristics, timing, and associated symptoms\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"Must perform ECG and chest X-ray as initial diagnostic tests\",\n \"level\": \"A1\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Should not delay evaluation with unnecessary tests in acute presentation\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n ],\n \"specialty\": \"Emergency Medicine\",\n \"difficulty\": \"Medium\"\n}\n```\n\n### 🔄 Review Process\n1. **Community Review**: Contributions will be reviewed by medical professionals\n2. **Quality Check**: Ensure clinical accuracy and appropriate difficulty\n3. **Integration**: Approved contributions will be merged into the main dataset\n4. **Attribution**: Contributors will be acknowledged in our contributor list\n\n### 📞 Contact\nFor questions about contributing, please:\n- Open an issue with the `question` label\n- Join our discussion in the GitHub Discussions section\n\nThank you for helping build a better benchmark for medical AI evaluation! 🙏\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n🤗 <a href=\"https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview\" target=\"_blank\">HuggingFace Dataset</a> ｜\n📄 <a href=\"https://arxiv.org/abs/2510.13734\" target=\"_blank\">arXiv</a> ｜\n🌐 <a href=\"README_ZH.md\">中文</a> ｜\n"}, "d04bb3dea168e9e1e4395ae2c60fa14e40ea2633": {"t": 1785766591.6586654, "n": 36411, "h": "72911d5dd00c14f5", "c": "# Medical AI Evaluation Framework: Clinical Benchmark Dataset and Automated Assessment Pipeline\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n - [Installation](#installation)\n - [Basic Usage](#basic-usage)\n- [System Architecture](#system-architecture)\n- [Data Format](#data-format)\n - [Input Format](#input-format)\n - [Output Format](#output-format)\n- [Configuration](#configuration)\n - [Basic Configuration](#basic-configuration)\n - [Advanced Configuration](#advanced-configuration)\n- [Usage Guide](#usage-guide)\n - [Command Line Interface](#command-line-interface)\n - [Python API](#python-api)\n - [Examples](#examples)\n- [Evaluation Framework](#evaluation-framework)\n - [Scoring System](#scoring-system)\n - [Evaluation Metrics](#evaluation-metrics)\n- [Dataset](#dataset)\n - [Thoracic Surgery Dataset](#thoracic-surgery-dataset)\n - [Data Quality](#data-quality)\n- [Advanced Features](#advanced-features)\n - [Parallel Processing](#parallel-processing)\n - [Voting Strategies](#voting-strategies)\n - [Custom Models](#custom-models)\n- [Development](#development)\n - [Technical Documentation](#technical-documentation)\n - [API Reference](#api-reference)\n - [Troubleshooting](#troubleshooting)\n- [Community](#community)\n - [Contributing](#contributing)\n - [Citation](#citation)\n- [License](#license)\n\n## Overview\n\nThis Medical AI Evaluation Framework provides a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this framework includes both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.\n\nThe framework addresses the critical need for standardized evaluation of AI clinical decision-making by providing:\n- **Clinically Grounded Assessment**: Evaluation criteria based on real medical guidelines and expert knowledge\n- **Automated Pipeline**: Streamlined processing from raw responses to detailed performance metrics\n- **Multi-Model Support**: Simultaneous evaluation of multiple AI models with comparative analysis\n- **Scalable Architecture**: Efficient processing of large datasets with parallel execution capabilities\n\n## Key Features\n\n- **🏥 Medical-Specific Evaluation**: Specialized rubrics for clinical scenarios with positive/negative scoring\n- **🔄 Parallel Processing**: Simultaneous execution of Met/Not Met review and irrelevant content detection\n- **📊 Comprehensive Analytics**: Detailed statistical analysis with visualization reports\n- **🎯 Multi-Model Assessment**: Support for evaluating multiple AI models simultaneously\n- **⚙️ Flexible Configuration**: Customizable models, voting strategies, and evaluation parameters\n- **📈 Rich Visualization**: Automated generation of performance charts and comparative analysis\n- **🔧 Modular Design**: Independent modules for different evaluation stages\n- **📋 Standardized Output**: Consistent Excel-based reporting with detailed metrics\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd medical-ai-bench-eval\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Basic Usage\n\n```bash\n# Run complete evaluation pipeline\npython medical_evaluation_pipeline.py input_data.xlsx\n\n# With custom output file\npython medical_evaluation_pipeline.py input_data.xlsx -o results/evaluation_results.xlsx\n\n# Enable verbose logging\npython medical_evaluation_pipeline.py input_data.xlsx -v\n```\n\n## System Architecture\n\nThe system processes medical AI responses through a sophisticated pipeline that includes:\n\n1. **Input Processing**: Reads Excel files containing medical questions, evaluation rubrics, and AI model responses\n2. **Parallel Evaluation**: Simultaneously executes Met/Not Met review and irrelevant content detection\n3. **Intelligent Scoring**: Calculates comprehensive scores based on clinical evaluation criteria\n4. **Analysis & Reporting**: Generates detailed statistical reports and visualizations\n\n## System Architecture\n\n```\nInput Excel File\n↓\n┌─────────────────────────────────────────┐\n│ Parallel Processing Phase │\n├─────────────────┬─────────────────────┤\n│ Step 1: Met Review │ Step 2: Irrelevant Content │\n│ - Multi-model Review │ - Content Extraction │\n│ - Voting Decision │ - Level Assessment │\n│ - Result Summary │ - Voting Classification │\n└─────────────────┴─────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Result Merging │\n│ - Intelligent merging of parallel results │\n│ - Data integrity check │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 3: Score Calculation │\n│ - Multi-dimensional score statistics │\n│ - Irrelevant content deduction │\n│ - Normalization processing │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 4: Data Analysis & Visualization │\n│ - Statistical analysis reports │\n│ - Diverse charts │\n│ - CSV data export │\n└─────────────────────────────────────────┘\n↓\nOutput Excel File + Analysis Report\n```\n\n## Data Format\n\n### Input Format\n\nThe system accepts Excel files with the following structure:\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n#### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n### Output Format\n\nThe system generates multiple output files:\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n#### Scoring Metrics\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Configuration\n\n### Basic Configuration\n\n### Environment Requirements\n- **Python**: 3.12+\n- **Operating System**: Windows, macOS, Linux\n- **Core Dependencies**: pandas, numpy, matplotlib, seaborn, asyncio, langchain, openpyxl, xlsxwriter\n\n### Installation Steps\n\n1. **Clone the repository**:\n```bash\ngit clone <repository-url>\ncd medical-ai-bench-eval\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Configure API Keys** (if using external AI model services):\n\n ### API Configuration Guide\n\n #### Environment Variables Setup\n\n **Linux/Mac:**\n ```bash\n # OpenAI\n export OPENAI_API_KEY=\"sk-your-actual-key\"\n export OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n\n # Claude (Anthropic)\n export ANTHROPIC_API_KEY=\"sk-ant-your-actual-key\"\n export ANTHROPIC_BASE_URL=\"https://api.anthropic.com\"\n\n # Google Gemini\n export GOOGLE_API_KEY=\"your-google-api-key\"\n export GOOGLE_BASE_URL=\"https://generativelanguage.googleapis.com/v1beta\"\n\n # Moonshot Kimi\n export MOONSHOT_API_KEY=\"sk-your-moonshot-key\"\n export MOONSHOT_BASE_URL=\"https://api.moonshot.cn/v1\"\n\n # Alibaba Qwen\n export DASHSCOPE_API_KEY=\"your-dashscope-key\"\n export DASHSCOPE_BASE_URL=\"https://dashscope.aliyuncs.com/api/v1\"\n\n # Baichuan\n export BAICHUAN_API_KEY=\"your-baichuan-key\"\n export BAICHUAN_BASE_URL=\"https://api.baichuan-ai.com/v1\"\n\n # DeepSeek\n export DEEPSEEK_API_KEY=\"sk-your-deepseek-key\"\n export DEEPSEEK_BASE_URL=\"https://api.deepseek.com/v1\"\n\n # Zhipu ChatGLM\n export ZHIPU_API_KEY=\"your-zhipu-key\"\n export ZHIPU_BASE_URL=\"https://open.bigmodel.cn/api/paas/v4\"\n ```\n\n **Windows PowerShell:**\n ```powershell\n $Env:OPENAI_API_KEY=\"sk-your-actual-key\"\n $Env:OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n ```\n\n #### Model Configuration\n\n **⚠️ IMPORTANT: OpenAI-Compatible API Only**\n \n This system currently supports **ONLY OpenAI-compatible API interfaces**. All models must provide OpenAI-compatible endpoints, regardless of the actual provider.\n\n **🌐 Third-Party Multi-Model Platforms (Recommended)**\n \n For models that don't natively support OpenAI-compatible APIs (like Claude, Gemini), you can use third-party platforms that provide unified OpenAI-compatible access:\n \n - **[OpenRouter](https://openrouter.ai/)** - Access Claude, Gemini, GPT, and many other models via OpenAI-compatible API\n - **[Together AI](https://together.ai/)** - Various open-source and commercial models\n - **[Anyscale](https://anyscale.com/)** - Multiple LLMs via OpenAI-compatible endpoints\n\n Edit `config_model.yaml` to configure your preferred models:\n\n ```yaml\n models:\n m1: # OpenAI GPT-4 (Native)\n api_key_env: \"OPENAI_API_KEY\"\n api_base_env: \"OPENAI_BASE_URL\"\n model: \"gpt-4-turbo-preview\"\n temperature: 0.2\n max_tokens: 4000\n\n m2: # Kimi (OpenAI-Compatible)\n api_key_env: \"MOONSHOT_API_KEY\"\n api_base_env: \"MOONSHOT_BASE_URL\"\n model: \"kimi-latest\"\n temperature: 0.2\n max_tokens: 4000\n\n m3: # Qwen (OpenAI-Compatible)\n api_key_env: \"DASHSCOPE_API_KEY\"\n api_base_env: \"DASHSCOPE_BASE_URL\"\n model: \"qwen-max\"\n temperature: 0.1\n max_tokens: 4000\n\n m4: # DeepSeek (OpenAI-Compatible)\n api_key_env: \"DEEPSEEK_API_KEY\"\n api_base_env: \"DEEPSEEK_BASE_URL\"\n model: \"deepseek-chat\"\n temperature: 0.2\n max_tokens: 4000\n\n m5: # Local Model (via OpenAI-Compatible Server)\n api_key_env: \"LOCAL_API_KEY\"\n api_base_env: \"LOCAL_BASE_URL\"\n model: \"llama3-medical\"\n temperature: 0.2\n max_tokens: 4000\n\n # OpenRouter Examples (uncomment to use)\n # m6: # Claude via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"anthropic/claude-3-5-sonnet-20241022\"\n # temperature: 0.2\n # max_tokens: 4000\n\n # m7: # Gemini via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"google/gemini-pro-1.5\"\n # temperature: 0.1\n # max_tokens: 4000\n ```\n\n #### Supported vs Unsupported Providers\n\n | Provider | Status | API Type | Notes |\n |----------|--------|----------|-------|\n | **OpenAI** | ✅ Supported | Native OpenAI | Direct support |\n | **OpenRouter** | ✅ Supported | Multi-Model Platform | Access Claude, Gemini, GPT via OpenAI-compatible API |\n | **Together AI** | ✅ Supported | Multi-Model Platform | Various models via OpenAI-compatible API |\n | **Anyscale** | ✅ Supported | Multi-Model Platform | Multiple LLMs via OpenAI-compatible API |\n | **Kimi/Moonshot** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Qwen/DashScope** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **DeepSeek** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Baichuan** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Zhipu** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Claude/Anthropic** | ⚠️ Via OpenRouter | Native Anthropic API | Use OpenRouter for OpenAI-compatible access |\n | **Gemini/Google** | ⚠️ Via OpenRouter | Native Google API | Use OpenRouter for OpenAI-compatible access |\n\n #### Medical Model Recommendations (OpenAI-Compatible Only)\n\n | Use Case | Recommended Models | Temperature | Notes |\n |----------|-------------------|-------------|--------|\n | **Diagnostic Tasks** | gpt-4-turbo, deepseek-chat | 0.1-0.2 | High accuracy |\n | **Chinese Medical** | qwen-max, baichuan-medical | 0.2-0.3 | Better CJK support |\n | **Cost-effective** | deepseek-chat, kimi | 0.2-0.3 | Good balance |\n | **Real-time** | kimi, deepseek-chat | 0.1-0.2 | Fast response |\n | **Privacy-focused** | local-llama3-medical | 0.2-0.3 | On-premise via OpenAI-compatible server |\n\n #### Quick Model Setup Commands\n\n **For GPT-4 testing:**\n ```bash\n export OPENAI_API_KEY=\"your-key-here\"\n # Run with GPT-4\n python medical_evaluation_pipeline.py input.xlsx --judge-models m1\n ```\n\n **For Chinese medical models:**\n ```bash\n export DASHSCOPE_API_KEY=\"your-key-here\"\n # Run with Qwen\n python medical_evaluation_pipeline.py input.xlsx --judge-models m4\n ```\n\n **For mixed model evaluation:**\n ```bash\n export OPENAI_API_KEY=\"your-key\"\n export DASHSCOPE_API_KEY=\"your-key\"\n # Use GPT-4 and Qwen for evaluation\n python medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m4 \\\n --extract-model m5\n\n### Basic Usage\n\n1. **Prepare input data**:\n - Excel file format\n - Contains columns for questions, scoring criteria (JSON format), model responses, etc.\n - Default column names can be configured in `config.yaml`\n\n2. **Run the complete evaluation pipeline**:\n```bash\npython medical_evaluation_pipeline.py input.xlsx -o output.xlsx\n```\n\n3. **View results**:\n - Output Excel file contains detailed review results and scores\n - Data analysis reports are saved in the `data/output/analysis/` directory\n\n## ⚙️ System Configuration\n\n### Configuration File (config.yaml)\n\n```yaml\n# File path configuration\nfiles:\n input_file: \"data/medical_evaluation_result.xlsx\"\n output_file: \"data/output/processed_result.xlsx\"\n\n# Process control\npipeline:\n disable_met_nomet: false # Whether to disable Met review\n disable_irrelevant_extraction: false # Whether to disable irrelevant content detection\n exclude_irrelevant_scoring: false # Whether to exclude irrelevant content from scoring\n disable_data_analysis: false # Whether to disable data analysis\n analysis_config_file: \"config_visualization.yaml\"\n\n# Model configuration\nmodels:\n judge_models: [\"m1\", \"m2\", \"m3\"] # Met review models\n extract_model: \"m5\" # Irrelevant content extraction model\n grade_models: [\"m1\", \"m2\", \"m3\"] # Irrelevant content grading models\n voting_strategy: \"conservative\" # Voting strategy\n\n# Column name mapping\ncolumns:\n question_col: \"question\"\n rubric_col: \"final_merged_json\"\n answer_columns:\n - \"gpt_5_answer\"\n - \"gemini_2_5_pro_answer\"\n - \"claude_opus_4_answer\"\n\n# Scoring system\npoint:\n A1: 5\n A2: 3\n A3: 1\n S1: -1\n S2: -2\n S3: -3\n S4: -4\n```\n\n### Command Line Arguments\n\n```bash\n# Process control\n--disable-met-nomet # Skip Met review step\n--disable-irrelevant # Skip irrelevant content detection\n--exclude-irrelevant-scoring # Exclude irrelevant content deduction from scoring\n--disable-data-analysis # Skip data analysis\n\n# Model configuration\n--judge-models m1 m2 # Specify Met review models\n--extract-model m3 # Specify irrelevant content extraction model\n--grade-models m1 m2 m4 # Specify irrelevant content grading models\n--voting-strategy majority # Voting strategy: conservative/majority/average\n\n# Data column configuration\n--question-col \"Question\" # Question column name\n--rubric-col \"Evaluation Points\" # Evaluation points column name\n--answer-columns \"Response1\" \"Response2\" \"Response3\" # Model response column names\n```\n\n## 📊 Thoracic Surgery Evaluation Dataset\n\n### Dataset Overview\n\nWe provide a specialized **Thoracic Surgery Evaluation Dataset** (`data/input/GAPS-NSCLC-preview.xlsx`) containing 92 carefully curated clinical cases for evaluating AI models' performance in thoracic surgery scenarios. This dataset focuses on complex clinical decision-making in chest surgery, particularly for non-small cell lung cancer (NSCLC) staging and treatment planning.\n\n### Dataset Structure\n\n| Column Name | Description | Data Type |\n|-------------|-------------|-----------|\n| **question** | Clinical questions covering thoracic surgery scenarios | Text |\n| **分类** (Category) | Medical specialty classification | Text |\n| **rubrics** | Evaluation criteria in JSON format with scoring levels | JSON Array |\n| **gpt_5_answer** | GPT-4 model responses to clinical questions | Text |\n| **gemini_2_5_pro_answer** | Gemini 2.5 Pro model responses | Text |\n| **claude_opus_4_answer** | Claude Opus model responses | Text |\n\n### Clinical Focus Areas\n\nThe dataset covers critical aspects of thoracic surgery, including:\n\n- **Pre-operative Evaluation**: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)\n- **Diagnostic Procedures**: EBUS-TBNA, mediastinoscopy, PET-CT interpretation\n- **Staging Assessment**: TNM staging, mediastinal lymph node evaluation\n- **Treatment Planning**: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions\n- **Risk Assessment**: Pulmonary function evaluation, cardiac risk stratification\n- **Molecular Diagnostics**: EGFR, ALK, PD-L1 testing strategies\n\n### Evaluation Rubrics Format\n\nEach question includes detailed evaluation criteria in JSON format:\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Should perform enhanced chest CT covering thoracic inlet to upper abdomen\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"PET-CT is essential for accurate N-staging and excluding distant metastases\",\n \"level\": \"A2\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Incorrectly generalizing upper sulcus tumor MRI requirements to all IIB/IIIA patients\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n]\n```\n\n**Scoring Levels**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Sample Clinical Question\n\n**Question**: \"For NSCLC patients with IIB-IIIA staging, what comprehensive pre-treatment evaluation should be performed?\"\n\n**Key Evaluation Points**:\n- Pathological diagnosis and molecular characterization\n- Accurate clinical staging (T, N, M assessment)\n- Invasive mediastinal staging for N2 evaluation\n- Pulmonary and cardiac function assessment\n- Multidisciplinary team (MDT) treatment planning\n\n### Usage with GAPS Framework\n\nThis dataset is designed to work seamlessly with the GAPS evaluation pipeline:\n\n```bash\n# Evaluate thoracic surgery dataset\npython medical_evaluation_pipeline.py data/input/GAPS-NSCLC-preview.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -o results/thoracic_surgery_evaluation.xlsx\n```\n\n### Clinical Validation\n\nThe dataset has been validated by:\n- **Board-certified thoracic surgeons**\n- **Pulmonary oncologists** \n- **Medical imaging specialists**\n- **Pathologists specializing in lung cancer**\n\nAll evaluation criteria are based on current clinical guidelines including:\n- NCCN Guidelines for Non-Small Cell Lung Cancer\n- ESTS Guidelines for Intraoperative Lymph Node Staging\n- IASLC Staging Manual in Thoracic Oncology\n\n### Research Applications\n\nThis dataset enables research in:\n- **AI Clinical Decision Support**: Evaluating AI models' ability to provide accurate clinical recommendations\n- **Medical Education**: Training and assessment of clinical reasoning skills\n- **Quality Assurance**: Benchmarking AI systems against established clinical standards\n- **Comparative Analysis**: Cross-model performance evaluation in specialized medical domains\n\n### Data Quality Metrics\n\n- **Total Cases**: 92 clinical scenarios\n- **Completeness**: 100% data coverage across all columns\n- **Clinical Diversity**: Covers full spectrum of IIB-IIIA NSCLC presentations\n- **Expert Validation**: All cases reviewed by multidisciplinary clinical team\n\n---\n\n## Input Data Format\n\n### Excel File Structure\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n## Output Results\n\n### Main Output Files\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n### Scoring Metrics Description\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Advanced Features\n\n### 1. Parallel Processing Optimization\n\nThe system supports parallel execution of Step 1 (Met Review) and Step 2 (Irrelevant Content Detection):\n\n```bash\n# Enable all steps (auto parallel)\npython medical_evaluation_pipeline.py input.xlsx\n\n# Enable only Met Review\npython medical_evaluation_pipeline.py input.xlsx --disable-irrelevant\n\n# Enable only Irrelevant Content Detection\npython medical_evaluation_pipeline.py input.xlsx --disable-met-nomet\n```\n\n### 2. Voting Strategy\n\nSupports three voting strategies for processing multi-model results:\n- **conservative**: Conservative strategy, prioritizes more severe ratings\n- **majority**: Majority voting, selects the result with the most votes\n- **average**: Average strategy, calculates based on numerical average\n\n```bash\npython medical_evaluation_pipeline.py input.xlsx --voting-strategy conservative\n```\n\n### 3. Custom Model Configuration\n\n```bash\n# Use specific review model combinations\npython medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m2 \\\n --extract-model m5 \\\n --grade-models m1 m3 m4\n```\n\n### 4. Data Analysis Customization\n\nCreate `config_visualization.yaml` for custom analysis configuration:\n\n```yaml\nanalysis:\n input_file: \"data/processed_result.xlsx\"\n columns:\n model_scores:\n GPT-5: \"gpt_5_answer_judged_json_scores\"\n Gemini-2.5-Pro: \"gemini_2_5_pro_answer_judged_json_scores\"\n Claude-Opus-4: \"claude_opus_4_answer_judged_json_scores\"\n\nvisualization:\n charts:\n bar_overall: true\n boxplot_distribution: true\n heatmap_category: true\n radar_chart: true\n style:\n figure_size: [24, 16]\n dpi: 300\n```\n\n## 🔧 Independent Module Usage\n\n### 1. Run Met Review Separately\n\n```python\nimport asyncio\nfrom clinical_answer_judge import MultiModelMergedJsonJudge\n\nasync def run_judge():\n judge = MultiModelMergedJsonJudge(\n judge_models=['m1', 'm2', 'm3'],\n question_col='question',\n rubric_col='rubrics'\n )\n result = await judge.process_excel_file('input.xlsx', 'output.xlsx')\n return result\n\n# Run the async function\nresult = asyncio.run(run_judge())\n```\n\n### 2. Run Score Calculation Separately\n\n```python\nfrom clinical_scoring_calculator import process_excel_data\n\nresult = process_excel_data(\n input_file='judged_data.xlsx',\n output_file='scored_data.xlsx',\n model_columns=['gpt_5_answer_judged_json']\n)\n```\n\n### 3. Run Data Analysis Separately\n\n```python\nfrom enhanced_analyzer import EnhancedMedicalAnalyzer\n\nanalyzer = EnhancedMedicalAnalyzer('config_visualization.yaml')\nanalyzer.run_analysis()\n```\n\n## Usage Examples\n\n### Example 1: Complete Process Handling\n\n```bash\n# Process complete medical evaluation data\npython medical_evaluation_pipeline.py medical_data.xlsx \\\n -o results/final_report.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -v\n```\n\n### Example 2: Quick Testing\n\n```bash\n# Process only first 10 rows of data\npython medical_evaluation_pipeline.py test_data.xlsx \\\n --max_rows 10 \\\n --disable-data-analysis\n```\n\n### Example 3: Custom Column Names\n\n```bash\n# Use Chinese column names\npython medical_evaluation_pipeline.py data.xlsx \\\n --question-col \"Clinical Question\" \\\n --rubric-col \"Scoring Criteria\" \\\n --answer-columns \"GPT Response\" \"Claude Response\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **File Not Found Error**\n - Solution: Check if the input file path is correct\n\n2. **Column Name Mismatch**\n - Solution: Use `--question-col`, `--rubric-col` and other parameters to specify correct column names\n\n3. **Insufficient Memory**\n - Solution: Use `--max_rows` parameter to limit the number of rows processed\n\n4. **API Call Failure**\n - Solution: Check model configuration and network connection\n\n### Debug Mode\n\n```bash\n# Enable verbose logging to view processing details\npython medical_evaluation_pipeline.py input.xlsx -v\n\n# Keep temporary files for debugging\n# Modify the cleanup() call in the code\n```\n\n## Technical Documentation\n\n### Core Modules\n\n- `medical_evaluation_pipeline.py` - Main pipeline controller\n- `clinical_answer_judge.py` - Met/Not Met review module\n- `clinical_scoring_calculator.py` - Score calculation module\n- `enhanced_analyzer.py` - Data analysis and visualization module\n- `general_negative_points.py` - Irrelevant content processing module\n\n### File Structure\n\n```\nMedicalAiBenchEval/\n├── clinical_answer_judge.py # Multi-model review processor\n├── clinical_scoring_calculator.py # Score calculator\n├── medical_evaluation_pipeline.py # Main pipeline controller\n├── enhanced_analyzer.py # Data analyzer\n├── judge_engine.py # Review engine\n├── irrelevant_content_grading.py # Irrelevant content scoring\n├── extract_irrelevant_content.py # Irrelevant content extraction\n├── general_negative_points.py # Negative point processing\n├── config.yaml # Main configuration file\n├── config_model.yaml # Model configuration file\n├── config_visualization.yaml # Visualization configuration\n├── requirements.txt # Dependency list\n└── data/ # Data directory\n ├── input/ # Input files\n │ ├── GAPS-NSCLC-preview.xlsx # Thoracic surgery evaluation dataset (92 cases)\n │ └── community_contributions.xlsx # Community contributed cases\n └── output/ # Output files\n └── analysis/ # Analysis reports\n```\n\n### Data Flow\n\n1. **Input Phase**: Read Excel file, validate data format\n2. **Parallel Processing**: Simultaneously execute Met review and irrelevant content detection\n3. **Result Merging**: Intelligently merge outputs from both parallel steps\n4. **Score Calculation**: Calculate multi-dimensional scores based on merged results\n5. **Analysis Output**: Generate statistical reports and visualization charts\n\n### Performance Optimization\n\n- Asynchronous parallel processing improves efficiency\n- Batch processing reduces API call frequency\n- Intelligent retry mechanism handles network exceptions\n- Memory optimization handles large files\n\n## Contribution Guidelines\n\nWelcome to submit Issues and Pull Requests to improve system functionality.\n\n### Development Environment Setup\n\n```bash\ngit clone <repository>\ncd medical-ai-bench-eval\npip install -r requirements.txt\n```\n\n### Code Standards\n\n- Follow PEP 8 coding standards\n- Add detailed docstrings\n- Include unit tests\n- Update README documentation\n\n## Citation\n\nIf you use this system in your research, please cite our paper:\n\n**English:**\n> Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Jun Wang, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan Yang (2025). GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians. *arXiv preprint* arXiv:2510.13734. https://arxiv.org/abs/2510.13734\n\n```bibtex\n@article{chen2025gaps,\n title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},\n author={Chen, Xiuyuan and Sun, Tao and Su, Dexin and Yu, Ailing and Liu, Junwei and Chen, Zhe and Jin, Gangzeng and Wang, Xin and Liu, Jingnan and Xiao, Hansong and Zhou, Hualei and Tao, Dongjie and Guo, Chunxiao and Yang, Minghui and Xia, Yuan and Zhao, Jing and Fan, Qianrui and Wang, Yanyun and Zhen, Shuai and Chen, Kezhong and Wang, Jun and Sun, Zewen and Zhao, Heng and Guan, Tian and Wang, Shaodong and Chang, Geyun and Deng, Jiaming and Chen, Hongchengcheng and Feng, Kexin and Li, Ruzhen and Geng, Jiayi and Zhao, Changtai and Wang, Jun and Lin, Guihu and Li, Peihao and Liu, Liqi and Wei, Peng and Wang, Jian and Gu, Jinjie and Wang, Ping and Yang, Fan},\n journal={arXiv preprint arXiv:2510.13734},\n year={2025},\n url={https://arxiv.org/abs/2510.13734}\n}\n```\n\n## 🤝 Community Contribution / 共创呼吁\n\nWe welcome contributions from the medical AI community to expand and improve our evaluation dataset! Your expertise can help build a more comprehensive benchmark for medical AI systems.\n\n### 📋 How to Contribute\n\n#### Option 1: Submit an Issue\nCreate a new issue with the label `data-contribution` using the following template:\n\n```markdown\n**Medical Question**\n[Provide your clinical question here]\n\n**Evaluation Rubrics (JSON Format)**\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Your evaluation criterion description\",\n \"level\": \"A1|A2|A3|S1|S2|S3|S4\",\n \"type\": \"positive|negative\"\n },\n {\n \"id\": 2,\n \"claim\": \"Another evaluation criterion\",\n \"level\": \"A2\",\n \"type\": \"positive\"\n }\n]\n```\n\n**Medical Specialty**\n[e.g., Cardiology, Pulmonology, Oncology, etc.]\n\n**Question Type**\n[e.g., Diagnosis, Treatment, Risk Assessment, etc.]\n\n**Additional Context**\n[Any relevant clinical context or references]\n```\n\n#### Option 2: Submit a Pull Request\n1. Fork this repository\n2. Add your data to `data/input/community_contributions.xlsx` following the existing format:\n - **question**: Your medical question\n - **rubrics**: JSON array of evaluation criteria\n - **specialty**: Medical specialty area\n - **difficulty**: Easy/Medium/Hard\n - **source**: Your name/institution (optional)\n\n3. Ensure your contribution follows these guidelines:\n - **Clinical Relevance**: Questions should reflect real-world medical scenarios\n - **Evidence-Based**: Evaluation criteria should be based on established medical guidelines\n - **Clear Scoring**: Use our scoring system appropriately:\n - **Positive Points**: A1 (5 pts), A2 (3 pts), A3 (1 pt)\n - **Negative Points**: S1 (-1 pt), S2 (-2 pts), S3 (-3 pts), S4 (-4 pts)\n\n### 📝 Contribution Guidelines\n\n#### Evaluation Criteria Design\n- **A1 (Critical, 5 points)**: Essential medical knowledge that could impact patient safety\n- **A2 (Important, 3 points)**: Significant clinical considerations\n- **A3 (Helpful, 1 point)**: Additional relevant information\n- **S1 (Minor Error, -1 point)**: Small inaccuracies that don't affect core treatment\n- **S2 (Moderate Error, -2 points)**: Incorrect information that could mislead\n- **S3 (Major Error, -3 points)**: Serious medical errors\n- **S4 (Critical Error, -4 points)**: Dangerous misinformation that could harm patients\n\n#### Quality Standards\n- Questions should be **clinically realistic** and based on actual medical scenarios\n- Evaluation criteria must be **evidence-based** and reference current medical guidelines\n- Each question should have **3-8 evaluation points** for balanced assessment\n- Include both **positive criteria** (what should be mentioned) and **negative criteria** (what should be avoided)\n\n### 🎯 Priority Areas\nWe especially welcome contributions in:\n- **Rare Diseases**: Uncommon conditions that challenge AI systems\n- **Multidisciplinary Cases**: Complex cases requiring multiple specialties\n- **Cultural Considerations**: Cases reflecting diverse patient populations\n- **Emergency Medicine**: Time-critical decision-making scenarios\n- **Pediatric Medicine**: Age-specific medical considerations\n\n### 📊 Data Format Example\n```json\n{\n \"question\": \"A 65-year-old patient presents with chest pain and shortness of breath. How would you approach the initial evaluation?\",\n \"rubrics\": [\n {\n \"id\": 1,\n \"claim\": \"Should obtain detailed history including pain characteristics, timing, and associated symptoms\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"Must perform ECG and chest X-ray as initial diagnostic tests\",\n \"level\": \"A1\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Should not delay evaluation with unnecessary tests in acute presentation\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n ],\n \"specialty\": \"Emergency Medicine\",\n \"difficulty\": \"Medium\"\n}\n```\n\n### 🔄 Review Process\n1. **Community Review**: Contributions will be reviewed by medical professionals\n2. **Quality Check**: Ensure clinical accuracy and appropriate difficulty\n3. **Integration**: Approved contributions will be merged into the main dataset\n4. **Attribution**: Contributors will be acknowledged in our contributor list\n\n### 📞 Contact\nFor questions about contributing, please:\n- Open an issue with the `question` label\n- Join our discussion in the GitHub Discussions section\n\nThank you for helping build a better benchmark for medical AI evaluation! 🙏\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n"}, "cb3b07771a8be03709d80195f55bc8cafed3d61f": {"t": 1785766593.2522078, "n": 36441, "h": "51c5bb6375d4ca45", "c": "# Medical AI Evaluation Framework: Clinical Benchmark Dataset and Automated Assessment Pipeline\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n - [Installation](#installation)\n - [Basic Usage](#basic-usage)\n- [System Architecture](#system-architecture)\n- [Data Format](#data-format)\n - [Input Format](#input-format)\n - [Output Format](#output-format)\n- [Configuration](#configuration)\n - [Basic Configuration](#basic-configuration)\n - [Advanced Configuration](#advanced-configuration)\n- [Usage Guide](#usage-guide)\n - [Command Line Interface](#command-line-interface)\n - [Python API](#python-api)\n - [Examples](#examples)\n- [Evaluation Framework](#evaluation-framework)\n - [Scoring System](#scoring-system)\n - [Evaluation Metrics](#evaluation-metrics)\n- [Dataset](#dataset)\n - [Thoracic Surgery Dataset](#thoracic-surgery-dataset)\n - [Data Quality](#data-quality)\n- [Advanced Features](#advanced-features)\n - [Parallel Processing](#parallel-processing)\n - [Voting Strategies](#voting-strategies)\n - [Custom Models](#custom-models)\n- [Development](#development)\n - [Technical Documentation](#technical-documentation)\n - [API Reference](#api-reference)\n - [Troubleshooting](#troubleshooting)\n- [Community](#community)\n - [Contributing](#contributing)\n - [Citation](#citation)\n- [License](#license)\n\n## Overview\n\nThis Medical AI Evaluation Framework provides a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this framework includes both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.\n\nThe framework addresses the critical need for standardized evaluation of AI clinical decision-making by providing:\n- **Clinically Grounded Assessment**: Evaluation criteria based on real medical guidelines and expert knowledge\n- **Automated Pipeline**: Streamlined processing from raw responses to detailed performance metrics\n- **Multi-Model Support**: Simultaneous evaluation of multiple AI models with comparative analysis\n- **Scalable Architecture**: Efficient processing of large datasets with parallel execution capabilities\n\n## Key Features\n\n- **🏥 Medical-Specific Evaluation**: Specialized rubrics for clinical scenarios with positive/negative scoring\n- **🔄 Parallel Processing**: Simultaneous execution of Met/Not Met review and irrelevant content detection\n- **📊 Comprehensive Analytics**: Detailed statistical analysis with visualization reports\n- **🎯 Multi-Model Assessment**: Support for evaluating multiple AI models simultaneously\n- **⚙️ Flexible Configuration**: Customizable models, voting strategies, and evaluation parameters\n- **📈 Rich Visualization**: Automated generation of performance charts and comparative analysis\n- **🔧 Modular Design**: Independent modules for different evaluation stages\n- **📋 Standardized Output**: Consistent Excel-based reporting with detailed metrics\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd medical-ai-bench-eval\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Basic Usage\n\n```bash\n# Run complete evaluation pipeline\npython medical_evaluation_pipeline.py input_data.xlsx\n\n# With custom output file\npython medical_evaluation_pipeline.py input_data.xlsx -o results/evaluation_results.xlsx\n\n# Enable verbose logging\npython medical_evaluation_pipeline.py input_data.xlsx -v\n```\n\n## System Architecture\n\nThe system processes medical AI responses through a sophisticated pipeline that includes:\n\n1. **Input Processing**: Reads Excel files containing medical questions, evaluation rubrics, and AI model responses\n2. **Parallel Evaluation**: Simultaneously executes Met/Not Met review and irrelevant content detection\n3. **Intelligent Scoring**: Calculates comprehensive scores based on clinical evaluation criteria\n4. **Analysis & Reporting**: Generates detailed statistical reports and visualizations\n\n## System Architecture\n\n```\nInput Excel File\n↓\n┌─────────────────────────────────────────┐\n│ Parallel Processing Phase │\n├─────────────────┬─────────────────────┤\n│ Step 1: Met Review │ Step 2: Irrelevant Content │\n│ - Multi-model Review │ - Content Extraction │\n│ - Voting Decision │ - Level Assessment │\n│ - Result Summary │ - Voting Classification │\n└─────────────────┴─────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Result Merging │\n│ - Intelligent merging of parallel results │\n│ - Data integrity check │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 3: Score Calculation │\n│ - Multi-dimensional score statistics │\n│ - Irrelevant content deduction │\n│ - Normalization processing │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 4: Data Analysis & Visualization │\n│ - Statistical analysis reports │\n│ - Diverse charts │\n│ - CSV data export │\n└─────────────────────────────────────────┘\n↓\nOutput Excel File + Analysis Report\n```\n\n## Data Format\n\n### Input Format\n\nThe system accepts Excel files with the following structure:\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n#### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n### Output Format\n\nThe system generates multiple output files:\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n#### Scoring Metrics\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Configuration\n\n### Basic Configuration\n\n### Environment Requirements\n- **Python**: 3.12+\n- **Operating System**: Windows, macOS, Linux\n- **Core Dependencies**: pandas, numpy, matplotlib, seaborn, asyncio, langchain, openpyxl, xlsxwriter\n\n### Installation Steps\n\n1. **Clone the repository**:\n```bash\ngit clone <repository-url>\ncd medical-ai-bench-eval\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Configure API Keys** (if using external AI model services):\n\n ### API Configuration Guide\n\n #### Environment Variables Setup\n\n **Linux/Mac:**\n ```bash\n # OpenAI\n export OPENAI_API_KEY=\"sk-your-actual-key\"\n export OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n\n # Claude (Anthropic)\n export ANTHROPIC_API_KEY=\"sk-ant-your-actual-key\"\n export ANTHROPIC_BASE_URL=\"https://api.anthropic.com\"\n\n # Google Gemini\n export GOOGLE_API_KEY=\"your-google-api-key\"\n export GOOGLE_BASE_URL=\"https://generativelanguage.googleapis.com/v1beta\"\n\n # Moonshot Kimi\n export MOONSHOT_API_KEY=\"sk-your-moonshot-key\"\n export MOONSHOT_BASE_URL=\"https://api.moonshot.cn/v1\"\n\n # Alibaba Qwen\n export DASHSCOPE_API_KEY=\"your-dashscope-key\"\n export DASHSCOPE_BASE_URL=\"https://dashscope.aliyuncs.com/api/v1\"\n\n # Baichuan\n export BAICHUAN_API_KEY=\"your-baichuan-key\"\n export BAICHUAN_BASE_URL=\"https://api.baichuan-ai.com/v1\"\n\n # DeepSeek\n export DEEPSEEK_API_KEY=\"sk-your-deepseek-key\"\n export DEEPSEEK_BASE_URL=\"https://api.deepseek.com/v1\"\n\n # Zhipu ChatGLM\n export ZHIPU_API_KEY=\"your-zhipu-key\"\n export ZHIPU_BASE_URL=\"https://open.bigmodel.cn/api/paas/v4\"\n ```\n\n **Windows PowerShell:**\n ```powershell\n $Env:OPENAI_API_KEY=\"sk-your-actual-key\"\n $Env:OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n ```\n\n #### Model Configuration\n\n **⚠️ IMPORTANT: OpenAI-Compatible API Only**\n \n This system currently supports **ONLY OpenAI-compatible API interfaces**. All models must provide OpenAI-compatible endpoints, regardless of the actual provider.\n\n **🌐 Third-Party Multi-Model Platforms (Recommended)**\n \n For models that don't natively support OpenAI-compatible APIs (like Claude, Gemini), you can use third-party platforms that provide unified OpenAI-compatible access:\n \n - **[OpenRouter](https://openrouter.ai/)** - Access Claude, Gemini, GPT, and many other models via OpenAI-compatible API\n - **[Together AI](https://together.ai/)** - Various open-source and commercial models\n - **[Anyscale](https://anyscale.com/)** - Multiple LLMs via OpenAI-compatible endpoints\n\n Edit `config_model.yaml` to configure your preferred models:\n\n ```yaml\n models:\n m1: # OpenAI GPT-4 (Native)\n api_key_env: \"OPENAI_API_KEY\"\n api_base_env: \"OPENAI_BASE_URL\"\n model: \"gpt-4-turbo-preview\"\n temperature: 0.2\n max_tokens: 4000\n\n m2: # Kimi (OpenAI-Compatible)\n api_key_env: \"MOONSHOT_API_KEY\"\n api_base_env: \"MOONSHOT_BASE_URL\"\n model: \"kimi-latest\"\n temperature: 0.2\n max_tokens: 4000\n\n m3: # Qwen (OpenAI-Compatible)\n api_key_env: \"DASHSCOPE_API_KEY\"\n api_base_env: \"DASHSCOPE_BASE_URL\"\n model: \"qwen-max\"\n temperature: 0.1\n max_tokens: 4000\n\n m4: # DeepSeek (OpenAI-Compatible)\n api_key_env: \"DEEPSEEK_API_KEY\"\n api_base_env: \"DEEPSEEK_BASE_URL\"\n model: \"deepseek-chat\"\n temperature: 0.2\n max_tokens: 4000\n\n m5: # Local Model (via OpenAI-Compatible Server)\n api_key_env: \"LOCAL_API_KEY\"\n api_base_env: \"LOCAL_BASE_URL\"\n model: \"llama3-medical\"\n temperature: 0.2\n max_tokens: 4000\n\n # OpenRouter Examples (uncomment to use)\n # m6: # Claude via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"anthropic/claude-3-5-sonnet-20241022\"\n # temperature: 0.2\n # max_tokens: 4000\n\n # m7: # Gemini via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"google/gemini-pro-1.5\"\n # temperature: 0.1\n # max_tokens: 4000\n ```\n\n #### Supported vs Unsupported Providers\n\n | Provider | Status | API Type | Notes |\n |----------|--------|----------|-------|\n | **OpenAI** | ✅ Supported | Native OpenAI | Direct support |\n | **OpenRouter** | ✅ Supported | Multi-Model Platform | Access Claude, Gemini, GPT via OpenAI-compatible API |\n | **Together AI** | ✅ Supported | Multi-Model Platform | Various models via OpenAI-compatible API |\n | **Anyscale** | ✅ Supported | Multi-Model Platform | Multiple LLMs via OpenAI-compatible API |\n | **Kimi/Moonshot** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Qwen/DashScope** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **DeepSeek** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Baichuan** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Zhipu** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Claude/Anthropic** | ⚠️ Via OpenRouter | Native Anthropic API | Use OpenRouter for OpenAI-compatible access |\n | **Gemini/Google** | ⚠️ Via OpenRouter | Native Google API | Use OpenRouter for OpenAI-compatible access |\n\n #### Medical Model Recommendations (OpenAI-Compatible Only)\n\n | Use Case | Recommended Models | Temperature | Notes |\n |----------|-------------------|-------------|--------|\n | **Diagnostic Tasks** | gpt-4-turbo, deepseek-chat | 0.1-0.2 | High accuracy |\n | **Chinese Medical** | qwen-max, baichuan-medical | 0.2-0.3 | Better CJK support |\n | **Cost-effective** | deepseek-chat, kimi | 0.2-0.3 | Good balance |\n | **Real-time** | kimi, deepseek-chat | 0.1-0.2 | Fast response |\n | **Privacy-focused** | local-llama3-medical | 0.2-0.3 | On-premise via OpenAI-compatible server |\n\n #### Quick Model Setup Commands\n\n **For GPT-4 testing:**\n ```bash\n export OPENAI_API_KEY=\"your-key-here\"\n # Run with GPT-4\n python medical_evaluation_pipeline.py input.xlsx --judge-models m1\n ```\n\n **For Chinese medical models:**\n ```bash\n export DASHSCOPE_API_KEY=\"your-key-here\"\n # Run with Qwen\n python medical_evaluation_pipeline.py input.xlsx --judge-models m4\n ```\n\n **For mixed model evaluation:**\n ```bash\n export OPENAI_API_KEY=\"your-key\"\n export DASHSCOPE_API_KEY=\"your-key\"\n # Use GPT-4 and Qwen for evaluation\n python medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m4 \\\n --extract-model m5\n\n### Basic Usage\n\n1. **Prepare input data**:\n - Excel file format\n - Contains columns for questions, scoring criteria (JSON format), model responses, etc.\n - Default column names can be configured in `config.yaml`\n\n2. **Run the complete evaluation pipeline**:\n```bash\npython medical_evaluation_pipeline.py input.xlsx -o output.xlsx\n```\n\n3. **View results**:\n - Output Excel file contains detailed review results and scores\n - Data analysis reports are saved in the `data/output/analysis/` directory\n\n## ⚙️ System Configuration\n\n### Configuration File (config.yaml)\n\n```yaml\n# File path configuration\nfiles:\n input_file: \"data/medical_evaluation_result.xlsx\"\n output_file: \"data/output/processed_result.xlsx\"\n\n# Process control\npipeline:\n disable_met_nomet: false # Whether to disable Met review\n disable_irrelevant_extraction: false # Whether to disable irrelevant content detection\n exclude_irrelevant_scoring: false # Whether to exclude irrelevant content from scoring\n disable_data_analysis: false # Whether to disable data analysis\n analysis_config_file: \"config_visualization.yaml\"\n\n# Model configuration\nmodels:\n judge_models: [\"m1\", \"m2\", \"m3\"] # Met review models\n extract_model: \"m5\" # Irrelevant content extraction model\n grade_models: [\"m1\", \"m2\", \"m3\"] # Irrelevant content grading models\n voting_strategy: \"conservative\" # Voting strategy\n\n# Column name mapping\ncolumns:\n question_col: \"question\"\n rubric_col: \"final_merged_json\"\n answer_columns:\n - \"gpt_5_answer\"\n - \"gemini_2_5_pro_answer\"\n - \"claude_opus_4_answer\"\n\n# Scoring system\npoint:\n A1: 5\n A2: 3\n A3: 1\n S1: -1\n S2: -2\n S3: -3\n S4: -4\n```\n\n### Command Line Arguments\n\n```bash\n# Process control\n--disable-met-nomet # Skip Met review step\n--disable-irrelevant # Skip irrelevant content detection\n--exclude-irrelevant-scoring # Exclude irrelevant content deduction from scoring\n--disable-data-analysis # Skip data analysis\n\n# Model configuration\n--judge-models m1 m2 # Specify Met review models\n--extract-model m3 # Specify irrelevant content extraction model\n--grade-models m1 m2 m4 # Specify irrelevant content grading models\n--voting-strategy majority # Voting strategy: conservative/majority/average\n\n# Data column configuration\n--question-col \"Question\" # Question column name\n--rubric-col \"Evaluation Points\" # Evaluation points column name\n--answer-columns \"Response1\" \"Response2\" \"Response3\" # Model response column names\n```\n\n## 📊 Thoracic Surgery Evaluation Dataset\n\n### Dataset Overview\n\nWe provide a specialized **Thoracic Surgery Evaluation Dataset** (`data/input/specialty_evaluation_dataset.xlsx`) containing 92 carefully curated clinical cases for evaluating AI models' performance in thoracic surgery scenarios. This dataset focuses on complex clinical decision-making in chest surgery, particularly for non-small cell lung cancer (NSCLC) staging and treatment planning.\n\n### Dataset Structure\n\n| Column Name | Description | Data Type |\n|-------------|-------------|-----------|\n| **question** | Clinical questions covering thoracic surgery scenarios | Text |\n| **分类** (Category) | Medical specialty classification | Text |\n| **rubrics** | Evaluation criteria in JSON format with scoring levels | JSON Array |\n| **gpt_5_answer** | GPT-4 model responses to clinical questions | Text |\n| **gemini_2_5_pro_answer** | Gemini 2.5 Pro model responses | Text |\n| **claude_opus_4_answer** | Claude Opus model responses | Text |\n\n### Clinical Focus Areas\n\nThe dataset covers critical aspects of thoracic surgery, including:\n\n- **Pre-operative Evaluation**: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)\n- **Diagnostic Procedures**: EBUS-TBNA, mediastinoscopy, PET-CT interpretation\n- **Staging Assessment**: TNM staging, mediastinal lymph node evaluation\n- **Treatment Planning**: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions\n- **Risk Assessment**: Pulmonary function evaluation, cardiac risk stratification\n- **Molecular Diagnostics**: EGFR, ALK, PD-L1 testing strategies\n\n### Evaluation Rubrics Format\n\nEach question includes detailed evaluation criteria in JSON format:\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Should perform enhanced chest CT covering thoracic inlet to upper abdomen\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"PET-CT is essential for accurate N-staging and excluding distant metastases\",\n \"level\": \"A2\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Incorrectly generalizing upper sulcus tumor MRI requirements to all IIB/IIIA patients\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n]\n```\n\n**Scoring Levels**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Sample Clinical Question\n\n**Question**: \"For NSCLC patients with IIB-IIIA staging, what comprehensive pre-treatment evaluation should be performed?\"\n\n**Key Evaluation Points**:\n- Pathological diagnosis and molecular characterization\n- Accurate clinical staging (T, N, M assessment)\n- Invasive mediastinal staging for N2 evaluation\n- Pulmonary and cardiac function assessment\n- Multidisciplinary team (MDT) treatment planning\n\n### Usage with GAPS Framework\n\nThis dataset is designed to work seamlessly with the GAPS evaluation pipeline:\n\n```bash\n# Evaluate thoracic surgery dataset\npython medical_evaluation_pipeline.py data/input/specialty_evaluation_dataset.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -o results/thoracic_surgery_evaluation.xlsx\n```\n\n### Clinical Validation\n\nThe dataset has been validated by:\n- **Board-certified thoracic surgeons**\n- **Pulmonary oncologists** \n- **Medical imaging specialists**\n- **Pathologists specializing in lung cancer**\n\nAll evaluation criteria are based on current clinical guidelines including:\n- NCCN Guidelines for Non-Small Cell Lung Cancer\n- ESTS Guidelines for Intraoperative Lymph Node Staging\n- IASLC Staging Manual in Thoracic Oncology\n\n### Research Applications\n\nThis dataset enables research in:\n- **AI Clinical Decision Support**: Evaluating AI models' ability to provide accurate clinical recommendations\n- **Medical Education**: Training and assessment of clinical reasoning skills\n- **Quality Assurance**: Benchmarking AI systems against established clinical standards\n- **Comparative Analysis**: Cross-model performance evaluation in specialized medical domains\n\n### Data Quality Metrics\n\n- **Total Cases**: 92 clinical scenarios\n- **Completeness**: 100% data coverage across all columns\n- **Clinical Diversity**: Covers full spectrum of IIB-IIIA NSCLC presentations\n- **Expert Validation**: All cases reviewed by multidisciplinary clinical team\n\n---\n\n## Input Data Format\n\n### Excel File Structure\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n## Output Results\n\n### Main Output Files\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n### Scoring Metrics Description\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Advanced Features\n\n### 1. Parallel Processing Optimization\n\nThe system supports parallel execution of Step 1 (Met Review) and Step 2 (Irrelevant Content Detection):\n\n```bash\n# Enable all steps (auto parallel)\npython medical_evaluation_pipeline.py input.xlsx\n\n# Enable only Met Review\npython medical_evaluation_pipeline.py input.xlsx --disable-irrelevant\n\n# Enable only Irrelevant Content Detection\npython medical_evaluation_pipeline.py input.xlsx --disable-met-nomet\n```\n\n### 2. Voting Strategy\n\nSupports three voting strategies for processing multi-model results:\n- **conservative**: Conservative strategy, prioritizes more severe ratings\n- **majority**: Majority voting, selects the result with the most votes\n- **average**: Average strategy, calculates based on numerical average\n\n```bash\npython medical_evaluation_pipeline.py input.xlsx --voting-strategy conservative\n```\n\n### 3. Custom Model Configuration\n\n```bash\n# Use specific review model combinations\npython medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m2 \\\n --extract-model m5 \\\n --grade-models m1 m3 m4\n```\n\n### 4. Data Analysis Customization\n\nCreate `config_visualization.yaml` for custom analysis configuration:\n\n```yaml\nanalysis:\n input_file: \"data/processed_result.xlsx\"\n columns:\n model_scores:\n GPT-5: \"gpt_5_answer_judged_json_scores\"\n Gemini-2.5-Pro: \"gemini_2_5_pro_answer_judged_json_scores\"\n Claude-Opus-4: \"claude_opus_4_answer_judged_json_scores\"\n\nvisualization:\n charts:\n bar_overall: true\n boxplot_distribution: true\n heatmap_category: true\n radar_chart: true\n style:\n figure_size: [24, 16]\n dpi: 300\n```\n\n## 🔧 Independent Module Usage\n\n### 1. Run Met Review Separately\n\n```python\nimport asyncio\nfrom clinical_answer_judge import MultiModelMergedJsonJudge\n\nasync def run_judge():\n judge = MultiModelMergedJsonJudge(\n judge_models=['m1', 'm2', 'm3'],\n question_col='question',\n rubric_col='rubrics'\n )\n result = await judge.process_excel_file('input.xlsx', 'output.xlsx')\n return result\n\n# Run the async function\nresult = asyncio.run(run_judge())\n```\n\n### 2. Run Score Calculation Separately\n\n```python\nfrom clinical_scoring_calculator import process_excel_data\n\nresult = process_excel_data(\n input_file='judged_data.xlsx',\n output_file='scored_data.xlsx',\n model_columns=['gpt_5_answer_judged_json']\n)\n```\n\n### 3. Run Data Analysis Separately\n\n```python\nfrom enhanced_analyzer import EnhancedMedicalAnalyzer\n\nanalyzer = EnhancedMedicalAnalyzer('config_visualization.yaml')\nanalyzer.run_analysis()\n```\n\n## Usage Examples\n\n### Example 1: Complete Process Handling\n\n```bash\n# Process complete medical evaluation data\npython medical_evaluation_pipeline.py medical_data.xlsx \\\n -o results/final_report.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -v\n```\n\n### Example 2: Quick Testing\n\n```bash\n# Process only first 10 rows of data\npython medical_evaluation_pipeline.py test_data.xlsx \\\n --max_rows 10 \\\n --disable-data-analysis\n```\n\n### Example 3: Custom Column Names\n\n```bash\n# Use Chinese column names\npython medical_evaluation_pipeline.py data.xlsx \\\n --question-col \"Clinical Question\" \\\n --rubric-col \"Scoring Criteria\" \\\n --answer-columns \"GPT Response\" \"Claude Response\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **File Not Found Error**\n - Solution: Check if the input file path is correct\n\n2. **Column Name Mismatch**\n - Solution: Use `--question-col`, `--rubric-col` and other parameters to specify correct column names\n\n3. **Insufficient Memory**\n - Solution: Use `--max_rows` parameter to limit the number of rows processed\n\n4. **API Call Failure**\n - Solution: Check model configuration and network connection\n\n### Debug Mode\n\n```bash\n# Enable verbose logging to view processing details\npython medical_evaluation_pipeline.py input.xlsx -v\n\n# Keep temporary files for debugging\n# Modify the cleanup() call in the code\n```\n\n## Technical Documentation\n\n### Core Modules\n\n- `medical_evaluation_pipeline.py` - Main pipeline controller\n- `clinical_answer_judge.py` - Met/Not Met review module\n- `clinical_scoring_calculator.py` - Score calculation module\n- `enhanced_analyzer.py` - Data analysis and visualization module\n- `general_negative_points.py` - Irrelevant content processing module\n\n### File Structure\n\n```\nMedicalAiBenchEval/\n├── clinical_answer_judge.py # Multi-model review processor\n├── clinical_scoring_calculator.py # Score calculator\n├── medical_evaluation_pipeline.py # Main pipeline controller\n├── enhanced_analyzer.py # Data analyzer\n├── judge_engine.py # Review engine\n├── irrelevant_content_grading.py # Irrelevant content scoring\n├── extract_irrelevant_content.py # Irrelevant content extraction\n├── general_negative_points.py # Negative point processing\n├── config.yaml # Main configuration file\n├── config_model.yaml # Model configuration file\n├── config_visualization.yaml # Visualization configuration\n├── requirements.txt # Dependency list\n└── data/ # Data directory\n ├── input/ # Input files\n │ ├── specialty_evaluation_dataset.xlsx # Thoracic surgery evaluation dataset (92 cases)\n │ └── community_contributions.xlsx # Community contributed cases\n └── output/ # Output files\n └── analysis/ # Analysis reports\n```\n\n### Data Flow\n\n1. **Input Phase**: Read Excel file, validate data format\n2. **Parallel Processing**: Simultaneously execute Met review and irrelevant content detection\n3. **Result Merging**: Intelligently merge outputs from both parallel steps\n4. **Score Calculation**: Calculate multi-dimensional scores based on merged results\n5. **Analysis Output**: Generate statistical reports and visualization charts\n\n### Performance Optimization\n\n- Asynchronous parallel processing improves efficiency\n- Batch processing reduces API call frequency\n- Intelligent retry mechanism handles network exceptions\n- Memory optimization handles large files\n\n## Contribution Guidelines\n\nWelcome to submit Issues and Pull Requests to improve system functionality.\n\n### Development Environment Setup\n\n```bash\ngit clone <repository>\ncd medical-ai-bench-eval\npip install -r requirements.txt\n```\n\n### Code Standards\n\n- Follow PEP 8 coding standards\n- Add detailed docstrings\n- Include unit tests\n- Update README documentation\n\n## Citation\n\nIf you use this system in your research, please cite our paper:\n\n**English:**\n> Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Jun Wang, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan Yang (2025). GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians. *arXiv preprint* arXiv:2510.13734. https://arxiv.org/abs/2510.13734\n\n```bibtex\n@article{chen2025gaps,\n title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},\n author={Chen, Xiuyuan and Sun, Tao and Su, Dexin and Yu, Ailing and Liu, Junwei and Chen, Zhe and Jin, Gangzeng and Wang, Xin and Liu, Jingnan and Xiao, Hansong and Zhou, Hualei and Tao, Dongjie and Guo, Chunxiao and Yang, Minghui and Xia, Yuan and Zhao, Jing and Fan, Qianrui and Wang, Yanyun and Zhen, Shuai and Chen, Kezhong and Wang, Jun and Sun, Zewen and Zhao, Heng and Guan, Tian and Wang, Shaodong and Chang, Geyun and Deng, Jiaming and Chen, Hongchengcheng and Feng, Kexin and Li, Ruzhen and Geng, Jiayi and Zhao, Changtai and Wang, Jun and Lin, Guihu and Li, Peihao and Liu, Liqi and Wei, Peng and Wang, Jian and Gu, Jinjie and Wang, Ping and Yang, Fan},\n journal={arXiv preprint arXiv:2510.13734},\n year={2025},\n url={https://arxiv.org/abs/2510.13734}\n}\n```\n\n## 🤝 Community Contribution / 共创呼吁\n\nWe welcome contributions from the medical AI community to expand and improve our evaluation dataset! Your expertise can help build a more comprehensive benchmark for medical AI systems.\n\n### 📋 How to Contribute\n\n#### Option 1: Submit an Issue\nCreate a new issue with the label `data-contribution` using the following template:\n\n```markdown\n**Medical Question**\n[Provide your clinical question here]\n\n**Evaluation Rubrics (JSON Format)**\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Your evaluation criterion description\",\n \"level\": \"A1|A2|A3|S1|S2|S3|S4\",\n \"type\": \"positive|negative\"\n },\n {\n \"id\": 2,\n \"claim\": \"Another evaluation criterion\",\n \"level\": \"A2\",\n \"type\": \"positive\"\n }\n]\n```\n\n**Medical Specialty**\n[e.g., Cardiology, Pulmonology, Oncology, etc.]\n\n**Question Type**\n[e.g., Diagnosis, Treatment, Risk Assessment, etc.]\n\n**Additional Context**\n[Any relevant clinical context or references]\n```\n\n#### Option 2: Submit a Pull Request\n1. Fork this repository\n2. Add your data to `data/input/community_contributions.xlsx` following the existing format:\n - **question**: Your medical question\n - **rubrics**: JSON array of evaluation criteria\n - **specialty**: Medical specialty area\n - **difficulty**: Easy/Medium/Hard\n - **source**: Your name/institution (optional)\n\n3. Ensure your contribution follows these guidelines:\n - **Clinical Relevance**: Questions should reflect real-world medical scenarios\n - **Evidence-Based**: Evaluation criteria should be based on established medical guidelines\n - **Clear Scoring**: Use our scoring system appropriately:\n - **Positive Points**: A1 (5 pts), A2 (3 pts), A3 (1 pt)\n - **Negative Points**: S1 (-1 pt), S2 (-2 pts), S3 (-3 pts), S4 (-4 pts)\n\n### 📝 Contribution Guidelines\n\n#### Evaluation Criteria Design\n- **A1 (Critical, 5 points)**: Essential medical knowledge that could impact patient safety\n- **A2 (Important, 3 points)**: Significant clinical considerations\n- **A3 (Helpful, 1 point)**: Additional relevant information\n- **S1 (Minor Error, -1 point)**: Small inaccuracies that don't affect core treatment\n- **S2 (Moderate Error, -2 points)**: Incorrect information that could mislead\n- **S3 (Major Error, -3 points)**: Serious medical errors\n- **S4 (Critical Error, -4 points)**: Dangerous misinformation that could harm patients\n\n#### Quality Standards\n- Questions should be **clinically realistic** and based on actual medical scenarios\n- Evaluation criteria must be **evidence-based** and reference current medical guidelines\n- Each question should have **3-8 evaluation points** for balanced assessment\n- Include both **positive criteria** (what should be mentioned) and **negative criteria** (what should be avoided)\n\n### 🎯 Priority Areas\nWe especially welcome contributions in:\n- **Rare Diseases**: Uncommon conditions that challenge AI systems\n- **Multidisciplinary Cases**: Complex cases requiring multiple specialties\n- **Cultural Considerations**: Cases reflecting diverse patient populations\n- **Emergency Medicine**: Time-critical decision-making scenarios\n- **Pediatric Medicine**: Age-specific medical considerations\n\n### 📊 Data Format Example\n```json\n{\n \"question\": \"A 65-year-old patient presents with chest pain and shortness of breath. How would you approach the initial evaluation?\",\n \"rubrics\": [\n {\n \"id\": 1,\n \"claim\": \"Should obtain detailed history including pain characteristics, timing, and associated symptoms\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"Must perform ECG and chest X-ray as initial diagnostic tests\",\n \"level\": \"A1\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Should not delay evaluation with unnecessary tests in acute presentation\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n ],\n \"specialty\": \"Emergency Medicine\",\n \"difficulty\": \"Medium\"\n}\n```\n\n### 🔄 Review Process\n1. **Community Review**: Contributions will be reviewed by medical professionals\n2. **Quality Check**: Ensure clinical accuracy and appropriate difficulty\n3. **Integration**: Approved contributions will be merged into the main dataset\n4. **Attribution**: Contributors will be acknowledged in our contributor list\n\n### 📞 Contact\nFor questions about contributing, please:\n- Open an issue with the `question` label\n- Join our discussion in the GitHub Discussions section\n\nThank you for helping build a better benchmark for medical AI evaluation! 🙏\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n"}, "9b0e4d2d05ac0310a56680f966214cb7b646eeab": {"t": 1785766594.8506937, "n": 36444, "h": "2f89392f39234006", "c": "# Medical AI Evaluation Framework: Clinical Benchmark Dataset and Automated Assessment Pipeline\n\n[](https://opensource.org/licenses/MIT)\n[](https://www.python.org/downloads/)\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Key Features](#key-features)\n- [Quick Start](#quick-start)\n - [Installation](#installation)\n - [Basic Usage](#basic-usage)\n- [System Architecture](#system-architecture)\n- [Data Format](#data-format)\n - [Input Format](#input-format)\n - [Output Format](#output-format)\n- [Configuration](#configuration)\n - [Basic Configuration](#basic-configuration)\n - [Advanced Configuration](#advanced-configuration)\n- [Usage Guide](#usage-guide)\n - [Command Line Interface](#command-line-interface)\n - [Python API](#python-api)\n - [Examples](#examples)\n- [Evaluation Framework](#evaluation-framework)\n - [Scoring System](#scoring-system)\n - [Evaluation Metrics](#evaluation-metrics)\n- [Dataset](#dataset)\n - [Thoracic Surgery Dataset](#thoracic-surgery-dataset)\n - [Data Quality](#data-quality)\n- [Advanced Features](#advanced-features)\n - [Parallel Processing](#parallel-processing)\n - [Voting Strategies](#voting-strategies)\n - [Custom Models](#custom-models)\n- [Development](#development)\n - [Technical Documentation](#technical-documentation)\n - [API Reference](#api-reference)\n - [Troubleshooting](#troubleshooting)\n- [Community](#community)\n - [Contributing](#contributing)\n - [Citation](#citation)\n- [License](#license)\n\n## Overview\n\nThis Medical AI Evaluation Framework provides a comprehensive evaluation system designed specifically for assessing AI models in clinical scenarios. Based on the GAPS (Grounded, Automated, Personalized, Scalable) methodology, this framework includes both a curated clinical benchmark dataset and an automated assessment pipeline for medical AI systems.\n\nThe framework addresses the critical need for standardized evaluation of AI clinical decision-making by providing:\n- **Clinically Grounded Assessment**: Evaluation criteria based on real medical guidelines and expert knowledge\n- **Automated Pipeline**: Streamlined processing from raw responses to detailed performance metrics\n- **Multi-Model Support**: Simultaneous evaluation of multiple AI models with comparative analysis\n- **Scalable Architecture**: Efficient processing of large datasets with parallel execution capabilities\n\n## Key Features\n\n- **🏥 Medical-Specific Evaluation**: Specialized rubrics for clinical scenarios with positive/negative scoring\n- **🔄 Parallel Processing**: Simultaneous execution of Met/Not Met review and irrelevant content detection\n- **📊 Comprehensive Analytics**: Detailed statistical analysis with visualization reports\n- **🎯 Multi-Model Assessment**: Support for evaluating multiple AI models simultaneously\n- **⚙️ Flexible Configuration**: Customizable models, voting strategies, and evaluation parameters\n- **📈 Rich Visualization**: Automated generation of performance charts and comparative analysis\n- **🔧 Modular Design**: Independent modules for different evaluation stages\n- **📋 Standardized Output**: Consistent Excel-based reporting with detailed metrics\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone <repository-url>\ncd medical-ai-bench-eval\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### Basic Usage\n\n```bash\n# Run complete evaluation pipeline\npython medical_evaluation_pipeline.py input_data.xlsx\n\n# With custom output file\npython medical_evaluation_pipeline.py input_data.xlsx -o results/evaluation_results.xlsx\n\n# Enable verbose logging\npython medical_evaluation_pipeline.py input_data.xlsx -v\n```\n\n## System Architecture\n\nThe system processes medical AI responses through a sophisticated pipeline that includes:\n\n1. **Input Processing**: Reads Excel files containing medical questions, evaluation rubrics, and AI model responses\n2. **Parallel Evaluation**: Simultaneously executes Met/Not Met review and irrelevant content detection\n3. **Intelligent Scoring**: Calculates comprehensive scores based on clinical evaluation criteria\n4. **Analysis & Reporting**: Generates detailed statistical reports and visualizations\n\n## System Architecture\n\n```\nInput Excel File\n↓\n┌─────────────────────────────────────────┐\n│ Parallel Processing Phase │\n├─────────────────┬─────────────────────┤\n│ Step 1: Met Review │ Step 2: Irrelevant Content │\n│ - Multi-model Review │ - Content Extraction │\n│ - Voting Decision │ - Level Assessment │\n│ - Result Summary │ - Voting Classification │\n└─────────────────┴─────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Result Merging │\n│ - Intelligent merging of parallel results │\n│ - Data integrity check │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 3: Score Calculation │\n│ - Multi-dimensional score statistics │\n│ - Irrelevant content deduction │\n│ - Normalization processing │\n└─────────────────────────────────────────┘\n↓\n┌─────────────────────────────────────────┐\n│ Step 4: Data Analysis & Visualization │\n│ - Statistical analysis reports │\n│ - Diverse charts │\n│ - CSV data export │\n└─────────────────────────────────────────┘\n↓\nOutput Excel File + Analysis Report\n```\n\n## Data Format\n\n### Input Format\n\nThe system accepts Excel files with the following structure:\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n#### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n### Output Format\n\nThe system generates multiple output files:\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n#### Scoring Metrics\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Configuration\n\n### Basic Configuration\n\n### Environment Requirements\n- **Python**: 3.12+\n- **Operating System**: Windows, macOS, Linux\n- **Core Dependencies**: pandas, numpy, matplotlib, seaborn, asyncio, langchain, openpyxl, xlsxwriter\n\n### Installation Steps\n\n1. **Clone the repository**:\n```bash\ngit clone <repository-url>\ncd medical-ai-bench-eval\n```\n\n2. **Install dependencies**:\n```bash\npip install -r requirements.txt\n```\n\n3. **Configure API Keys** (if using external AI model services):\n\n ### API Configuration Guide\n\n #### Environment Variables Setup\n\n **Linux/Mac:**\n ```bash\n # OpenAI\n export OPENAI_API_KEY=\"sk-your-actual-key\"\n export OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n\n # Claude (Anthropic)\n export ANTHROPIC_API_KEY=\"sk-ant-your-actual-key\"\n export ANTHROPIC_BASE_URL=\"https://api.anthropic.com\"\n\n # Google Gemini\n export GOOGLE_API_KEY=\"your-google-api-key\"\n export GOOGLE_BASE_URL=\"https://generativelanguage.googleapis.com/v1beta\"\n\n # Moonshot Kimi\n export MOONSHOT_API_KEY=\"sk-your-moonshot-key\"\n export MOONSHOT_BASE_URL=\"https://api.moonshot.cn/v1\"\n\n # Alibaba Qwen\n export DASHSCOPE_API_KEY=\"your-dashscope-key\"\n export DASHSCOPE_BASE_URL=\"https://dashscope.aliyuncs.com/api/v1\"\n\n # Baichuan\n export BAICHUAN_API_KEY=\"your-baichuan-key\"\n export BAICHUAN_BASE_URL=\"https://api.baichuan-ai.com/v1\"\n\n # DeepSeek\n export DEEPSEEK_API_KEY=\"sk-your-deepseek-key\"\n export DEEPSEEK_BASE_URL=\"https://api.deepseek.com/v1\"\n\n # Zhipu ChatGLM\n export ZHIPU_API_KEY=\"your-zhipu-key\"\n export ZHIPU_BASE_URL=\"https://open.bigmodel.cn/api/paas/v4\"\n ```\n\n **Windows PowerShell:**\n ```powershell\n $Env:OPENAI_API_KEY=\"sk-your-actual-key\"\n $Env:OPENAI_BASE_URL=\"https://api.openai.com/v1\"\n ```\n\n #### Model Configuration\n\n **⚠️ IMPORTANT: OpenAI-Compatible API Only**\n \n This system currently supports **ONLY OpenAI-compatible API interfaces**. All models must provide OpenAI-compatible endpoints, regardless of the actual provider.\n\n **🌐 Third-Party Multi-Model Platforms (Recommended)**\n \n For models that don't natively support OpenAI-compatible APIs (like Claude, Gemini), you can use third-party platforms that provide unified OpenAI-compatible access:\n \n - **[OpenRouter](https://openrouter.ai/)** - Access Claude, Gemini, GPT, and many other models via OpenAI-compatible API\n - **[Together AI](https://together.ai/)** - Various open-source and commercial models\n - **[Anyscale](https://anyscale.com/)** - Multiple LLMs via OpenAI-compatible endpoints\n\n Edit `config_model.yaml` to configure your preferred models:\n\n ```yaml\n models:\n m1: # OpenAI GPT-4 (Native)\n api_key_env: \"OPENAI_API_KEY\"\n api_base_env: \"OPENAI_BASE_URL\"\n model: \"gpt-4-turbo-preview\"\n temperature: 0.2\n max_tokens: 4000\n\n m2: # Kimi (OpenAI-Compatible)\n api_key_env: \"MOONSHOT_API_KEY\"\n api_base_env: \"MOONSHOT_BASE_URL\"\n model: \"kimi-latest\"\n temperature: 0.2\n max_tokens: 4000\n\n m3: # Qwen (OpenAI-Compatible)\n api_key_env: \"DASHSCOPE_API_KEY\"\n api_base_env: \"DASHSCOPE_BASE_URL\"\n model: \"qwen-max\"\n temperature: 0.1\n max_tokens: 4000\n\n m4: # DeepSeek (OpenAI-Compatible)\n api_key_env: \"DEEPSEEK_API_KEY\"\n api_base_env: \"DEEPSEEK_BASE_URL\"\n model: \"deepseek-chat\"\n temperature: 0.2\n max_tokens: 4000\n\n m5: # Local Model (via OpenAI-Compatible Server)\n api_key_env: \"LOCAL_API_KEY\"\n api_base_env: \"LOCAL_BASE_URL\"\n model: \"llama3-medical\"\n temperature: 0.2\n max_tokens: 4000\n\n # OpenRouter Examples (uncomment to use)\n # m6: # Claude via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"anthropic/claude-3-5-sonnet-20241022\"\n # temperature: 0.2\n # max_tokens: 4000\n\n # m7: # Gemini via OpenRouter\n # api_key_env: \"OPENROUTER_API_KEY\"\n # api_base_env: \"OPENROUTER_BASE_URL\" # https://openrouter.ai/api/v1\n # model: \"google/gemini-pro-1.5\"\n # temperature: 0.1\n # max_tokens: 4000\n ```\n\n #### Supported vs Unsupported Providers\n\n | Provider | Status | API Type | Notes |\n |----------|--------|----------|-------|\n | **OpenAI** | ✅ Supported | Native OpenAI | Direct support |\n | **OpenRouter** | ✅ Supported | Multi-Model Platform | Access Claude, Gemini, GPT via OpenAI-compatible API |\n | **Together AI** | ✅ Supported | Multi-Model Platform | Various models via OpenAI-compatible API |\n | **Anyscale** | ✅ Supported | Multi-Model Platform | Multiple LLMs via OpenAI-compatible API |\n | **Kimi/Moonshot** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Qwen/DashScope** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **DeepSeek** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Baichuan** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Zhipu** | ✅ Supported | OpenAI-Compatible | Via compatible endpoint |\n | **Claude/Anthropic** | ⚠️ Via OpenRouter | Native Anthropic API | Use OpenRouter for OpenAI-compatible access |\n | **Gemini/Google** | ⚠️ Via OpenRouter | Native Google API | Use OpenRouter for OpenAI-compatible access |\n\n #### Medical Model Recommendations (OpenAI-Compatible Only)\n\n | Use Case | Recommended Models | Temperature | Notes |\n |----------|-------------------|-------------|--------|\n | **Diagnostic Tasks** | gpt-4-turbo, deepseek-chat | 0.1-0.2 | High accuracy |\n | **Chinese Medical** | qwen-max, baichuan-medical | 0.2-0.3 | Better CJK support |\n | **Cost-effective** | deepseek-chat, kimi | 0.2-0.3 | Good balance |\n | **Real-time** | kimi, deepseek-chat | 0.1-0.2 | Fast response |\n | **Privacy-focused** | local-llama3-medical | 0.2-0.3 | On-premise via OpenAI-compatible server |\n\n #### Quick Model Setup Commands\n\n **For GPT-4 testing:**\n ```bash\n export OPENAI_API_KEY=\"your-key-here\"\n # Run with GPT-4\n python medical_evaluation_pipeline.py input.xlsx --judge-models m1\n ```\n\n **For Chinese medical models:**\n ```bash\n export DASHSCOPE_API_KEY=\"your-key-here\"\n # Run with Qwen\n python medical_evaluation_pipeline.py input.xlsx --judge-models m4\n ```\n\n **For mixed model evaluation:**\n ```bash\n export OPENAI_API_KEY=\"your-key\"\n export DASHSCOPE_API_KEY=\"your-key\"\n # Use GPT-4 and Qwen for evaluation\n python medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m4 \\\n --extract-model m5\n\n### Basic Usage\n\n1. **Prepare input data**:\n - Excel file format\n - Contains columns for questions, scoring criteria (JSON format), model responses, etc.\n - Default column names can be configured in `config.yaml`\n\n2. **Run the complete evaluation pipeline**:\n```bash\npython medical_evaluation_pipeline.py input.xlsx -o output.xlsx\n```\n\n3. **View results**:\n - Output Excel file contains detailed review results and scores\n - Data analysis reports are saved in the `data/output/analysis/` directory\n\n## ⚙️ System Configuration\n\n### Configuration File (config.yaml)\n\n```yaml\n# File path configuration\nfiles:\n input_file: \"data/medical_evaluation_result.xlsx\"\n output_file: \"data/output/processed_result.xlsx\"\n\n# Process control\npipeline:\n disable_met_nomet: false # Whether to disable Met review\n disable_irrelevant_extraction: false # Whether to disable irrelevant content detection\n exclude_irrelevant_scoring: false # Whether to exclude irrelevant content from scoring\n disable_data_analysis: false # Whether to disable data analysis\n analysis_config_file: \"config_visualization.yaml\"\n\n# Model configuration\nmodels:\n judge_models: [\"m1\", \"m2\", \"m3\"] # Met review models\n extract_model: \"m5\" # Irrelevant content extraction model\n grade_models: [\"m1\", \"m2\", \"m3\"] # Irrelevant content grading models\n voting_strategy: \"conservative\" # Voting strategy\n\n# Column name mapping\ncolumns:\n question_col: \"question\"\n rubric_col: \"final_merged_json\"\n answer_columns:\n - \"gpt_5_answer\"\n - \"gemini_2_5_pro_answer\"\n - \"claude_opus_4_answer\"\n\n# Scoring system\npoint:\n A1: 5\n A2: 3\n A3: 1\n S1: -1\n S2: -2\n S3: -3\n S4: -4\n```\n\n### Command Line Arguments\n\n```bash\n# Process control\n--disable-met-nomet # Skip Met review step\n--disable-irrelevant # Skip irrelevant content detection\n--exclude-irrelevant-scoring # Exclude irrelevant content deduction from scoring\n--disable-data-analysis # Skip data analysis\n\n# Model configuration\n--judge-models m1 m2 # Specify Met review models\n--extract-model m3 # Specify irrelevant content extraction model\n--grade-models m1 m2 m4 # Specify irrelevant content grading models\n--voting-strategy majority # Voting strategy: conservative/majority/average\n\n# Data column configuration\n--question-col \"Question\" # Question column name\n--rubric-col \"Evaluation Points\" # Evaluation points column name\n--answer-columns \"Response1\" \"Response2\" \"Response3\" # Model response column names\n```\n\n## 📊 Thoracic Surgery Evaluation Dataset\n\n### Dataset Overview\n\nWe provide a specialized **Thoracic Surgery Evaluation Dataset** (`data/input/specialty_evaluation_dataset.xlsx`) containing 92 carefully curated clinical cases for evaluating AI models' performance in thoracic surgery scenarios. This dataset focuses on complex clinical decision-making in chest surgery, particularly for non-small cell lung cancer (NSCLC) staging and treatment planning.\n\n### Dataset Structure\n\n| Column Name | Description | Data Type |\n|-------------|-------------|-----------|\n| **question** | Clinical questions covering thoracic surgery scenarios | Text |\n| **分类** (Category) | Medical specialty classification | Text |\n| **rubrics** | Evaluation criteria in JSON format with scoring levels | JSON Array |\n| **gpt_5_answer** | GPT-4 model responses to clinical questions | Text |\n| **gemini_2_5_pro_answer** | Gemini 2.5 Pro model responses | Text |\n| **claude_opus_4_answer** | Claude Opus model responses | Text |\n\n### Clinical Focus Areas\n\nThe dataset covers critical aspects of thoracic surgery, including:\n\n- **Pre-operative Evaluation**: Comprehensive assessment protocols for NSCLC patients (IIB-IIIA staging)\n- **Diagnostic Procedures**: EBUS-TBNA, mediastinoscopy, PET-CT interpretation\n- **Staging Assessment**: TNM staging, mediastinal lymph node evaluation\n- **Treatment Planning**: Surgical vs. non-surgical approaches, neoadjuvant therapy decisions\n- **Risk Assessment**: Pulmonary function evaluation, cardiac risk stratification\n- **Molecular Diagnostics**: EGFR, ALK, PD-L1 testing strategies\n\n### Evaluation Rubrics Format\n\nEach question includes detailed evaluation criteria in JSON format:\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Should perform enhanced chest CT covering thoracic inlet to upper abdomen\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"PET-CT is essential for accurate N-staging and excluding distant metastases\",\n \"level\": \"A2\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Incorrectly generalizing upper sulcus tumor MRI requirements to all IIB/IIIA patients\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n]\n```\n\n**Scoring Levels**:\n- **A1 (5 points)**: Critical medical knowledge affecting patient safety\n- **A2 (3 points)**: Important clinical considerations \n- **A3 (1 point)**: Additional relevant information\n- **S1 (-1 point)**: Minor inaccuracies not affecting core treatment\n- **S2 (-2 points)**: Incorrect information that could mislead\n- **S3 (-3 points)**: Serious medical errors\n- **S4 (-4 points)**: Dangerous misinformation that could harm patients\n\n### Sample Clinical Question\n\n**Question**: \"For NSCLC patients with IIB-IIIA staging, what comprehensive pre-treatment evaluation should be performed?\"\n\n**Key Evaluation Points**:\n- Pathological diagnosis and molecular characterization\n- Accurate clinical staging (T, N, M assessment)\n- Invasive mediastinal staging for N2 evaluation\n- Pulmonary and cardiac function assessment\n- Multidisciplinary team (MDT) treatment planning\n\n### Usage with GAPS Framework\n\nThis dataset is designed to work seamlessly with the GAPS evaluation pipeline:\n\n```bash\n# Evaluate thoracic surgery dataset\npython medical_evaluation_pipeline.py data/input/specialty_evaluation_dataset.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -o results/thoracic_surgery_evaluation.xlsx\n```\n\n### Clinical Validation\n\nThe dataset has been validated by:\n- **Board-certified thoracic surgeons**\n- **Pulmonary oncologists** \n- **Medical imaging specialists**\n- **Pathologists specializing in lung cancer**\n\nAll evaluation criteria are based on current clinical guidelines including:\n- NCCN Guidelines for Non-Small Cell Lung Cancer\n- ESTS Guidelines for Intraoperative Lymph Node Staging\n- IASLC Staging Manual in Thoracic Oncology\n\n### Research Applications\n\nThis dataset enables research in:\n- **AI Clinical Decision Support**: Evaluating AI models' ability to provide accurate clinical recommendations\n- **Medical Education**: Training and assessment of clinical reasoning skills\n- **Quality Assurance**: Benchmarking AI systems against established clinical standards\n- **Comparative Analysis**: Cross-model performance evaluation in specialized medical domains\n\n### Data Quality Metrics\n\n- **Total Cases**: 92 clinical scenarios\n- **Completeness**: 100% data coverage across all columns\n- **Clinical Diversity**: Covers full spectrum of IIB-IIIA NSCLC presentations\n- **Expert Validation**: All cases reviewed by multidisciplinary clinical team\n\n---\n\n## Input Data Format\n\n### Excel File Structure\n\n| Column Name | Description | Example |\n|-------------|-------------|---------|\n| question | Medical question | \"Patient presents with chest pain symptoms, how to diagnose?\" |\n| final_merged_json | Evaluation points JSON | [{\"id\":1,\"claim\":\"Need to inquire about symptoms\",\"level\":\"A1\"}] |\n| gpt_5_answer | GPT model response | \"First need to ask the patient about symptoms in detail...\" |\n| gemini_2_5_pro_answer | Gemini model response | \"Recommend performing ECG examination...\" |\n| claude_opus_4_answer | Claude model response | \"Should consider acute coronary syndrome...\" |\n\n### Evaluation Points JSON Format\n\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Need to ask about symptom duration\",\n \"level\": \"A1\",\n \"desc\": \"Detailed inquiry about chest pain duration, nature, etc.\"\n },\n {\n \"id\": 2,\n \"claim\": \"Should avoid mentioning unrelated treatment plans\",\n \"level\": \"S2\",\n \"desc\": \"Should not mention treatments unrelated to chest pain\"\n }\n]\n```\n\n**Level Description**:\n- **A1-A3**: Positive points (A1=5 points, A2=3 points, A3=1 point)\n- **S1-S4**: Negative points (S1=-1 point, S2=-2 points, S3=-3 points, S4=-4 points)\n\n## Output Results\n\n### Main Output Files\n\n1. **Final Score Excel** (`processed_result_final_YYYYMMDD_HHMMSS.xlsx`)\n - Contains all original data\n - Met/Not Met review results\n - Irrelevant content detection results\n - Detailed scoring statistics\n\n2. **Data Analysis Report** (`data/output/analysis/`)\n - `medical_evaluation_report_YYYYMMDD_HHMMSS.png` - Visualization charts\n - `medical_analysis_report_YYYYMMDD_HHMMSS.txt` - Detailed analysis report\n - `model_performance_summary_YYYYMMDD_HHMMSS.csv` - Performance summary\n\n### Scoring Metrics Description\n\n| Metric | Description |\n|--------|-------------|\n| max_possible | Theoretical maximum score (sum of all positive points) |\n| final_total_score | Actual score (Met items score - irrelevant content deduction) |\n| normalized | Normalized score (between 0-1) |\n| positive_total_count | Number of positive point hits |\n| rubric_total_count | Number of negative point hits |\n| irrelevant_total_count | Total irrelevant content count |\n\n## Advanced Features\n\n### 1. Parallel Processing Optimization\n\nThe system supports parallel execution of Step 1 (Met Review) and Step 2 (Irrelevant Content Detection):\n\n```bash\n# Enable all steps (auto parallel)\npython medical_evaluation_pipeline.py input.xlsx\n\n# Enable only Met Review\npython medical_evaluation_pipeline.py input.xlsx --disable-irrelevant\n\n# Enable only Irrelevant Content Detection\npython medical_evaluation_pipeline.py input.xlsx --disable-met-nomet\n```\n\n### 2. Voting Strategy\n\nSupports three voting strategies for processing multi-model results:\n- **conservative**: Conservative strategy, prioritizes more severe ratings\n- **majority**: Majority voting, selects the result with the most votes\n- **average**: Average strategy, calculates based on numerical average\n\n```bash\npython medical_evaluation_pipeline.py input.xlsx --voting-strategy conservative\n```\n\n### 3. Custom Model Configuration\n\n```bash\n# Use specific review model combinations\npython medical_evaluation_pipeline.py input.xlsx \\\n --judge-models m1 m2 \\\n --extract-model m5 \\\n --grade-models m1 m3 m4\n```\n\n### 4. Data Analysis Customization\n\nCreate `config_visualization.yaml` for custom analysis configuration:\n\n```yaml\nanalysis:\n input_file: \"data/processed_result.xlsx\"\n columns:\n model_scores:\n GPT-5: \"gpt_5_answer_judged_json_scores\"\n Gemini-2.5-Pro: \"gemini_2_5_pro_answer_judged_json_scores\"\n Claude-Opus-4: \"claude_opus_4_answer_judged_json_scores\"\n\nvisualization:\n charts:\n bar_overall: true\n boxplot_distribution: true\n heatmap_category: true\n radar_chart: true\n style:\n figure_size: [24, 16]\n dpi: 300\n```\n\n## 🔧 Independent Module Usage\n\n### 1. Run Met Review Separately\n\n```python\nimport asyncio\nfrom clinical_answer_judge import MultiModelMergedJsonJudge\n\nasync def run_judge():\n judge = MultiModelMergedJsonJudge(\n judge_models=['m1', 'm2', 'm3'],\n question_col='question',\n rubric_col='rubrics'\n )\n result = await judge.process_excel_file('input.xlsx', 'output.xlsx')\n return result\n\n# Run the async function\nresult = asyncio.run(run_judge())\n```\n\n### 2. Run Score Calculation Separately\n\n```python\nfrom clinical_scoring_calculator import process_excel_data\n\nresult = process_excel_data(\n input_file='judged_data.xlsx',\n output_file='scored_data.xlsx',\n model_columns=['gpt_5_answer_judged_json']\n)\n```\n\n### 3. Run Data Analysis Separately\n\n```python\nfrom enhanced_analyzer import EnhancedMedicalAnalyzer\n\nanalyzer = EnhancedMedicalAnalyzer('config_visualization.yaml')\nanalyzer.run_analysis()\n```\n\n## Usage Examples\n\n### Example 1: Complete Process Handling\n\n```bash\n# Process complete medical evaluation data\npython medical_evaluation_pipeline.py medical_data.xlsx \\\n -o results/final_report.xlsx \\\n --judge-models m1 m2 m3 \\\n --voting-strategy conservative \\\n -v\n```\n\n### Example 2: Quick Testing\n\n```bash\n# Process only first 10 rows of data\npython medical_evaluation_pipeline.py test_data.xlsx \\\n --max_rows 10 \\\n --disable-data-analysis\n```\n\n### Example 3: Custom Column Names\n\n```bash\n# Use Chinese column names\npython medical_evaluation_pipeline.py data.xlsx \\\n --question-col \"Clinical Question\" \\\n --rubric-col \"Scoring Criteria\" \\\n --answer-columns \"GPT Response\" \"Claude Response\"\n```\n\n## Troubleshooting\n\n### Common Issues\n\n1. **File Not Found Error**\n - Solution: Check if the input file path is correct\n\n2. **Column Name Mismatch**\n - Solution: Use `--question-col`, `--rubric-col` and other parameters to specify correct column names\n\n3. **Insufficient Memory**\n - Solution: Use `--max_rows` parameter to limit the number of rows processed\n\n4. **API Call Failure**\n - Solution: Check model configuration and network connection\n\n### Debug Mode\n\n```bash\n# Enable verbose logging to view processing details\npython medical_evaluation_pipeline.py input.xlsx -v\n\n# Keep temporary files for debugging\n# Modify the cleanup() call in the code\n```\n\n## Technical Documentation\n\n### Core Modules\n\n- `medical_evaluation_pipeline.py` - Main pipeline controller\n- `clinical_answer_judge.py` - Met/Not Met review module\n- `clinical_scoring_calculator.py` - Score calculation module\n- `enhanced_analyzer.py` - Data analysis and visualization module\n- `general_negative_points.py` - Irrelevant content processing module\n\n### File Structure\n\n```\nmedical-ai-bench-eval/\n├── clinical_answer_judge.py # Multi-model review processor\n├── clinical_scoring_calculator.py # Score calculator\n├── medical_evaluation_pipeline.py # Main pipeline controller\n├── enhanced_analyzer.py # Data analyzer\n├── judge_engine.py # Review engine\n├── irrelevant_content_grading.py # Irrelevant content scoring\n├── extract_irrelevant_content.py # Irrelevant content extraction\n├── general_negative_points.py # Negative point processing\n├── config.yaml # Main configuration file\n├── config_model.yaml # Model configuration file\n├── config_visualization.yaml # Visualization configuration\n├── requirements.txt # Dependency list\n└── data/ # Data directory\n ├── input/ # Input files\n │ ├── specialty_evaluation_dataset.xlsx # Thoracic surgery evaluation dataset (92 cases)\n │ └── community_contributions.xlsx # Community contributed cases\n └── output/ # Output files\n └── analysis/ # Analysis reports\n```\n\n### Data Flow\n\n1. **Input Phase**: Read Excel file, validate data format\n2. **Parallel Processing**: Simultaneously execute Met review and irrelevant content detection\n3. **Result Merging**: Intelligently merge outputs from both parallel steps\n4. **Score Calculation**: Calculate multi-dimensional scores based on merged results\n5. **Analysis Output**: Generate statistical reports and visualization charts\n\n### Performance Optimization\n\n- Asynchronous parallel processing improves efficiency\n- Batch processing reduces API call frequency\n- Intelligent retry mechanism handles network exceptions\n- Memory optimization handles large files\n\n## Contribution Guidelines\n\nWelcome to submit Issues and Pull Requests to improve system functionality.\n\n### Development Environment Setup\n\n```bash\ngit clone <repository>\ncd medical-ai-bench-eval\npip install -r requirements.txt\n```\n\n### Code Standards\n\n- Follow PEP 8 coding standards\n- Add detailed docstrings\n- Include unit tests\n- Update README documentation\n\n## Citation\n\nIf you use this system in your research, please cite our paper:\n\n**English:**\n> Xiuyuan Chen, Tao Sun, Dexin Su, Ailing Yu, Junwei Liu, Zhe Chen, Gangzeng Jin, Xin Wang, Jingnan Liu, Hansong Xiao, Hualei Zhou, Dongjie Tao, Chunxiao Guo, Minghui Yang, Yuan Xia, Jing Zhao, Qianrui Fan, Yanyun Wang, Shuai Zhen, Kezhong Chen, Jun Wang, Zewen Sun, Heng Zhao, Tian Guan, Shaodong Wang, Geyun Chang, Jiaming Deng, Hongchengcheng Chen, Kexin Feng, Ruzhen Li, Jiayi Geng, Changtai Zhao, Jun Wang, Guihu Lin, Peihao Li, Liqi Liu, Peng Wei, Jian Wang, Jinjie Gu, Ping Wang, Fan Yang (2025). GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians. *arXiv preprint* arXiv:2510.13734. https://arxiv.org/abs/2510.13734\n\n```bibtex\n@article{chen2025gaps,\n title={GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians},\n author={Chen, Xiuyuan and Sun, Tao and Su, Dexin and Yu, Ailing and Liu, Junwei and Chen, Zhe and Jin, Gangzeng and Wang, Xin and Liu, Jingnan and Xiao, Hansong and Zhou, Hualei and Tao, Dongjie and Guo, Chunxiao and Yang, Minghui and Xia, Yuan and Zhao, Jing and Fan, Qianrui and Wang, Yanyun and Zhen, Shuai and Chen, Kezhong and Wang, Jun and Sun, Zewen and Zhao, Heng and Guan, Tian and Wang, Shaodong and Chang, Geyun and Deng, Jiaming and Chen, Hongchengcheng and Feng, Kexin and Li, Ruzhen and Geng, Jiayi and Zhao, Changtai and Wang, Jun and Lin, Guihu and Li, Peihao and Liu, Liqi and Wei, Peng and Wang, Jian and Gu, Jinjie and Wang, Ping and Yang, Fan},\n journal={arXiv preprint arXiv:2510.13734},\n year={2025},\n url={https://arxiv.org/abs/2510.13734}\n}\n```\n\n## 🤝 Community Contribution / 共创呼吁\n\nWe welcome contributions from the medical AI community to expand and improve our evaluation dataset! Your expertise can help build a more comprehensive benchmark for medical AI systems.\n\n### 📋 How to Contribute\n\n#### Option 1: Submit an Issue\nCreate a new issue with the label `data-contribution` using the following template:\n\n```markdown\n**Medical Question**\n[Provide your clinical question here]\n\n**Evaluation Rubrics (JSON Format)**\n```json\n[\n {\n \"id\": 1,\n \"claim\": \"Your evaluation criterion description\",\n \"level\": \"A1|A2|A3|S1|S2|S3|S4\",\n \"type\": \"positive|negative\"\n },\n {\n \"id\": 2,\n \"claim\": \"Another evaluation criterion\",\n \"level\": \"A2\",\n \"type\": \"positive\"\n }\n]\n```\n\n**Medical Specialty**\n[e.g., Cardiology, Pulmonology, Oncology, etc.]\n\n**Question Type**\n[e.g., Diagnosis, Treatment, Risk Assessment, etc.]\n\n**Additional Context**\n[Any relevant clinical context or references]\n```\n\n#### Option 2: Submit a Pull Request\n1. Fork this repository\n2. Add your data to `data/input/community_contributions.xlsx` following the existing format:\n - **question**: Your medical question\n - **rubrics**: JSON array of evaluation criteria\n - **specialty**: Medical specialty area\n - **difficulty**: Easy/Medium/Hard\n - **source**: Your name/institution (optional)\n\n3. Ensure your contribution follows these guidelines:\n - **Clinical Relevance**: Questions should reflect real-world medical scenarios\n - **Evidence-Based**: Evaluation criteria should be based on established medical guidelines\n - **Clear Scoring**: Use our scoring system appropriately:\n - **Positive Points**: A1 (5 pts), A2 (3 pts), A3 (1 pt)\n - **Negative Points**: S1 (-1 pt), S2 (-2 pts), S3 (-3 pts), S4 (-4 pts)\n\n### 📝 Contribution Guidelines\n\n#### Evaluation Criteria Design\n- **A1 (Critical, 5 points)**: Essential medical knowledge that could impact patient safety\n- **A2 (Important, 3 points)**: Significant clinical considerations\n- **A3 (Helpful, 1 point)**: Additional relevant information\n- **S1 (Minor Error, -1 point)**: Small inaccuracies that don't affect core treatment\n- **S2 (Moderate Error, -2 points)**: Incorrect information that could mislead\n- **S3 (Major Error, -3 points)**: Serious medical errors\n- **S4 (Critical Error, -4 points)**: Dangerous misinformation that could harm patients\n\n#### Quality Standards\n- Questions should be **clinically realistic** and based on actual medical scenarios\n- Evaluation criteria must be **evidence-based** and reference current medical guidelines\n- Each question should have **3-8 evaluation points** for balanced assessment\n- Include both **positive criteria** (what should be mentioned) and **negative criteria** (what should be avoided)\n\n### 🎯 Priority Areas\nWe especially welcome contributions in:\n- **Rare Diseases**: Uncommon conditions that challenge AI systems\n- **Multidisciplinary Cases**: Complex cases requiring multiple specialties\n- **Cultural Considerations**: Cases reflecting diverse patient populations\n- **Emergency Medicine**: Time-critical decision-making scenarios\n- **Pediatric Medicine**: Age-specific medical considerations\n\n### 📊 Data Format Example\n```json\n{\n \"question\": \"A 65-year-old patient presents with chest pain and shortness of breath. How would you approach the initial evaluation?\",\n \"rubrics\": [\n {\n \"id\": 1,\n \"claim\": \"Should obtain detailed history including pain characteristics, timing, and associated symptoms\",\n \"level\": \"A1\",\n \"type\": \"positive\"\n },\n {\n \"id\": 2,\n \"claim\": \"Must perform ECG and chest X-ray as initial diagnostic tests\",\n \"level\": \"A1\", \n \"type\": \"positive\"\n },\n {\n \"id\": 3,\n \"claim\": \"Should not delay evaluation with unnecessary tests in acute presentation\",\n \"level\": \"S2\",\n \"type\": \"negative\"\n }\n ],\n \"specialty\": \"Emergency Medicine\",\n \"difficulty\": \"Medium\"\n}\n```\n\n### 🔄 Review Process\n1. **Community Review**: Contributions will be reviewed by medical professionals\n2. **Quality Check**: Ensure clinical accuracy and appropriate difficulty\n3. **Integration**: Approved contributions will be merged into the main dataset\n4. **Attribution**: Contributors will be acknowledged in our contributor list\n\n### 📞 Contact\nFor questions about contributing, please:\n- Open an issue with the `question` label\n- Join our discussion in the GitHub Discussions section\n\nThank you for helping build a better benchmark for medical AI evaluation! 🙏\n\n---\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n"}}}} |