{"repo": "abhijeetnmishra/agents", "files": {"1_foundations/community_contributions/stevek_2_lab2_python/README.md": {"1dab7188ba062275fa37cdf3b817077a81faf883": {"t": 1786068580.8866353, "n": 4008, "h": "5c86783e4f5bdced", "c": "# Multi-Model Evaluator (2_lab2.py)\n\nA Python script that evaluates and compares the performance of multiple AI language models by generating a challenging question, collecting responses from various providers, and ranking them using a judge model.\n\n## Overview\n\nThis script performs the following steps:\n1. **Question Generation**: Uses an OpenAI model to generate a challenging, real-world question\n2. **Multi-Model Evaluation**: Sends the question to multiple AI models from different providers\n3. **Response Collection**: Gathers and displays all responses with timing information\n4. **Judging**: Uses a judge model to rank the responses based on correctness, depth, clarity, and helpfulness\n\n## Prerequisites\n\n- Python 3.7 or higher\n- API keys for the AI providers you want to test (at minimum, OpenAI API key is required)\n\n## Installation\n\n1. **Install required Python packages:**\n\n```bash\npip install openai anthropic python-dotenv\n```\n\nOr if you have a requirements file:\n\n```bash\npip install -r requirements.txt\n```\n\nRequired packages:\n- `openai` - For OpenAI API calls and OpenAI-compatible APIs\n- `anthropic` - For Anthropic/Claude API calls\n- `python-dotenv` - For loading environment variables from `.env` file\n\n## Environment Setup\n\n1. **Create a `.env` file** in the same directory as `2_lab2.py` (or in the project root)\n\n2. **Add your API keys** to the `.env` file:\n\n```env\n# Required\nOPENAI_API_KEY=your_openai_api_key_here\n\n# Optional (add only if you want to test these providers)\nANTHROPIC_API_KEY=your_anthropic_api_key_here\nGOOGLE_API_KEY=your_google_api_key_here\nDEEPSEEK_API_KEY=your_deepseek_api_key_here\nGROQ_API_KEY=your_groq_api_key_here\nOLLAMA_BASE_URL=http://localhost:11434\n```\n\n**Note:** Only `OPENAI_API_KEY` is strictly required. The script will skip providers for which API keys are missing.\n\n## Supported Models\n\nThe script is configured to test the following models (you can modify the `COMPETITORS` list in the script):\n\n- **Claude Sonnet 4.5** (Anthropic) - Requires `ANTHROPIC_API_KEY`\n- **GPT-5 Nano** (OpenAI) - Requires `OPENAI_API_KEY`\n- **Gemini 2.0 Flash** (Google) - Requires `GOOGLE_API_KEY`\n- **Llama 3.2** (via Ollama) - Requires `OLLAMA_BASE_URL` pointing to local Ollama instance\n- **DeepSeek Chat** (DeepSeek) - Requires `DEEPSEEK_API_KEY`\n- **GPT-OSS-120B** (via Groq) - Requires `GROQ_API_KEY`\n\n## Usage\n\n1. **Ensure your `.env` file is set up** with at least the `OPENAI_API_KEY`\n\n2. **Run the script:**\n\n```bash\npython 2_lab2.py\n```\n\nThe script will:\n- Generate a challenging question\n- Display the question\n- Query each configured model (skipping those without API keys)\n- Display each response with timing information\n- Use a judge model to rank all responses\n- Display the final rankings with scores and justifications\n\n## Customization\n\nYou can customize the script by modifying:\n\n- **`QUESTION_GENERATOR_MODEL`** (line 167): The model used to generate questions (default: `\"gpt-4.1-mini\"`)\n- **`JUDGE_MODEL`** (line 319): The model used to judge responses (default: `\"o3-mini\"`)\n- **`COMPETITORS`** list (lines 196-227): Add, remove, or modify the models to test\n\n## Notes\n\n- Models without corresponding API keys will be skipped gracefully\n- The script uses OpenAI's Responses API for some models and standard Chat Completions API for others\n- Ollama requires a local instance running and accessible at the `OLLAMA_BASE_URL`\n- Response times are measured and displayed for each model\n- The judge model outputs JSON-formatted rankings with scores (0-10) and justifications\n\n## Troubleshooting\n\n- **\"OPENAI_API_KEY is required\"**: Make sure your `.env` file contains a valid OpenAI API key\n- **\"ANTHROPIC_API_KEY missing\"**: This is expected if you don't have an Anthropic key. The script will skip Anthropic models\n- **Ollama connection errors**: Ensure Ollama is running locally and accessible at the configured `OLLAMA_BASE_URL`\n- **Import errors**: Make sure all required packages are installed: `pip install openai anthropic python-dotenv`\n\n"}}}}