{"repo": "chapoteaud/homepage", "files": {"examples/deepseek-r1-vs-openai-o1/README.md": {"dc7356aa32a7e48a8390a68998bda342c2006be5": {"t": 1785747281.2810893, "n": 3760, "h": "712a8b71b474d0f7", "c": "# DeepSeek-R1 vs OpenAI o1 Comparison\n\nThis example demonstrates how to benchmark DeepSeek's R1 model against OpenAI's o1 model using the Massive Multitask Language Understanding (MMLU) benchmark, focusing on reasoning-heavy subjects.\n\n## Prerequisites\n\n- promptfoo CLI installed (`npm install -g promptfoo` or `brew install promptfoo`)\n- OpenAI API key set as `OPENAI_API_KEY`\n- DeepSeek API key set as `DEEPSEEK_API_KEY`\n- Hugging Face account and access token (for MMLU dataset)\n\n## Hugging Face Authentication\n\nTo access the MMLU dataset, you'll need to authenticate with Hugging Face:\n\n1. Create a Hugging Face account at [huggingface.co](https://huggingface.co) if you don't have one\n2. Generate an access token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)\n3. Set your token as an environment variable:\n\n   ```bash\n   export HF_TOKEN=your_token_here\n   ```\n\n   Or add it to your `.env` file:\n\n   ```env\n   HF_TOKEN=your_token_here\n   ```\n\n## Running the Eval\n\n1. Get a local copy of the promptfooconfig.\n\n   You can clone this repository and from the root directory run:\n\n   ```bash\n   cd examples/deepseek-r1-vs-openai-o1\n   ```\n\n   or you can get the example with:\n\n   ```bash\n   promptfoo init --example deepseek-r1-vs-openai-o1\n   ```\n\n2. Run the evaluation:\n\n   ```bash\n   promptfoo eval\n   ```\n\n3. View the results in a web interface:\n\n   ```bash\n   promptfoo view\n   ```\n\n## What's Being Tested\n\nThis comparison evaluates both models on reasoning tasks from the MMLU benchmark, specifically:\n\n1. **Abstract Algebra**: Advanced mathematical reasoning\n2. **Formal Logic**: Logical statement analysis\n3. **High School Mathematics**: Core problem-solving\n4. **College Mathematics**: Advanced mathematical concepts\n5. **Logical Fallacies**: Flaw identification in reasoning\n\nEach subject uses 10 questions to keep the test manageable. You can edit this in `promptfooconfig.yaml`.\n\n## Test Structure\n\nThe configuration in `promptfooconfig.yaml`:\n\n1. **Prompt Template**: Encourages step-by-step reasoning for multiple choice questions\n2. **Quality Checks**:\n   - 60-second timeout per question\n   - Required step-by-step reasoning\n   - Clear final answer format\n3. **Evaluation Metrics**:\n   - Accuracy\n   - Reasoning quality\n   - Response time\n   - Format adherence\n\n## Customizing\n\nYou can modify the test by editing `promptfooconfig.yaml`:\n\n1. Add more MMLU subjects:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics\n   ```\n\n2. Try different prompting strategies:\n\n   ```yaml\n   prompts:\n     # Zero-shot with step-by-step reasoning (default)\n     - |\n       You are an expert test taker. Please solve the following multiple choice question step by step.\n\n       Question: {{question}}\n\n       Options:\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Think through this step by step, then provide your final answer in the format \"Therefore, the answer is A/B/C/D.\"\n\n     # Zero-shot with direct answer\n     - |\n       Question: {{question}}\n\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Answer with just the letter (A/B/C/D) of the correct option.\n   ```\n\n3. Change the number of questions:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics&limit=20 # Test 20 questions per subject\n   ```\n\n4. Adjust quality requirements:\n\n   ```yaml\n   defaultTest:\n     assert:\n       - type: latency\n         threshold: 30000 # Stricter 30-second timeout\n   ```\n\n## Additional Resources\n\n- [DeepSeek provider documentation](https://promptfoo.dev/docs/providers/deepseek)\n- [MMLU benchmark](https://huggingface.co/datasets/cais/mmlu)\n"}, "7ef3b75451af98c7d6b848817c487020d0fe178d": {"t": 1785772120.1802475, "n": 3760, "h": "712a8b71b474d0f7", "c": "# DeepSeek-R1 vs OpenAI o1 Comparison\n\nThis example demonstrates how to benchmark DeepSeek's R1 model against OpenAI's o1 model using the Massive Multitask Language Understanding (MMLU) benchmark, focusing on reasoning-heavy subjects.\n\n## Prerequisites\n\n- promptfoo CLI installed (`npm install -g promptfoo` or `brew install promptfoo`)\n- OpenAI API key set as `OPENAI_API_KEY`\n- DeepSeek API key set as `DEEPSEEK_API_KEY`\n- Hugging Face account and access token (for MMLU dataset)\n\n## Hugging Face Authentication\n\nTo access the MMLU dataset, you'll need to authenticate with Hugging Face:\n\n1. Create a Hugging Face account at [huggingface.co](https://huggingface.co) if you don't have one\n2. Generate an access token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)\n3. Set your token as an environment variable:\n\n   ```bash\n   export HF_TOKEN=your_token_here\n   ```\n\n   Or add it to your `.env` file:\n\n   ```env\n   HF_TOKEN=your_token_here\n   ```\n\n## Running the Eval\n\n1. Get a local copy of the promptfooconfig.\n\n   You can clone this repository and from the root directory run:\n\n   ```bash\n   cd examples/deepseek-r1-vs-openai-o1\n   ```\n\n   or you can get the example with:\n\n   ```bash\n   promptfoo init --example deepseek-r1-vs-openai-o1\n   ```\n\n2. Run the evaluation:\n\n   ```bash\n   promptfoo eval\n   ```\n\n3. View the results in a web interface:\n\n   ```bash\n   promptfoo view\n   ```\n\n## What's Being Tested\n\nThis comparison evaluates both models on reasoning tasks from the MMLU benchmark, specifically:\n\n1. **Abstract Algebra**: Advanced mathematical reasoning\n2. **Formal Logic**: Logical statement analysis\n3. **High School Mathematics**: Core problem-solving\n4. **College Mathematics**: Advanced mathematical concepts\n5. **Logical Fallacies**: Flaw identification in reasoning\n\nEach subject uses 10 questions to keep the test manageable. You can edit this in `promptfooconfig.yaml`.\n\n## Test Structure\n\nThe configuration in `promptfooconfig.yaml`:\n\n1. **Prompt Template**: Encourages step-by-step reasoning for multiple choice questions\n2. **Quality Checks**:\n   - 60-second timeout per question\n   - Required step-by-step reasoning\n   - Clear final answer format\n3. **Evaluation Metrics**:\n   - Accuracy\n   - Reasoning quality\n   - Response time\n   - Format adherence\n\n## Customizing\n\nYou can modify the test by editing `promptfooconfig.yaml`:\n\n1. Add more MMLU subjects:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics\n   ```\n\n2. Try different prompting strategies:\n\n   ```yaml\n   prompts:\n     # Zero-shot with step-by-step reasoning (default)\n     - |\n       You are an expert test taker. Please solve the following multiple choice question step by step.\n\n       Question: {{question}}\n\n       Options:\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Think through this step by step, then provide your final answer in the format \"Therefore, the answer is A/B/C/D.\"\n\n     # Zero-shot with direct answer\n     - |\n       Question: {{question}}\n\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Answer with just the letter (A/B/C/D) of the correct option.\n   ```\n\n3. Change the number of questions:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics&limit=20 # Test 20 questions per subject\n   ```\n\n4. Adjust quality requirements:\n\n   ```yaml\n   defaultTest:\n     assert:\n       - type: latency\n         threshold: 30000 # Stricter 30-second timeout\n   ```\n\n## Additional Resources\n\n- [DeepSeek provider documentation](https://promptfoo.dev/docs/providers/deepseek)\n- [MMLU benchmark](https://huggingface.co/datasets/cais/mmlu)\n"}, "8b210e60ed737d28a7d0afb82258a4a96b91d5c2": {"t": 1785772121.396051, "n": 3788, "h": "446853755c23ed0b", "c": "# DeepSeek-R1 vs OpenAI o1 Comparison\n\nThis example demonstrates how to benchmark DeepSeek's R1 model against OpenAI's o1 model using the Massive Multitask Language Understanding (MMLU) benchmark, focusing on reasoning-heavy subjects.\n\n## Prerequisites\n\n- promptfoo CLI installed (`npm install -g promptfoo` or `brew install promptfoo`)\n- OpenAI API key set as `OPENAI_API_KEY`\n- DeepSeek API key set as `DEEPSEEK_API_KEY`\n- Hugging Face account and access token (for MMLU dataset)\n\n## Hugging Face Authentication\n\nTo access the MMLU dataset, you'll need to authenticate with Hugging Face:\n\n1. Create a Hugging Face account at [huggingface.co](https://huggingface.co) if you don't have one\n2. Generate an access token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)\n3. Set your token as an environment variable:\n\n   ```bash\n   export HUGGING_FACE_HUB_TOKEN=your_token_here\n   ```\n\n   Or add it to your `.env` file:\n\n   ```env\n   HUGGING_FACE_HUB_TOKEN=your_token_here\n   ```\n\n## Running the Eval\n\n1. Get a local copy of the promptfooconfig.\n\n   You can clone this repository and from the root directory run:\n\n   ```bash\n   cd examples/deepseek-r1-vs-openai-o1\n   ```\n\n   or you can get the example with:\n\n   ```bash\n   promptfoo init --example deepseek-r1-vs-openai-o1\n   ```\n\n2. Run the evaluation:\n\n   ```bash\n   promptfoo eval\n   ```\n\n3. View the results in a web interface:\n\n   ```bash\n   promptfoo view\n   ```\n\n## What's Being Tested\n\nThis comparison evaluates both models on reasoning tasks from the MMLU benchmark, specifically:\n\n1. **Abstract Algebra**: Advanced mathematical reasoning\n2. **Formal Logic**: Logical statement analysis\n3. **High School Mathematics**: Core problem-solving\n4. **College Mathematics**: Advanced mathematical concepts\n5. **Logical Fallacies**: Flaw identification in reasoning\n\nEach subject uses 10 questions to keep the test manageable. You can edit this in `promptfooconfig.yaml`.\n\n## Test Structure\n\nThe configuration in `promptfooconfig.yaml`:\n\n1. **Prompt Template**: Encourages step-by-step reasoning for multiple choice questions\n2. **Quality Checks**:\n   - 60-second timeout per question\n   - Required step-by-step reasoning\n   - Clear final answer format\n3. **Evaluation Metrics**:\n   - Accuracy\n   - Reasoning quality\n   - Response time\n   - Format adherence\n\n## Customizing\n\nYou can modify the test by editing `promptfooconfig.yaml`:\n\n1. Add more MMLU subjects:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics\n   ```\n\n2. Try different prompting strategies:\n\n   ```yaml\n   prompts:\n     # Zero-shot with step-by-step reasoning (default)\n     - |\n       You are an expert test taker. Please solve the following multiple choice question step by step.\n\n       Question: {{question}}\n\n       Options:\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Think through this step by step, then provide your final answer in the format \"Therefore, the answer is A/B/C/D.\"\n\n     # Zero-shot with direct answer\n     - |\n       Question: {{question}}\n\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Answer with just the letter (A/B/C/D) of the correct option.\n   ```\n\n3. Change the number of questions:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics&limit=20 # Test 20 questions per subject\n   ```\n\n4. Adjust quality requirements:\n\n   ```yaml\n   defaultTest:\n     assert:\n       - type: latency\n         threshold: 30000 # Stricter 30-second timeout\n   ```\n\n## Additional Resources\n\n- [DeepSeek provider documentation](https://promptfoo.dev/docs/providers/deepseek)\n- [MMLU benchmark](https://huggingface.co/datasets/cais/mmlu)\n"}, "d51b2ad00e2515e1bc68617ddabc2f56909f5cac": {"t": 1785772123.0428581, "n": 3730, "h": "63964c793175d9c7", "c": "# DeepSeek-R1 vs OpenAI o1 Comparison\n\nThis example demonstrates how to benchmark DeepSeek's R1 model against OpenAI's o1 model using the Massive Multitask Language Understanding (MMLU) benchmark, focusing on reasoning-heavy subjects.\n\n## Prerequisites\n\n- promptfoo CLI installed (`npm install -g promptfoo` or `brew install promptfoo`)\n- OpenAI API key set as `OPENAI_API_KEY`\n- DeepSeek API key set as `DEEPSEEK_API_KEY`\n- Hugging Face account and access token (for MMLU dataset)\n\n## Hugging Face Authentication\n\nTo access the MMLU dataset, you'll need to authenticate with Hugging Face:\n\n1. Create a Hugging Face account at [huggingface.co](https://huggingface.co) if you don't have one\n2. Generate an access token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens)\n3. Set your token as an environment variable:\n\n   ```sh\n   export HUGGING_FACE_HUB_TOKEN=your_token_here\n   ```\n\n   Or add it to your `.env` file:\n\n   ```\n   HUGGING_FACE_HUB_TOKEN=your_token_here\n   ```\n\n## Running the Eval\n\n1. Get a local copy of the promptfooconfig.\n\nYou can clone this repository and from the root directory run:\n\n```sh\ncd examples/deepseek-r1-vs-openai-o1\n```\n\nor you can get the example with:\n\n```sh\npromptfoo init --example deepseek-r1-vs-openai-o1\n```\n\n2. Run the evaluation:\n\n   ```sh\n   promptfoo eval\n   ```\n\n3. View the results in a web interface:\n\n   ```sh\n   promptfoo view\n   ```\n\n## What's Being Tested\n\nThis comparison evaluates both models on reasoning tasks from the MMLU benchmark, specifically:\n\n1. **Abstract Algebra**: Advanced mathematical reasoning\n2. **Formal Logic**: Logical statement analysis\n3. **High School Mathematics**: Core problem-solving\n4. **College Mathematics**: Advanced mathematical concepts\n5. **Logical Fallacies**: Flaw identification in reasoning\n\nEach subject uses 10 questions to keep the test manageable. You can edit this in `promptfooconfig.yaml`.\n\n## Test Structure\n\nThe configuration in `promptfooconfig.yaml`:\n\n1. **Prompt Template**: Encourages step-by-step reasoning for multiple choice questions\n2. **Quality Checks**:\n   - 60-second timeout per question\n   - Required step-by-step reasoning\n   - Clear final answer format\n3. **Evaluation Metrics**:\n   - Accuracy\n   - Reasoning quality\n   - Response time\n   - Format adherence\n\n## Customizing\n\nYou can modify the test by editing `promptfooconfig.yaml`:\n\n1. Add more MMLU subjects:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics\n   ```\n\n2. Try different prompting strategies:\n\n   ```yaml\n   prompts:\n     # Zero-shot with step-by-step reasoning (default)\n     - |\n       You are an expert test taker. Please solve the following multiple choice question step by step.\n\n       Question: {{question}}\n\n       Options:\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Think through this step by step, then provide your final answer in the format \"Therefore, the answer is A/B/C/D.\"\n\n     # Zero-shot with direct answer\n     - |\n       Question: {{question}}\n\n       A) {{choices[0]}}\n       B) {{choices[1]}}\n       C) {{choices[2]}}\n       D) {{choices[3]}}\n\n       Answer with just the letter (A/B/C/D) of the correct option.\n   ```\n\n3. Change the number of questions:\n\n   ```yaml\n   tests:\n     - huggingface://datasets/cais/mmlu?split=test&subset=physics&limit=20 # Test 20 questions per subject\n   ```\n\n4. Adjust quality requirements:\n\n   ```yaml\n   defaultTest:\n     assert:\n       - type: latency\n         threshold: 30000 # Stricter 30-second timeout\n   ```\n\n## Additional Resources\n\n- [DeepSeek provider documentation](/docs/providers/deepseek)\n- [MMLU benchmark](https://huggingface.co/datasets/cais/mmlu)\n"}}}}