1 line
38 KiB
JSON
1 line
38 KiB
JSON
{"repo": "areid987/gptme", "files": {"docs/evals.rst": {"c92f35c3782680824cbc69384233e19ac44a139d": {"t": 1785746504.3627412, "n": 2911, "h": "cd64f9f3061aeb22", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created an evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still tiny and under development, but the eval harness is fully functional.\n\nRecommended Model\n-----------------\n\nThe recommended model is **Claude Sonnet 4** (``anthropic/claude-sonnet-4-20250514`` and ``openrouter/anthropic/claude-sonnet-4``) for its:\n\n- Strong agentic capabilities\n- Strong coder capabilities\n- Strong performance across all tool types and formats\n- Reasoning capabilities\n- Vision & computer use capabilities\n\nDecent alternatives include:\n\n- GPT-4o (``openai/gpt-4o``)\n- Llama 3.1 405B (``openrouter/meta-llama/llama-3.1-405b-instruct``)\n- DeepSeek V3 (``deepseek/deepseek-chat``)\n- DeepSeek R1 (``deepseek/deepseek-reasoner``)\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with Claude 3.7 Sonnet like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model anthropic/claude-sonnet-4-20250514\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"ANTHROPIC_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model anthropic/claude-sonnet-4-20250514\n\nAvailable Evals\n---------------\n\nThe current evaluations test basic tool use in gptme, such as the ability to: read, write, patch files; run code in ipython, commands in the shell; use git and create new projects with npm and cargo. It also has basic tests for web browsing and data extraction.\n\n.. This is where we want to get to:\n\n The evaluation suite tests models on:\n\n 1. Tool Usage\n\n - Shell commands and file operations\n - Git operations\n - Web browsing and data extraction\n - Project navigation and understanding\n\n 2. Programming Tasks\n\n - Code completion and generation\n - Bug fixing and debugging\n - Documentation writing\n - Test creation\n\n 3. Reasoning\n\n - Multi-step problem solving\n - Tool selection and sequencing\n - Error handling and recovery\n - Self-correction\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/gptme/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "9cf990576a2ee5c673818f1d6d3744c27876021c": {"t": 1785765042.8304374, "n": 2911, "h": "cd64f9f3061aeb22", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created an evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still tiny and under development, but the eval harness is fully functional.\n\nRecommended Model\n-----------------\n\nThe recommended model is **Claude Sonnet 4** (``anthropic/claude-sonnet-4-20250514`` and ``openrouter/anthropic/claude-sonnet-4``) for its:\n\n- Strong agentic capabilities\n- Strong coder capabilities\n- Strong performance across all tool types and formats\n- Reasoning capabilities\n- Vision & computer use capabilities\n\nDecent alternatives include:\n\n- GPT-4o (``openai/gpt-4o``)\n- Llama 3.1 405B (``openrouter/meta-llama/llama-3.1-405b-instruct``)\n- DeepSeek V3 (``deepseek/deepseek-chat``)\n- DeepSeek R1 (``deepseek/deepseek-reasoner``)\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with Claude 3.7 Sonnet like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model anthropic/claude-sonnet-4-20250514\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"ANTHROPIC_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model anthropic/claude-sonnet-4-20250514\n\nAvailable Evals\n---------------\n\nThe current evaluations test basic tool use in gptme, such as the ability to: read, write, patch files; run code in ipython, commands in the shell; use git and create new projects with npm and cargo. It also has basic tests for web browsing and data extraction.\n\n.. This is where we want to get to:\n\n The evaluation suite tests models on:\n\n 1. Tool Usage\n\n - Shell commands and file operations\n - Git operations\n - Web browsing and data extraction\n - Project navigation and understanding\n\n 2. Programming Tasks\n\n - Code completion and generation\n - Bug fixing and debugging\n - Documentation writing\n - Test creation\n\n 3. Reasoning\n\n - Multi-step problem solving\n - Tool selection and sequencing\n - Error handling and recovery\n - Self-correction\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/gptme/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "2740eca57898a3cd6a8b04c30e15dda6f8e4c2e9": {"t": 1785765044.2089703, "n": 2870, "h": "a78fb20ccb111b02", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created an evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still tiny and under development, but the eval harness is fully functional.\n\nRecommended Model\n-----------------\n\nThe recommended model is **Claude Sonnet 4** (``anthropic/claude-sonnet-4-20250514``) for its:\n\n- Strong agentic capabilities\n- Strong coder capabilities\n- Strong performance across all tool types and formats\n- Reasoning capabilities\n- Vision & computer use capabilities\n\nDecent alternatives include:\n\n- GPT-4o (``openai/gpt-4o``)\n- Llama 3.1 405B (``openrouter/meta-llama/llama-3.1-405b-instruct``)\n- DeepSeek V3 (``deepseek/deepseek-chat``)\n- DeepSeek R1 (``deepseek/deepseek-reasoner``)\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with Claude 3.7 Sonnet like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"ANTHROPIC_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nAvailable Evals\n---------------\n\nThe current evaluations test basic tool use in gptme, such as the ability to: read, write, patch files; run code in ipython, commands in the shell; use git and create new projects with npm and cargo. It also has basic tests for web browsing and data extraction.\n\n.. This is where we want to get to:\n\n The evaluation suite tests models on:\n\n 1. Tool Usage\n\n - Shell commands and file operations\n - Git operations\n - Web browsing and data extraction\n - Project navigation and understanding\n\n 2. Programming Tasks\n\n - Code completion and generation\n - Bug fixing and debugging\n - Documentation writing\n - Test creation\n\n 3. Reasoning\n\n - Multi-step problem solving\n - Tool selection and sequencing\n - Error handling and recovery\n - Self-correction\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/gptme/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "e88bd56778fd2d928ec22abfe946b4973ce48645": {"t": 1785765045.7820427, "n": 2832, "h": "779229cdbc83bbf6", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created an evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still tiny and under development, but the eval harness is fully functional.\n\nRecommended Model\n-----------------\n\nThe recommended model is **Claude 3.7 Sonnet** (``anthropic/claude-3-7-sonnet-20250219``) for its:\n\n- Strong coder capabilities\n- Strong performance across all tool types\n- Reasoning capabilities\n- Vision & computer use capabilities\n\nDecent alternatives include:\n\n- GPT-4o (``openai/gpt-4o``)\n- Llama 3.1 405B (``openrouter/meta-llama/llama-3.1-405b-instruct``)\n- DeepSeek V3 (``deepseek/deepseek-chat``)\n- DeepSeek R1 (``deepseek/deepseek-reasoner``)\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with Claude 3.7 Sonnet like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"ANTHROPIC_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nAvailable Evals\n---------------\n\nThe current evaluations test basic tool use in gptme, such as the ability to: read, write, patch files; run code in ipython, commands in the shell; use git and create new projects with npm and cargo. It also has basic tests for web browsing and data extraction.\n\n.. This is where we want to get to:\n\n The evaluation suite tests models on:\n\n 1. Tool Usage\n\n - Shell commands and file operations\n - Git operations\n - Web browsing and data extraction\n - Project navigation and understanding\n\n 2. Programming Tasks\n\n - Code completion and generation\n - Bug fixing and debugging\n - Documentation writing\n - Test creation\n\n 3. Reasoning\n\n - Multi-step problem solving\n - Tool selection and sequencing\n - Error handling and recovery\n - Self-correction\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/gptme/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "6ac6f6e78c82bd842235ffb668ccd0761d32e482": {"t": 1785765047.0612085, "n": 2829, "h": "c1229b16f02439cb", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created an evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still tiny and under development, but the eval harness is fully functional.\n\nRecommended Model\n-----------------\n\nThe recommended model is **Claude 3.7 Sonnet** (``anthropic/claude-3-7-sonnet-20250219``) for its:\n\n- Strong coder capabilities\n- Strong performance across all tool types\n- Reasoning capabilities\n- Vision & computer use capabilities\n\nDecent alternatives include:\n\n- GPT-4o (``openai/gpt-4o``)\n- Llama 3.1 405B (``openrouter/meta-llama/llama-3.1-405b-instruct``)\n- DeepSeek V3 (``deepseek/deepseek-chat``)\n- DeepSeek R1 (``deepseek/deepseek-reasoner``)\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with Claude 3.7 Sonnet like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"ANTHROPIC_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nAvailable Evals\n---------------\n\nThe current evaluations test basic tool use in gptme, such as the ability to: read, write, patch files; run code in ipython, commands in the shell; use git and create new projects with npm and cargo. It also has basic tests for web browsing and data extraction.\n\n.. This is where we want to get to:\n\n The evaluation suite tests models on:\n\n 1. Tool Usage\n - Shell commands and file operations\n - Git operations\n - Web browsing and data extraction\n - Project navigation and understanding\n\n 2. Programming Tasks\n - Code completion and generation\n - Bug fixing and debugging\n - Documentation writing\n - Test creation\n\n 3. Reasoning\n - Multi-step problem solving\n - Tool selection and sequencing\n - Error handling and recovery\n - Self-correction\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/gptme/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "92965cd1428090842da5368e3724698ff11ecb3b": {"t": 1785765048.5125887, "n": 2833, "h": "bcd372569e9c69c6", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created an evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still tiny and under development, but the eval harness is fully functional.\n\nRecommended Model\n-----------------\n\nThe recommended model is **Claude 3.7 Sonnet** (``anthropic/claude-3-7-sonnet-20250219``) for its:\n\n- Strong coder capabilities\n- Strong performance across all tool types\n- Reasoning capabilities\n- Vision & computer use capabilities\n\nDecent alternatives include:\n\n- GPT-4o (``openai/gpt-4o``)\n- Llama 3.1 405B (``openrouter/meta-llama/llama-3.1-405b-instruct``)\n- DeepSeek V3 (``deepseek/deepseek-chat``)\n- DeepSeek R1 (``deepseek/deepseek-reasoner``)\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with Claude 3.7 Sonnet like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"ANTHROPIC_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model anthropic/claude-3-7-sonnet-20250219\n\nAvailable Evals\n---------------\n\nThe current evaluations test basic tool use in gptme, such as the ability to: read, write, patch files; run code in ipython, commands in the shell; use git and create new projects with npm and cargo. It also has basic tests for web browsing and data extraction.\n\n.. This is where we want to get to:\n\n The evaluation suite tests models on:\n\n 1. Tool Usage\n - Shell commands and file operations\n - Git operations\n - Web browsing and data extraction\n - Project navigation and understanding\n\n 2. Programming Tasks\n - Code completion and generation\n - Bug fixing and debugging\n - Documentation writing\n - Test creation\n\n 3. Reasoning\n - Multi-step problem solving\n - Tool selection and sequencing\n - Error handling and recovery\n - Self-correction\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/ErikBjare/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "970fc47eb300e74de141a5489b04cbbef792e7c1": {"t": 1785765049.8989785, "n": 1412, "h": "595eb65f55163cfb", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created a evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still under development, but the eval harness is mostly complete.\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with gpt-4o like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model openai/gpt-4o\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"OPENAI_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model openai/gpt-4o\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals such as SWE-Bench, but have not finished it (see `PR #142 <https://github.com/ErikBjare/gptme/pull/142>`_).\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "03f8972d89a08cb33422da1dc13da91577f5b6b4": {"t": 1785765051.391629, "n": 1350, "h": "021609369be2e549", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created a evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still under development, but the eval harness is mostly complete.\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with gpt-4o like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model openai/gpt-4o\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"OPENAI_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model openai/gpt-4o\n\n\nResults\n-------\n\nHere are the results of the evals we have run so far:\n\n.. command-output:: gptme-eval eval_results/*/eval_results.csv\n :cwd: ..\n :shell:\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals, such as SWE-Bench, but have not yet done so.\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "e4eb81caa37cd58cc5b5116ff4ea022311fd4730": {"t": 1785765052.655675, "n": 1997, "h": "d0c8cefb89a9f308", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created a evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still under development, but the eval harness is mostly complete.\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with gpt-4o like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model openai/gpt-4o\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"OPENAI_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model openai/gpt-4o\n\n\nExample run\n-----------\n\nHere's the output from a run of the eval suite:\n\n.. code-block::\n\n $ gptme-eval eval_results/20240917_172916/eval_results.csv\n === Model Comparison ===\n Model init-git init-rust hello hello-patch hello-ask init-react prime100\n ------------------------------------ ---------- ----------- --------- ------------- ----------- ------------ ----------\n openai/gpt-4o ✅ 7.74s ✅ 9.62s ✅ 5.02s ✅ 5.06s ✅ 4.69s ❌ timeout ✅ 7.48s\n openai/o1-mini ✅ 18.44s ✅ 21.63s ✅ 21.20s ✅ 27.39s ❌ timeout ❌ 42.65s ✅ 17.99s\n anthropic/claude-3-5-sonnet-20240620 ❌ timeout ❌ timeout ✅ 8.77s ✅ 7.09s ✅ 8.08s ❌ timeout ✅ 11.26s\n\n\nWe are working on making the evals more robust, informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals, such as SWE-Bench, but have not yet done so.\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "c1987fa307ae18cab6d2561a4a28ea4543e44574": {"t": 1785765054.6624682, "n": 2494, "h": "8e76c55475267227", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created a evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still under development, but the eval harness is mostly complete.\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with gpt-4o like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model openai/gpt-4o\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"OPENAI_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model openai/gpt-4o\n\n\nExample run\n-----------\n\nHere's the output from a run of the eval suite:\n\n.. code-block::\n\n $ gptme-eval eval_results/eval_results_20240830_203143.csv\n +-----------------------------------------------+-------------+------------+-----------+---------------+------------+\n | Model | hello-ask | prime100 | hello | hello-patch | init-git |\n +===============================================+=============+============+===========+===============+============+\n | openai/gpt-4o | ✅ 17.89s | ✅ 18.69s | ✅ 17.66s | ✅ 13.95s | ✅ 20.14s |\n +-----------------------------------------------+-------------+------------+-----------+---------------+------------+\n | anthropic/claude-3-5-sonnet-20240620 | ✅ 21.06s | ✅ 16.46s | ❌ 0.00s | ❌ 0.00s | ❌ 0.00s |\n +-----------------------------------------------+-------------+------------+-----------+---------------+------------+\n | openrouter/meta-llama/llama-3.1-405b-instruct | ✅ 8.68s | ✅ 15.81s | ✅ 12.70s | ✅ 12.12s | ✅ 13.36s |\n +-----------------------------------------------+-------------+------------+-----------+---------------+------------+\n\n\nNote that in this particular run, something went wrong with Anthropic. We are working on making the evals more robust and informative, and challenging.\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals, such as SWE-Bench, but have not yet done so.\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "9c13a60df24d0f34a0f013876ef8d8974db526a2": {"t": 1785765056.7784753, "n": 1191, "h": "d6814124e236762e", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created a evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still under development, but the eval harness is mostly complete.\n\nUsage\n-----\n\nYou can run the simple ``hello`` eval with gpt-4o like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model openai/gpt-4o\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"OPENAI_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/eval_results \\\n gptme-eval hello --model openai/gpt-4o\n\n\nExample run\n-----------\n\nHere's the output from a run of the eval suite: TODO\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals, such as SWE-Bench, but have not yet done so.\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}, "5a0c47e7f8e3032866ae051459d968241a87943e": {"t": 1785765059.5542154, "n": 1167, "h": "8964f592f9241b7d", "c": "Evals\n=====\n\ngptme provides LLMs with a wide variety of tools, but how well do models make use of them? Which tasks can they complete, and which ones do they struggle with? How far can they get on their own, without any human intervention?\n\nTo answer these questions, we have created a evaluation suite that tests the capabilities of LLMs on a wide variety of tasks.\n\n.. note::\n The evaluation suite is still under development, but the eval harness is mostly complete.\n\nYou can run the simple ``hello`` eval with gpt-4o like this:\n\n.. code-block:: bash\n\n gptme-eval hello --model openai/gpt-4o\n\nHowever, we recommend running it in Docker to improve isolation and reproducibility:\n\n.. code-block:: bash\n\n make build-docker\n docker run \\\n -e \"OPENAI_API_KEY=<your api key>\" \\\n -v $(pwd)/eval_results:/app/gptme/eval_results \\\n gptme --timeout 60 $@\n\n\nExample run\n-----------\n\nHere's the output from a run of the eval suite: TODO\n\n\nOther evals\n-----------\n\nWe have considered running gptme on other evals, such as SWE-Bench, but have not yet done so.\n\nIf you are interested in running gptme on other evals, drop a comment in the issues!\n"}}, ".github/workflows/test.yml": {"c92f35c3782680824cbc69384233e19ac44a139d": {"t": 1785747047.84406, "n": 2849, "h": "83e9fcaa2e8debe7", "c": "name: Test\n\non:\n push:\n branches: [ master ]\n pull_request:\n branches: [ master ]\n\nenv:\n OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}\n OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}\n\njobs:\n test:\n name: Test with `${{ matrix.extras }}` and ${{ matrix.model }} on ${{ matrix.os }}\n runs-on: ${{ matrix.os }}\n env:\n RELEASE: false\n strategy:\n fail-fast: false\n matrix:\n os: [ubuntu-latest]\n python_version: ['3.10']\n extras: ['-E all']\n model: ['openai/gpt-4o-mini', 'anthropic/claude-3-haiku-20240307']\n #include:\n # deepseek, no extras, latest python\n #- os: ubuntu-latest\n # python_version: '3.13'\n # extras: ''\n # model: 'deepseek/deepseek-chat'\n # openrouter\n #- os: ubuntu-latest\n # python_version: '3.10'\n # extras: ''\n # model: 'openrouter/deepseek/deepseek-chat'\n # openrouter (llama)\n #- os: ubuntu-latest\n # python_version: '3.10'\n # extras: ''\n # model: 'openrouter/meta-llama/llama-3.3-70B-instruct'\n\n steps:\n - uses: actions/checkout@v5\n with:\n submodules: 'recursive'\n - name: Install apt dependencies\n run: sudo apt-get install universal-ctags pandoc tmux x11-xserver-utils xvfb\n\n - name: Set up Xvfb\n run: |\n Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &\n echo \"DISPLAY=:99\" >> $GITHUB_ENV\n\n - name: Install poetry\n run: pipx install poetry\n\n - name: Set up Python\n uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python_version }}\n cache: 'poetry'\n\n - name: Install dependencies\n run: |\n make build\n poetry install ${{ matrix.extras }}\n\n - name: Install playwright\n if: contains(matrix.extras, 'browser') || contains(matrix.extras, 'all')\n run: poetry run playwright install chromium\n\n # old comment: OpenAI sometimes randomly aborts connections\n - name: Run tests\n uses: nick-fields/retry@v3\n env:\n TERM: xterm\n MODEL: ${{ matrix.model }}\n with:\n timeout_minutes: 5\n max_attempts: 1 # favor pytest retries (mark with flaky)\n retry_wait_seconds: 10\n command: make test SLOW=true\n\n - name: Upload coverage reports to Codecov\n uses: codecov/codecov-action@v5\n env:\n MODEL: ${{ matrix.model }}\n with:\n token: ${{ secrets.CODECOV_TOKEN }}\n env_vars: MODEL\n flags: ${{ matrix.model }}\n\n - name: Upload test results to Codecov\n if: ${{ !cancelled() }}\n uses: codecov/test-results-action@v1\n with:\n token: ${{ secrets.CODECOV_TOKEN }}\n"}, "d5ad404af9d746b79a383dae950e21e9b2101611": {"t": 1785765041.7357314, "n": 2849, "h": "83e9fcaa2e8debe7", "c": "name: Test\n\non:\n push:\n branches: [ master ]\n pull_request:\n branches: [ master ]\n\nenv:\n OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}\n OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}\n\njobs:\n test:\n name: Test with `${{ matrix.extras }}` and ${{ matrix.model }} on ${{ matrix.os }}\n runs-on: ${{ matrix.os }}\n env:\n RELEASE: false\n strategy:\n fail-fast: false\n matrix:\n os: [ubuntu-latest]\n python_version: ['3.10']\n extras: ['-E all']\n model: ['openai/gpt-4o-mini', 'anthropic/claude-3-haiku-20240307']\n #include:\n # deepseek, no extras, latest python\n #- os: ubuntu-latest\n # python_version: '3.13'\n # extras: ''\n # model: 'deepseek/deepseek-chat'\n # openrouter\n #- os: ubuntu-latest\n # python_version: '3.10'\n # extras: ''\n # model: 'openrouter/deepseek/deepseek-chat'\n # openrouter (llama)\n #- os: ubuntu-latest\n # python_version: '3.10'\n # extras: ''\n # model: 'openrouter/meta-llama/llama-3.3-70B-instruct'\n\n steps:\n - uses: actions/checkout@v5\n with:\n submodules: 'recursive'\n - name: Install apt dependencies\n run: sudo apt-get install universal-ctags pandoc tmux x11-xserver-utils xvfb\n\n - name: Set up Xvfb\n run: |\n Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &\n echo \"DISPLAY=:99\" >> $GITHUB_ENV\n\n - name: Install poetry\n run: pipx install poetry\n\n - name: Set up Python\n uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python_version }}\n cache: 'poetry'\n\n - name: Install dependencies\n run: |\n make build\n poetry install ${{ matrix.extras }}\n\n - name: Install playwright\n if: contains(matrix.extras, 'browser') || contains(matrix.extras, 'all')\n run: poetry run playwright install chromium\n\n # old comment: OpenAI sometimes randomly aborts connections\n - name: Run tests\n uses: nick-fields/retry@v3\n env:\n TERM: xterm\n MODEL: ${{ matrix.model }}\n with:\n timeout_minutes: 5\n max_attempts: 1 # favor pytest retries (mark with flaky)\n retry_wait_seconds: 10\n command: make test SLOW=true\n\n - name: Upload coverage reports to Codecov\n uses: codecov/codecov-action@v5\n env:\n MODEL: ${{ matrix.model }}\n with:\n token: ${{ secrets.CODECOV_TOKEN }}\n env_vars: MODEL\n flags: ${{ matrix.model }}\n\n - name: Upload test results to Codecov\n if: ${{ !cancelled() }}\n uses: codecov/test-results-action@v1\n with:\n token: ${{ secrets.CODECOV_TOKEN }}\n"}, "aadb326572b4e708e89dc2b0844e4838e82177a5": {"t": 1785765042.956443, "n": 2849, "h": "89b7e6b42a99688b", "c": "name: Test\n\non:\n push:\n branches: [ master ]\n pull_request:\n branches: [ master ]\n\nenv:\n OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}\n OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}\n\njobs:\n test:\n name: Test with `${{ matrix.extras }}` and ${{ matrix.model }} on ${{ matrix.os }}\n runs-on: ${{ matrix.os }}\n env:\n RELEASE: false\n strategy:\n fail-fast: false\n matrix:\n os: [ubuntu-latest]\n python_version: ['3.10']\n extras: ['-E all']\n model: ['openai/gpt-4o-mini', 'anthropic/claude-3-haiku-20240307']\n #include:\n # deepseek, no extras, latest python\n #- os: ubuntu-latest\n # python_version: '3.13'\n # extras: ''\n # model: 'deepseek/deepseek-chat'\n # openrouter\n #- os: ubuntu-latest\n # python_version: '3.10'\n # extras: ''\n # model: 'openrouter/deepseek/deepseek-chat'\n # openrouter (llama)\n #- os: ubuntu-latest\n # python_version: '3.10'\n # extras: ''\n # model: 'openrouter/meta-llama/llama-3.3-70B-instruct'\n\n steps:\n - uses: actions/checkout@v4\n with:\n submodules: 'recursive'\n - name: Install apt dependencies\n run: sudo apt-get install universal-ctags pandoc tmux x11-xserver-utils xvfb\n\n - name: Set up Xvfb\n run: |\n Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &\n echo \"DISPLAY=:99\" >> $GITHUB_ENV\n\n - name: Install poetry\n run: pipx install poetry\n\n - name: Set up Python\n uses: actions/setup-python@v5\n with:\n python-version: ${{ matrix.python_version }}\n cache: 'poetry'\n\n - name: Install dependencies\n run: |\n make build\n poetry install ${{ matrix.extras }}\n\n - name: Install playwright\n if: contains(matrix.extras, 'browser') || contains(matrix.extras, 'all')\n run: poetry run playwright install chromium\n\n # old comment: OpenAI sometimes randomly aborts connections\n - name: Run tests\n uses: nick-fields/retry@v3\n env:\n TERM: xterm\n MODEL: ${{ matrix.model }}\n with:\n timeout_minutes: 5\n max_attempts: 1 # favor pytest retries (mark with flaky)\n retry_wait_seconds: 10\n command: make test SLOW=true\n\n - name: Upload coverage reports to Codecov\n uses: codecov/codecov-action@v5\n env:\n MODEL: ${{ matrix.model }}\n with:\n token: ${{ secrets.CODECOV_TOKEN }}\n env_vars: MODEL\n flags: ${{ matrix.model }}\n\n - name: Upload test results to Codecov\n if: ${{ !cancelled() }}\n uses: codecov/test-results-action@v1\n with:\n token: ${{ secrets.CODECOV_TOKEN }}\n"}}}} |