1 line
85 KiB
JSON
1 line
85 KiB
JSON
{"repo": "agi4engineering/engdesign", "files": {"README.md": {"c8ec338cc5f846d792d93728b00c9448111139e5": {"t": 1785746488.8205073, "n": 12583, "h": "ad4b0d7fb18cf056", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission, \"Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.\"\n\nOf these 101 tasks, 34 rely on proprietary scientific softwares (e.g., MATLAB or Cadence) and may not run on every system. We provide the complete input datasets and evaluation scripts for these tasks as well —— simply follow the detailed setup instructions to configure the required environments and run them.\n\nThe remaining 67 tasks have no license restrictions and can be evaluated using our hand‑authored scripts. To remove licensing barriers, we’ve extracted these into **EngDesign-Open**, a standalone subset whose repository includes evaluation scripts for all 67 tasks without any proprietary dependencies.\n\nOur evaluation framework currently integrates with twelve LLM variants: GPT‑4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, Gemini‑2.0‑flash, Gemini‑2.5‑pro‑preview‑05‑06, DeepSeek‑Chat, DeepSeek‑Reasoner, Claude‑3‑7‑Sonnet, and Claude‑3‑7‑Sonnet (Extended Reasoning Mode).\n\n---\n## 🚀 Run EngDesign-Open\n\nEngDesign-Open contains **all 67 tasks without license restrictions**. You can run them by following these steps:\n\n### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n### 2. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n### 3. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n### 4. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v /path/to/your/local/directory:/app --entrypoint bash engdesign-sim\n ```\n\n### 5. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), you can run benchmark tasks using the following commands.\n\n#### (1) Run All Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --k 1\n ```\n\n#### (2) Run Specific Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n ```\n\n#### Parameter Descriptions\n\n| Parameter | Description |\n|---------------------|-----------------------------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders (e.g. `./EngDesign-Open`) |\n| `--task_list` | *(Optional)* Names of specific tasks to run (e.g. `AB_01 AB_02`). If not set, all tasks will run |\n| `--model` | Model to use (Names of the twelve LLM variants using in the commands are: **gpt-4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, gemini-2.0-flash, gemini-2.5-pro-preview-05-06, deepseek-chat, deepseek-reasoner, claude-3-7, and claude-3-7-thinking**)|\n| `--api_key` | Your API key for the corresponding provider (OpenAI, Google, DeepSeek, Anthropic, etc.) |\n| `--k` | Number of repetitions per task |\n| `--reasoning_effort`| *(Optional)* Use `high` for o3 or o4-mini models with high-effort reasoning mode |\n\n### 6. Example Commands for All 12 Supported Models\n\n#### OpenAI Models\n\n(1) GPT-4o:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by GPT-4o will be saved in the `{task_id}_log_gpt-4o_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) o1 - OpenAI GPT-4 variant (baseline configuration):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o1 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o1 will be saved in the `{task_id}_log_o1_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(3) o3 - OpenAI GPT-4 variant (enhanced reasoning):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o3 will be saved in the `{task_id}_log_o3_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(4) o3-high - OpenAI GPT-4 variant (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o3-high will be saved in the `{task_id}_log_o3_high_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(5) o4-mini - OpenAI GPT-4 Mini (lightweight version):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o4-mini will be saved in the `{task_id}_log_o4-mini_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(6) o4-mini-high - OpenAI GPT-4 Mini (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o4-mini-high will be saved in the `{task_id}_log_o4-mini_high_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### Gemini Models (Google)\n\n(1) Gemini‑2.0‑flash:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.0-flash \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Gemini‑2.0‑flash will be saved in the `{task_id}_log_gemini-2.0-flash_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) Gemini‑2.5‑pro‑preview‑05‑06:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.5-pro-preview-05-06 \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Gemini‑2.5‑pro‑preview‑05‑06 will be saved in the `{task_id}_log_gemini-2.5-pro-preview-05-06_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### DeepSeek Models\n\n(1) DeepSeek‑Chat:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-chat \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by DeepSeek‑Chat will be saved in the `{task_id}_log_deepseek-chat_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) DeepSeek‑Reasoner:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-reasoner \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by DeepSeek‑Reasoner will be saved in the `{task_id}_log_deepseek-reasoner_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### Claude Models (Anthropic)\n\n(1) Claude‑3‑7‑Sonnet:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7 \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Claude‑3‑7‑Sonnet will be saved in the `{task_id}_log_claude-3-7_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) Claude‑3‑7‑Sonnet (Extended Reasoning Mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7-thinking \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Claude‑3‑7‑Sonnet (Extended Reasoning Mode) will be saved in the `{task_id}_log_claude-3-7-thinking_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n### 7. Other Important Information\n\n#### (1) Replace API Keys\n\nMake sure to replace the api_key in the commands with your actual API keys for the corresponding provider.\n\n#### (2) Find the Task Outputs\n\nYou can find the model's corresponding output files in the `logs` folder within each task directory, where you can view the scores and the model's generated outputs. See Section 6 *(Example Commands for All 12 Supported Models)* for details on the output file names.\n\nPlease note that each task folder in this GitHub repository contains a `logs` folder with pre-generated results from our own runs. **We strongly recommend renaming the original `logs` folder before running your own experiments** to avoid confusion with our provided outputs.\n\nIn particular, for certain models, we have renamed the output log files. As a result, the log files you generate may partially overlap in filename with those already present in the repository, while others may differ. Please be cautious when interpreting or comparing results. (This is also one of the reasons why we recommend renaming the original `logs` folder.)\n\n### 8. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/ # The task folders without license restrictions\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n├── evaluation/ # The driver script for running the benchmark\n│ └── evaluate_llm.py\n├── Dockerfile # Docker configuration for containerized benchmarking\n└── docker_requirements.txt # Dependency list for installing in the Docker environment\n````\n"}, "8352b4d9697cb555312fd815372e0e280ddaf159": {"t": 1785764863.2305546, "n": 12583, "h": "e78b7c03925db62b", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission, \"Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.\"\n\nOf these 101 tasks, 48 rely on proprietary scientific softwares (e.g., MATLAB or Cadence) and may not run on every system. We provide the complete input datasets and evaluation scripts for these tasks as well —— simply follow the detailed setup instructions to configure the required environments and run them.\n\nThe remaining 53 tasks have no license restrictions and can be evaluated using our hand‑authored scripts. To remove licensing barriers, we’ve extracted these into **EngDesign-Open**, a standalone subset whose repository includes evaluation scripts for all 53 tasks without any proprietary dependencies.\n\nOur evaluation framework currently integrates with twelve LLM variants: GPT‑4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, Gemini‑2.0‑flash, Gemini‑2.5‑pro‑preview‑05‑06, DeepSeek‑Chat, DeepSeek‑Reasoner, Claude‑3‑7‑Sonnet, and Claude‑3‑7‑Sonnet (Extended Reasoning Mode).\n\n---\n## 🚀 Run EngDesign-Open\n\nEngDesign-Open contains **all 53 tasks without license restrictions**. You can run them by following these steps:\n\n### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n### 2. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n### 3. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n### 4. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v /path/to/your/local/directory:/app --entrypoint bash engdesign-sim\n ```\n\n### 5. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), you can run benchmark tasks using the following commands.\n\n#### (1) Run All Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --k 1\n ```\n\n#### (2) Run Specific Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n ```\n\n#### Parameter Descriptions\n\n| Parameter | Description |\n|---------------------|-----------------------------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders (e.g. `./EngDesign-Open`) |\n| `--task_list` | *(Optional)* Names of specific tasks to run (e.g. `AB_01 AB_02`). If not set, all tasks will run |\n| `--model` | Model to use (Names of the twelve LLM variants using in the commands are: **gpt-4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, gemini-2.0-flash, gemini-2.5-pro-preview-05-06, deepseek-chat, deepseek-reasoner, claude-3-7, and claude-3-7-thinking**)|\n| `--api_key` | Your API key for the corresponding provider (OpenAI, Google, DeepSeek, Anthropic, etc.) |\n| `--k` | Number of repetitions per task |\n| `--reasoning_effort`| *(Optional)* Use `high` for o3 or o4-mini models with high-effort reasoning mode |\n\n### 6. Example Commands for All 12 Supported Models\n\n#### OpenAI Models\n\n(1) GPT-4o:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by GPT-4o will be saved in the `{task_id}_log_gpt-4o_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) o1 - OpenAI GPT-4 variant (baseline configuration):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o1 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o1 will be saved in the `{task_id}_log_o1_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(3) o3 - OpenAI GPT-4 variant (enhanced reasoning):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o3 will be saved in the `{task_id}_log_o3_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(4) o3-high - OpenAI GPT-4 variant (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o3-high will be saved in the `{task_id}_log_o3_high_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(5) o4-mini - OpenAI GPT-4 Mini (lightweight version):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o4-mini will be saved in the `{task_id}_log_o4-mini_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(6) o4-mini-high - OpenAI GPT-4 Mini (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o4-mini-high will be saved in the `{task_id}_log_o4-mini_high_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### Gemini Models (Google)\n\n(1) Gemini‑2.0‑flash:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.0-flash \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Gemini‑2.0‑flash will be saved in the `{task_id}_log_gemini-2.0-flash_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) Gemini‑2.5‑pro‑preview‑05‑06:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.5-pro-preview-05-06 \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Gemini‑2.5‑pro‑preview‑05‑06 will be saved in the `{task_id}_log_gemini-2.5-pro-preview-05-06_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### DeepSeek Models\n\n(1) DeepSeek‑Chat:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-chat \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by DeepSeek‑Chat will be saved in the `{task_id}_log_deepseek-chat_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) DeepSeek‑Reasoner:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-reasoner \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by DeepSeek‑Reasoner will be saved in the `{task_id}_log_deepseek-reasoner_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### Claude Models (Anthropic)\n\n(1) Claude‑3‑7‑Sonnet:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7 \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Claude‑3‑7‑Sonnet will be saved in the `{task_id}_log_claude-3-7_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) Claude‑3‑7‑Sonnet (Extended Reasoning Mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7-thinking \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Claude‑3‑7‑Sonnet (Extended Reasoning Mode) will be saved in the `{task_id}_log_claude-3-7-thinking_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n### 7. Other Important Information\n\n#### (1) Replace API Keys\n\nMake sure to replace the api_key in the commands with your actual API keys for the corresponding provider.\n\n#### (2) Find the Task Outputs\n\nYou can find the model's corresponding output files in the `logs` folder within each task directory, where you can view the scores and the model's generated outputs. See Section 6 *(Example Commands for All 12 Supported Models)* for details on the output file names.\n\nPlease note that each task folder in this GitHub repository contains a `logs` folder with pre-generated results from our own runs. **We strongly recommend renaming the original `logs` folder before running your own experiments** to avoid confusion with our provided outputs.\n\nIn particular, for certain models, we have renamed the output log files. As a result, the log files you generate may partially overlap in filename with those already present in the repository, while others may differ. Please be cautious when interpreting or comparing results. (This is also one of the reasons why we recommend renaming the original `logs` folder.)\n\n### 8. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/ # The task folders without license restrictions\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n├── evaluation/ # The driver script for running the benchmark\n│ └── evaluate_llm.py\n├── Dockerfile # Docker configuration for containerized benchmarking\n└── docker_requirements.txt # Dependency list for installing in the Docker environment\n````\n"}, "b7c3bd590df70255f09a4e692961b6687334de11": {"t": 1785764864.7475297, "n": 12635, "h": "a8fe626107241807", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission, \"Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.\"\n\nOf these 101 tasks, 48 rely on proprietary scientific softwares (e.g., MATLAB or Cadence) and may not run on every system. We provide the complete input datasets and evaluation scripts for these tasks as well —— simply follow the detailed setup instructions to configure the required environments and run them.\n\nThe remaining 53 tasks have no license restrictions and can be evaluated using our hand‑authored scripts. To remove licensing barriers, we’ve extracted these into **EngDesign-Open**, a standalone subset whose repository includes evaluation scripts for all 53 tasks without any proprietary dependencies.\n\nOur evaluation framework currently integrates with twelve LLM variants: GPT‑4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, Gemini‑2.0‑flash, Gemini‑2.5‑pro‑preview‑05‑06, DeepSeek‑Chat, DeepSeek‑Reasoner, Claude‑3‑7‑Sonnet, and Claude‑3‑7‑Sonnet (Extended Reasoning Mode).\n\n---\n## 🚀 Run EngDesign-Open\n\nEngDesign-Open contains **all 53 tasks without license restrictions**. You can run them by following these steps:\n\n### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n### 2. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n### 3. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n### 4. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v /path/to/your/local/directory:/app --entrypoint bash engdesign-sim\n ```\n\n### 5. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), you can run benchmark tasks using the following commands.\n\n#### (1) Run All Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --k 1\n ```\n\n#### (2) Run Specific Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n ```\n\n#### Parameter Descriptions\n\n| Parameter | Description |\n|---------------------|-----------------------------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders (e.g. `./EngDesign-Open`) |\n| `--task_list` | *(Optional)* Names of specific tasks to run (e.g. `AB_01 AB_02`). If not set, all tasks will run |\n| `--model` | Model to use (Names of the twelve LLM variants using in the commands are: **gpt-4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, gemini-2.0-flash, gemini-2.5-pro-preview-05-06, deepseek-chat, deepseek-reasoner, claude-3-7, and claude-3-7-thinking**)|\n| `--api_key` | Your API key for the corresponding provider (OpenAI, Google, DeepSeek, Anthropic, etc.) |\n| `--k` | Number of repetitions per task |\n| `--reasoning_effort`| *(Optional)* Use `high` for o3 or o4-mini models with high-effort reasoning mode |\n\n### 6. Example Commands for All 12 Supported Models\n\n#### OpenAI Models\n\n(1) GPT-4o:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by GPT-4o will be saved in the `{task_id}_log_gpt-4o_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) o1 - OpenAI GPT-4 variant (baseline configuration):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o1 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o1 will be saved in the `{task_id}_log_o1_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(3) o3 - OpenAI GPT-4 variant (enhanced reasoning):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o3 will be saved in the `{task_id}_log_o3_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(4) o3-high - OpenAI GPT-4 variant (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o3-high will be saved in the `{task_id}_log_o3_high_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(5) o4-mini - OpenAI GPT-4 Mini (lightweight version):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o4-mini will be saved in the `{task_id}_log_o4-mini_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(6) o4-mini-high - OpenAI GPT-4 Mini (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by o4-mini-high will be saved in the `{task_id}_log_o4-mini_high_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### Gemini Models (Google)\n\n(1) Gemini‑2.0‑flash:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.0-flash \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Gemini‑2.0‑flash will be saved in the `{task_id}_log_gemini-2.0-flash_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) Gemini‑2.5‑pro‑preview‑05‑06:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.5-pro-preview-05-06 \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Gemini‑2.5‑pro‑preview‑05‑06 will be saved in the `{task_id}_log_gemini-2.5-pro-preview-05-06_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### DeepSeek Models\n\n(1) DeepSeek‑Chat:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-chat \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by DeepSeek‑Chat will be saved in the `{task_id}_log_deepseek-chat_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) DeepSeek‑Reasoner:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-reasoner \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by DeepSeek‑Reasoner will be saved in the `{task_id}_log_deepseek-reasoner_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n#### Claude Models (Anthropic)\n\n(1) Claude‑3‑7‑Sonnet:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7 \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Claude‑3‑7‑Sonnet will be saved in the `{task_id}_log_claude-3-7_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n(2) Claude‑3‑7‑Sonnet (Extended Reasoning Mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7-thinking \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\nThe results and scores generated by Claude‑3‑7‑Sonnet (Extended Reasoning Mode) will be saved in the `{task_id}_log_claude-3-7-thinking_{i}.jsonl` file located in the `logs` folder within each task directory, where `i` indicates the trial number.\n\n### 7. Other Important Information\n\n#### (1) Replace API Keys\n\nMake sure to replace the api_key in the commands with your actual API keys for the corresponding provider.\n\n#### (2) Find the Task Outputs\n\nYou can find the model's corresponding output files in the `logs` folder within each task directory, where you can view the scores and the model's generated outputs. See Section 6 *(Example Commands for All 12 Supported Models)* for details on the output file names.\n\nPlease note that each task folder in this GitHub repository contains a `logs` folder with pre-generated results from our own runs. **We strongly recommend renaming the original `logs` folder before running your own experiments** to avoid confusion with our provided outputs.\n\nIn particular, for certain models, we have renamed the output log files. As a result, the log files you generate may partially overlap in filename with those already present in the repository, while others may differ. Please be cautious when interpreting or comparing results. (This is also one of the reasons why we recommend renaming the original `logs` folder.)\n\n### 8. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```\n\n---\n\n## 🚀 Run Tasks Requiring Scientific Softwares\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/ # The task folders without license restrictions\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n├── evaluation/ # The driver script for running the benchmark\n│ └── evaluate_llm.py\n├── Dockerfile # Docker configuration for containerized benchmarking\n└── docker_requirements.txt # Dependency list for installing in the Docker environment\n````\n"}, "4989c666eabbed5c1545f6d554e8440a43f4b832": {"t": 1785764866.081663, "n": 9381, "h": "92b9f99d653764bf", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission, \"Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.\"\n\nOf these 101 tasks, 48 rely on proprietary scientific softwares (e.g., MATLAB or Cadence) and may not run on every system. We provide the complete input datasets and evaluation scripts for these tasks as well —— simply follow the detailed setup instructions to configure the required environments and run them.\n\nThe remaining 53 tasks have no license restrictions and can be evaluated using our hand‑authored scripts. To remove licensing barriers, we’ve extracted these into **EngDesign-Open**, a standalone subset whose repository includes evaluation scripts for all 53 tasks without any proprietary dependencies.\n\nOur evaluation framework currently integrates with twelve LLM variants: GPT‑4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, Gemini‑2.0‑flash, Gemini‑2.5‑pro‑preview‑05‑06, DeepSeek‑Chat, DeepSeek‑Reasoner, Claude‑3‑7‑Sonnet, and Claude‑3‑7‑Sonnet (Extended Reasoning Mode).\n\n---\n## 🚀 Run EngDesign-Open\n\nEngDesign-Open contains **all 53 tasks without license restrictions**. You can run them by following these steps:\n\n### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n### 2. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n### 3. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n### 4. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v the_actual_full_path_to_your_local_project_directory --entrypoint bash engdesign-sim\n ```\n\n### 5. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), you can run benchmark tasks using the following commands.\n\n#### (1) Run All Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --k 1\n ```\n\n#### (2) Run Specific Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n ```\n\n#### Parameter Descriptions\n\n| Parameter | Description |\n|---------------------|-----------------------------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders (e.g. `./EngDesign-Open`) |\n| `--task_list` | *(Optional)* Names of specific tasks to run (e.g. `AB_01 AB_02`). If not set, all tasks will run |\n| `--model` | Model to use (Names of the twelve LLM variants using in the commands are: **gpt-4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, gemini-2.0-flash, gemini-2.5-pro-preview-05-06, deepseek-chat, deepseek-reasoner, claude-3-7, and claude-3-7-thinking**)|\n| `--api_key` | Your API key for the corresponding provider (OpenAI, Google, DeepSeek, Anthropic, etc.) |\n| `--k` | Number of repetitions per task |\n| `--reasoning_effort`| *(Optional)* Use `high` for o3 or o4-mini models with high-effort reasoning mode |\n\n### 6. Example Commands for All 12 Supported Models\n\n#### OpenAI Models\n\n(1) gpt-4o:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) o1 - OpenAI GPT-4 variant (baseline configuration):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o1 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(3) o3 - OpenAI GPT-4 variant (enhanced reasoning):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(4) o3-high - OpenAI GPT-4 variant (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(5) o4-mini - OpenAI GPT-4 Mini (lightweight version):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(6) o4-mini-high - OpenAI GPT-4 Mini (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### Gemini Models (Google)\n\n(1) gemini-2.0-flash:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.0-flash \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) gemini-2.5-pro-preview-05-06:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.5-pro-preview-05-06 \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### DeepSeek Models\n\n(1) deepseek-chat:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-chat \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) deepseek-reasoner:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-reasoner \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### Claude Models (Anthropic)\n\n(1) claude-3-7-sonnet:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7 \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) claude-3-7-sonnet (extended reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7-thinking \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n### 7. Other Important Information\n\n#### (1) Replace API Keys\n\nMake sure to replace the api_key in the commands with your actual API keys for the corresponding provider.\n\n#### (2) Find the Task Outputs\n\nYou can find the model's corresponding output files in the `logs` folder within each task directory, where you can view the scores and the model's generated outputs.\n\n### 8. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```\n\n---\n\n## 🚀 Run Tasks Requiring Scientific Softwares\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/ # The task folders without license restrictions\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n├── evaluation/ # The driver script for running the benchmark\n│ └── evaluate_llm.py\n├── Dockerfile # Docker configuration for containerized benchmarking\n└── docker_requirements.txt # Dependency list for installing in the Docker environment\n````\n"}, "35d4fd064459a5ff77b75e72d16ef46c2bec5b28": {"t": 1785764867.376966, "n": 9380, "h": "b134576aeb148427", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission, \"Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.\"\n\nOf these 101 tasks, 48 rely on proprietary scientific softwares (e.g., MATLAB or Cadence) and may not run on every system. We provide the complete input datasets and evaluation scripts for these tasks as well —— simply follow the detailed setup instructions to configure the required environments and run them.\n\nThe remaining 53 tasks have no license restrictions and can be evaluated using our hand‑authored scripts. To remove licensing barriers, we’ve extracted these into **EngDesign-Open**, a standalone subset whose repository includes evaluation scripts for all 53 tasks without any proprietary dependencies.\n\nOur evaluation framework currently integrates with twelve LLM variants: GPT‑4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, Gemini‑2.0‑flash, Gemini‑2.5‑pro‑preview‑05‑06, DeepSeek‑Chat, DeepSeek‑Reasoner, Claude‑3‑7‑Sonnet, and Claude‑3‑7‑Sonnet (Extended Reasoning Mode).\n\n---\n## 🚀 Run EngDesign-Open\n\nEngDesign-Open contains **all 53 tasks without license restrictions**. You can run them by following these steps:\n\n### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n### 2. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n### 3. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n### 4. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v the_actual_full_path_to_your_local_project_directory --entrypoint bash engdesign-sim\n ```\n\n### 5. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), you can run benchmark tasks using the following commands.\n\n#### (1) Run All Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --k 1\n ```\n\n#### (2) Run Specific Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n ```\n\n#### 🛠️ Parameter Descriptions\n\n| Parameter | Description |\n|---------------------|-----------------------------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders (e.g. `./EngDesign-Open`) |\n| `--task_list` | *(Optional)* Names of specific tasks to run (e.g. `AB_01 AB_02`). If not set, all tasks will run |\n| `--model` | Model to use (Names of the twelve LLM variants using in the commands are: gpt-4o, o1, o3, o3‑high, o4‑mini, o4‑mini‑high, gemini-2.0-flash, gemini-2.5-pro-preview-05-06, deepseek-chat, deepseek-reasoner, claude-3-7, and claude-3-7-thinking)|\n| `--api_key` | Your API key for the corresponding provider (OpenAI, Google, DeepSeek, Anthropic, etc.) |\n| `--k` | Number of repetitions per task |\n| `--reasoning_effort`| *(Optional)* Use `high` for o3 or o4-mini models with high-effort reasoning mode |\n\n### 6. Example Commands for All 12 Supported Models\n\n#### OpenAI Models\n\n(1) gpt-4o:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) o1 - OpenAI GPT-4 variant (baseline configuration):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o1 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(3) o3 - OpenAI GPT-4 variant (enhanced reasoning):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(4) o3-high - OpenAI GPT-4 variant (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o3 \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(5) o4-mini - OpenAI GPT-4 Mini (lightweight version):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(6) o4-mini-high - OpenAI GPT-4 Mini (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model o4-mini \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### Gemini Models (Google)\n\n(1) gemini-2.0-flash:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.0-flash \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) gemini-2.5-pro-preview-05-06:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model gemini-2.5-pro-preview-05-06 \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### DeepSeek Models\n\n(1) deepseek-chat:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-chat \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) deepseek-reasoner:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model deepseek-reasoner \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### Claude Models (Anthropic)\n\n(1) claude-3-7-sonnet:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7 \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) claude-3-7-sonnet (extended reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/evaluate_llm.py \\\n --model claude-3-7-thinking \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n### 7. Other Important Information\n\n#### (1) Replace API Keys\n\nMake sure to replace the api_key in the commands with your actual API keys for the corresponding provider.\n\n#### (2) Find the Task Outputs\n\nYou can find the model's corresponding output files in the `logs` folder within each task directory, where you can view the scores and the model's generated outputs.\n\n### 8. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```\n\n---\n\n## 🚀 Run Tasks Requiring Scientific Softwares\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/ # The task folders without license restrictions\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n├── evaluation/ # The driver script for running the benchmark\n│ └── evaluate_llm.py\n├── Dockerfile # Docker configuration for containerized benchmarking\n└── docker_requirements.txt # Dependency list for installing in the Docker environment\n````\n"}, "77e260193e95d6be3e3386d69960557164f93701": {"t": 1785764869.1641734, "n": 9354, "h": "35defb63974c6c3f", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission: \n**“Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.”**\n\nAmong the 101 tasks in **EngDesign**, 48 require domain-specific scientific software such as MATLAB or Cadence for evaluation, which may not be excuable for all machines. The remaining 53 tasks are fully open-source and can be evaluated using manually authored scripts. To facilitate broader community adoption without licensing constraints, we have consolidated these 53 tasks into a subset called **EngDesign-Open**.\n\nThe 48 tasks that rely on external software dependencies and licensed environments are less accessible for universal execution. As a result, our GitHub repository currently includes evaluation scripts only for the 53 open-source tasks within EngDesign-Open.\n\nThe evaluation script we provide currently supports 12 different LLM versions: GPT-4o, o1, o3, o3-high, o4-mini, o4-mini-high, gemini-2.0-flash, gemini-2.5-pro-preview-05-06, deepseek-chat, deepseek-reasoner, claude-3-7-sonnet, and claude-3-7-sonnet (extended reasoning mode).\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/ # The open-source task folders\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n├── evaluation/ # The driver script for running the benchmark\n│ └── eval_openai_llm.py\n├── Dockerfile # Docker configuration for containerized benchmarking\n└── docker_requirements.txt # Dependency list for installing in the Docker environment\n````\n\n---\n## 🚀 How to Run All Open Source Tasks (EngDesign-Open)\n\nEngDesign-Open contains **all 53 open-source tasks**. You can run them by following these steps:\n\n### Setup Instructions\n\n#### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n#### 2. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n#### 3. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n#### 4. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v the_actual_full_path_to_your_local_project_directory --entrypoint bash engdesign-sim\n ```\n\n#### 5. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), you can run benchmark tasks using the following commands.\n\n##### (1) Run All Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --k 1\n ```\n\n##### (2) Run Specific Tasks with a Given Model\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n ```\n\n##### 🛠️ Parameter Descriptions\n\n| Parameter | Description |\n|---------------------|-----------------------------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders (e.g. `./EngDesign-Open`) |\n| `--task_list` | *(Optional)* Names of specific tasks to run (e.g. `AB_01 AB_02`). If not set, all tasks will run |\n| `--model` | Model to use (the specified 12 LLM versions) |\n| `--api_key` | Your API key for the corresponding provider (OpenAI, Google, DeepSeek, Anthropic, etc.) |\n| `--k` | Number of repetitions per task |\n| `--reasoning_effort`| *(Optional)* Use `high` for o3 or o4-mini models with high-effort reasoning mode |\n\n#### 6. Example Commands for All 12 Supported Models\n\n##### 🧠 OpenAI Models\n\n(1) gpt-4o:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model gpt-4o \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) o1 - OpenAI GPT-4 variant (baseline configuration):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model o1 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(3) o3 - OpenAI GPT-4 variant (enhanced reasoning):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model o3 \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(4) o3-high - OpenAI GPT-4 variant (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model o3 \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(5) o4-mini - OpenAI GPT-4 Mini (lightweight version):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model o4-mini \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(6) o4-mini-high - OpenAI GPT-4 Mini (high-effort reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model o4-mini \\\n --reasoning_effort high \\\n --api_key your_openai_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n##### 🔮 Gemini Models (Google)\n\n(1) gemini-2.0-flash:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model gemini-2.0-flash \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) gemini-2.5-pro-preview-05-06:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model gemini-2.5-pro-preview-05-06 \\\n --api_key your_gemini_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n##### 🔍 DeepSeek Models\n\n(1) deepseek-chat:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model deepseek-chat \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) deepseek-reasoner:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model deepseek-reasoner \\\n --api_key your_deepseek_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n##### 🤖 Claude Models (Anthropic)\n\n(1) claude-3-7-sonnet:\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model claude-3-7 \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n(2) claude-3-7-sonnet (extended reasoning mode):\n```bash\nxvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm.py \\\n --model claude-3-7-thinking \\\n --api_key your_anthropic_api_key \\\n --task_dir ./EngDesign-Open \\\n --task_list AB_01 AB_02 \\\n --k 1\n```\n\n#### 7. Other Important Information\n\n##### (1) Replace API Keys\n\nMake sure to replace the api_key in the commands with your actual API keys for the corresponding provider.\n\n##### (2) Find the Task Outputs\n\nYou can find the model's corresponding output files in the `logs` folder within each task directory, where you can view the scores and the model's generated outputs.\n\n#### 8. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n#### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```"}, "fb893f909bca01bdc4dbb26d2a67fa2cd7855537": {"t": 1785764870.5576913, "n": 4215, "h": "5aa8d437a4980c11", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission: \n**“Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.”**\n\nAmong the 101 tasks in **EngDesign**, 48 require domain-specific scientific software such as MATLAB or Cadence for evaluation, which may not be excuable for all machines. The remaining 53 tasks are fully open-source and can be evaluated using manually authored scripts. To facilitate broader community adoption without licensing constraints, we have consolidated these 53 tasks into a subset called **EngDesign-Open**.\nSince the remaining 48 tasks depend on proprietary software, we currently provide run commands only for the open-source subset.\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n└── evaluation/ # Driver scripts & helpers for running the benchmark\n ├── eval_openai_llm.py\n └── eval_openai_llm_new.py\n````\n\n---\n## 🚀 How to Run All Open Source Tasks (EngDesign-Open)\n\nEngDesign-Open contains **all 53 open source tasks**. You can run them by following these steps:\n\n### Setup Instructions\n\n#### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n#### 2. Replace API Keys\n\nReplace the top of evaluation/eval_openai_llm_new.py with your actual OpenAI API keys before building the container.\n\n#### 3. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n#### 4. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n#### 5. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v the_actual_full_path_to_your_local_project_directory --entrypoint bash engdesign-sim\n ```\n\n#### 6. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), run one of the following:\n\n##### (1) Run All Tasks\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm_new.py \\\n --task_dir ./EngDesign-Open --model gpt-4o --k 1\n ```\n\n##### (2) Run Specific Tasks\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm_new.py \\\n --task_dir ./EngDesign-Open --task_list AB_01 AB_02 --model gpt-4o --k 1\n ```\n\n##### Parameter Description\n| Parameter | Description |\n|---------------|-----------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders |\n| `--task_list` | *(Optional)* Names of specific tasks to run. If not set, all tasks will run |\n| `--model` | Model to use, e.g., `gpt-4o` |\n| `--k` | Number of repetitions per task |\n\n#### 7. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n#### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```"}, "728f1f28c8f8c5677a1a81f0895e725725a5f64a": {"t": 1785764871.8551993, "n": 4138, "h": "2f520e5183df0414", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission: \n**“Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.”**\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n└── evaluation/ # Driver scripts & helpers for running the benchmark\n ├── eval_openai_llm.py\n └── eval_openai_llm_new.py\n````\n\n---\n\n## 🚀 Quickstart\n\n1. **Install dependencies**\n\n ```bash\n pip install -r requirements.txt\n ```\n\n2. **Configure your OpenAI key**\n Edit the top of `evaluation/eval_openai_llm.py` (or set the `OPENAI_API_KEY` environment variable).\n\n3. **Run the full benchmark**\n\n ```bash\n python evaluation/eval_openai_llm.py --task_dir tasks\n ```\n\n> **Note:**\n>\n> * Some tasks rely on external tools (e.g., MATLAB, SPICE), which may not be excuable for all machines.\n\n\n\n---\n## 🚀 How to Run All Open Source Tasks (EngDesign-Open)\n\nEngDesign-Open contains **all 53 open source tasks**. You can run them by following these steps:\n\n### Setup Instructions\n\n#### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n#### 2. Replace API Keys\n\nReplace the top of evaluation/eval_openai_llm_new.py with your actual OpenAI API keys before building the container.\n\n#### 3. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n#### 4. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n#### 5. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v the_actual_full_path_to_your_local_project_directory --entrypoint bash engdesign-sim\n ```\n\n#### 6. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), run one of the following:\n\n##### (1) Run All Tasks\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm_new.py \\\n --task_dir ./EngDesign-Open --model gpt-4o --k 1\n ```\n\n##### (2) Run Specific Tasks\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm_new.py \\\n --task_dir ./EngDesign-Open --task_list AB_01 AB_02 --model gpt-4o --k 1\n ```\n\n##### Parameter Description\n| Parameter | Description |\n|---------------|-----------------------------------------------------------------------------|\n| `--task_dir` | Directory containing the task folders |\n| `--task_list` | *(Optional)* Names of specific tasks to run. If not set, all tasks will run |\n| `--model` | Model to use, e.g., `gpt-4o` |\n| `--k` | Number of repetitions per task |\n\n#### 7. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n#### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```"}, "0d00e0a502e5382a0a9f299d49eb124939544692": {"t": 1785764873.0876293, "n": 3801, "h": "f87d62c5171cfaa3", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission: \n**“Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.”**\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── ...\n├── EngDesign-Open/\n│ ├── <task_id>/\n│ └── ...\n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n└── evaluation/ # Driver scripts & helpers for running the benchmark\n ├── eval_openai_llm.py\n └── eval_openai_llm_new.py\n````\n\n---\n\n## 🚀 Quickstart\n\n1. **Install dependencies**\n\n ```bash\n pip install -r requirements.txt\n ```\n\n2. **Configure your OpenAI key**\n Edit the top of `evaluation/eval_openai_llm.py` (or set the `OPENAI_API_KEY` environment variable).\n\n3. **Run the full benchmark**\n\n ```bash\n python evaluation/eval_openai_llm.py --task_dir tasks\n ```\n\n> **Note:**\n>\n> * Some tasks rely on external tools (e.g., MATLAB, SPICE), which may not be excuable for all machines.\n\n\n\n---\n## How to Run All Open Source Tasks (EngDesign-Open)\n\nEngDesign-Open contains **all 53 open source tasks**. You can run them by following these steps:\n\n### Setup Instructions\n\n#### 1. Install and Log in to Docker\n\n- Register at [hub.docker.com](https://hub.docker.com/) and **verify your email**.\n- Download and install Docker Desktop on your machine: [Download Docker Desktop](https://www.docker.com/products/docker-desktop/)\n- Launch Docker Desktop and log in to your account.\n- Make sure Docker Desktop has access to your drive (check settings).\n\n#### 2. Replace API Keys\n\nReplace the top of evaluation/eval_openai_llm_new.py with your actual OpenAI API keys before building the container.\n\n#### 3. Authenticate via CLI\n\nIn a terminal, run:\n\n ```bash\n docker login -u your_dockerhub_username\n ```\n\n#### 4. Build the Docker Image\n\nRun the following command in the root directory of this project:\n\n ```bash\n docker build -t engdesign-sim .\n ```\n\n#### 5. Start a Docker Container\n\nMount your local project directory and start a bash session in the container:\n\n ```bash\n docker run -it --rm -v the_actual_full_path_to_your_local_project_directory --entrypoint bash engdesign-sim\n ```\n\n#### 6. Run the Benchmark Tasks\n\nOnce inside the container (you'll see a prompt like root@xxxxxxxxxxxx:/app#), run one of the following:\n\n##### (1) Run All Tasks\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm_new.py \\\n --task_dir ./EngDesign-Open --model gpt-4o --k 1\n ```\n\n##### (2) Run Specific Tasks\n\n ```bash\n xvfb-run -a -e /dev/stdout --server-args=\"-screen 0 1024x768x24\" \\\n python3 evaluation/eval_openai_llm_new.py \\\n --task_dir ./EngDesign-Open --task_list AB_01 AB_02 --model gpt-4o --k 1\n ```\n\n##### Parameter Description\n --task_dir: Directory containing the task folders\n --task_list: (Optional) Names of specific tasks to run (If not set, all tasks in EngDesign-Open will be run.)\n --model: Model to use, e.g., gpt-4o\n --k: Number of repetitions per task\n\n#### 7. Exit the Container\n\nType ```exit``` to quit the container shell.\n\n#### Optional Cleanup\n\nRemove the image if needed:\n\n ```bash\n docker image rm engdesign-sim\n ```"}, "4fb85a50d381a01bd0ff0525e5770194b485cb18": {"t": 1785764874.4191344, "n": 1489, "h": "160d7b987bc578e7", "c": "# EngDesign\n\n**EngDesign** is a benchmark of 101 structured engineering‑design tasks spanning multiple domains. This repository supports our NeurIPS Datasets & Benchmarks track submission: \n**“Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.”**\n\n---\n\n## 📂 Repository Layout\n\n```text\n├── tasks/ # 101 individual task folders\n│ ├── <task_id>/ # e.g. XG_01\n│ │ ├── LLM_prompt.txt # Prompt presented to the LLM\n│ │ ├── output_structure.py # Defines the expected JSON/Python output schema via instructor\n│ │ ├── evaluate.py # Runs simulations & computes evaluation results\n│ │ ├── images/ # (Optional) Input images for multimodal tasks\n│ │ └── logs/ # Our evaluation logs\n│ └── … \n├── iterative_result/ # Logs from iterative design runs with GPT‑4o, o1, o3, o4‑mini\n└── evaluation/ # Driver scripts & helpers for running the benchmark\n └── eval_openai_llm.py\n````\n\n---\n\n## 🚀 Quickstart\n\n1. **Install dependencies**\n\n ```bash\n pip install -r requirements.txt\n ```\n\n2. **Configure your OpenAI key**\n Edit the top of `evaluation/eval_openai_llm.py` (or set the `OPENAI_API_KEY` environment variable).\n\n3. **Run the full benchmark**\n\n ```bash\n python evaluation/eval_openai_llm.py --task_dir tasks\n ```\n\n> **Note:**\n>\n> * Some tasks rely on external tools (e.g., MATLAB, SPICE), which may not be excuable for all machines.\n\n\n"}, "9e4203b8d34a7ced444f8ca414f21a4df16dc412": {"t": 1785764875.6041093, "n": 1335, "h": "e38584d44e04ef88", "c": "# EngDesign\n\nA benchmark of 101 structured engineering design tasks spanning multiple domains.\n\nThis repository contains all tasks included in **EngDesign**, developed for our NeurIPS Datasets and Benchmarks track submission:\n**\"Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs.\"**\n\n### 📁 Repository Structure\n\nThe repository includes a `tasks` folder with 101 engineering design tasks. Each task is organized as a separate folder containing at least the following components:\n\n* **`LLM_prompt.txt`**: The task description presented to the LLM.\n* **`output_structure.py`**: A Python script that defines the expected structured output format for the task.\n* **`evaluate.py`**: An evaluation script that runs simulations using the LLM-generated outputs and computes performance metrics.\n* **`images/`** *(optional)*: A directory containing images used as part of the task input (for multimodal tasks).\n* **`logs/`**: A directory containing all the evaluation logs for the task.\n* **Other utility files**: Additional Python scripts or resources needed to support evaluation.\n\n### ⚠️ Requirements\n\nSome tasks rely on domain-specific simulation tools such as **MATLAB** or **SPICE**, which may not be available in all environments. Please refer to the individual task folders for tool-specific instructions.\n"}, "9516ac4ef8094cc997f7e7be3e1eafba55e98300": {"t": 1785764876.9720201, "n": 92, "h": "1e675af3efbf54d7", "c": "# EngDesign\nA benchmark of 101 structured engineering design tasks across multiple domains.\n"}}}} |