{"repo": "aialt/pillagerbench", "files": {"main.py": {"4cc901351bde4f0cbb798ba4b9b6397b0fc2fa79": {"t": 1785745406.0519905, "n": 508, "h": "ea32300e87c1e697", "c": "import os\n\nimport hydra\n\nfrom api_keys import *\nfrom bench.config import Config\nfrom bench.pillager_bench import PillagerBench\n\n\n@hydra.main(config_path=\"configs\", config_name=\"benchmark\", version_base=\"1.3\")\ndef main(args: Config):\n # set openai api key\n os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n os.environ[\"DEEPSEEK_API_KEY\"] = deepseek_api_key\n os.environ[\"OPENROUTER_API_KEY\"] = openrouter_api_key\n\n bench = PillagerBench(args)\n bench.run()\n\nif __name__ == \"__main__\":\n main()\n"}, "0f689830eb5661dd5f1ff0b4c66e3e6165708529": {"t": 1785752847.7809122, "n": 508, "h": "ea32300e87c1e697", "c": "import os\n\nimport hydra\n\nfrom api_keys import *\nfrom bench.config import Config\nfrom bench.pillager_bench import PillagerBench\n\n\n@hydra.main(config_path=\"configs\", config_name=\"benchmark\", version_base=\"1.3\")\ndef main(args: Config):\n # set openai api key\n os.environ[\"OPENAI_API_KEY\"] = openai_api_key\n os.environ[\"DEEPSEEK_API_KEY\"] = deepseek_api_key\n os.environ[\"OPENROUTER_API_KEY\"] = openrouter_api_key\n\n bench = PillagerBench(args)\n bench.run()\n\nif __name__ == \"__main__\":\n main()\n"}}, "README.md": {"4cc901351bde4f0cbb798ba4b9b6397b0fc2fa79": {"t": 1785747261.1152532, "n": 4587, "h": "7a3bcbb58daf04d2", "c": "# PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments\n\nWe aim to investigate multi-agent systems in competitive team-vs-team scenarios in the Minecraft environment, and explore effective reinforcement learning techniques that enhance tactical play of Large Language Models.\n\nWelcome to PillagerBench, where the blocky world of Minecraft isn't just for fun and games, it's a warzone for the machines! Our benchmark suite is designed to push the boundaries of what virtual agents can achieve by adding dimensions of competition and team play to create complex and dynamic state spaces.\n\nCustomize your benchmark with our PillagerAgent extensible API, allowing you to add custom scenarios and new multi-agent systems.\n
\n\n---\n\n## Setup and Configuration\n\n### Requirements\n- **API Keys**: Obtain API keys from one or more of the following services:\n - OpenAI (for access to models like GPT-4o)\n - DeepSeek (for access to DeepSeek models)\n - OpenRouter (for access to a wide range of models)\n- **Ollama (optional)**: You have the option to use models from Ollama running locally.\n\n### Docker Installation Steps (recommended)\n\n1. Install [Docker](https://www.docker.com/) for your operating system.\n2. Clone the repository:\n```bash\ngit clone https://github.com/aialt/PillagerBench.git\ncd PillagerBench\n```\n3. Set-up your API keys:\n- Create a file named `api_keys.py` and add your API keys in this way:\n```py\nopenai_api_key = \"...\"\ndeepseek_api_key = \"...\"\nopenrouter_api_key = \"...\"\n```\n- Place this file in the root of the project directory.\n4. Build the Docker image:\n```bash\ndocker build -t PillagerBench .\n```\n5. Install NPM packages for Mineflayer:\n```bash\n./js_setup_docker.ps1\n``` \n6. Launch the Docker container:\n```bash\ndocker compose up -d\ndocker attach pillagerbench\n```\n7. Run a benchmark from config\n```bash\npython main.py -cn benchmark\n```\n\n### Local Installation Steps\n1. Install dependencies:\n - Python 3.10\n - Node.js 20 (with NPM)\n - Java 17\n2. Clone the repository:\n ```bash\n git clone https://github.com/aialt/PillagerBench.git\n ```\n3. Set-up your API keys:\n - Create a file named `api_keys.py` and add your API keys in this way:\n ```py\n openai_api_key = \"...\"\n deepseek_api_key = \"...\"\n openrouter_api_key = \"...\"\n ```\n - Place this file in the root of the project directory.\n4. Install NPM packages for Mineflayer:\n ```bash\n ./js_setup.sh\n ```\n5. Create a virtual environment:\n ```bash\n python -m venv venv\n source venv/bin/activate # On Windows, try venv\\Scripts\\activate\n ```\n6. Install the dependencies:\n ```bash\n pip install -r requirements.txt\n ```\n7. Run a benchmark from config\n ```bash\n python main.py -cn benchmark\n ```\n\n## QuickStart\n\n- Set-up [Hydra](https://hydra.cc/docs/intro/) test configs in the `configs` folder.\n- Run your test config:\n```bash\npython main.py -cn config_name\n```\n- Observe your test by joining the internal Minecraft server (requires Minecraft 1.19.4). The default address is `localhost:49172`, but the port is configurable.\n- Visualize results with `collate_results.py`: (requires editing the file to set options)\n```bash\npython collate_results.py\n```\n- Add additional test scenarios by adding classes to the `scenarios` folder that inherit from the `Scenario` base class. You can also add additional world saves to the `bench/mc_server` folder.\n- Add additional multi-agent systems by adding classes to the `agents` folder that inherit from the `Agent` base class.\n\n## Credits\n\nThis project is the Master's thesis of [Olivier Schipper](https://github.com/OliBomby), check out his other amazing projects!\n\n## Citation\n\nIf you find our work helpful, please leave us a star and cite our paper.\n\n```\n@INPROCEEDINGS{schipper2025pillagerbench,\n author={Schipper, Olivier and Zhang, Yudi and Du, Yali and Pechenizkiy, Mykola and Fang, Meng},\n booktitle={2025 IEEE Conference on Games (CoG)}, \n title={PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments}, \n year={2025},\n volume={},\n number={},\n pages={1-15},\n keywords={Adaptive learning;Games;Benchmark testing;Solids;Real-time systems;Cognition;Teamwork;Artificial intelligence;Multi-agent systems},\n doi={10.1109/CoG64752.2025.11114387},\n url={https://arxiv.org/abs/2509.06235}\n}\n```\n\n## License\n\nThis project is under the [MIT License](LICENSE).\n"}, "657e8e98b2c82eaf7c8eec3469c8dc14fbbbd513": {"t": 1785752850.0252488, "n": 4587, "h": "7a3bcbb58daf04d2", "c": "# PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments\n\nWe aim to investigate multi-agent systems in competitive team-vs-team scenarios in the Minecraft environment, and explore effective reinforcement learning techniques that enhance tactical play of Large Language Models.\n\nWelcome to PillagerBench, where the blocky world of Minecraft isn't just for fun and games, it's a warzone for the machines! Our benchmark suite is designed to push the boundaries of what virtual agents can achieve by adding dimensions of competition and team play to create complex and dynamic state spaces.\n\nCustomize your benchmark with our PillagerAgent extensible API, allowing you to add custom scenarios and new multi-agent systems.\n
\n\n---\n\n## Setup and Configuration\n\n### Requirements\n- **API Keys**: Obtain API keys from one or more of the following services:\n - OpenAI (for access to models like GPT-4o)\n - DeepSeek (for access to DeepSeek models)\n - OpenRouter (for access to a wide range of models)\n- **Ollama (optional)**: You have the option to use models from Ollama running locally.\n\n### Docker Installation Steps (recommended)\n\n1. Install [Docker](https://www.docker.com/) for your operating system.\n2. Clone the repository:\n```bash\ngit clone https://github.com/aialt/PillagerBench.git\ncd PillagerBench\n```\n3. Set-up your API keys:\n- Create a file named `api_keys.py` and add your API keys in this way:\n```py\nopenai_api_key = \"...\"\ndeepseek_api_key = \"...\"\nopenrouter_api_key = \"...\"\n```\n- Place this file in the root of the project directory.\n4. Build the Docker image:\n```bash\ndocker build -t PillagerBench .\n```\n5. Install NPM packages for Mineflayer:\n```bash\n./js_setup_docker.ps1\n``` \n6. Launch the Docker container:\n```bash\ndocker compose up -d\ndocker attach pillagerbench\n```\n7. Run a benchmark from config\n```bash\npython main.py -cn benchmark\n```\n\n### Local Installation Steps\n1. Install dependencies:\n - Python 3.10\n - Node.js 20 (with NPM)\n - Java 17\n2. Clone the repository:\n ```bash\n git clone https://github.com/aialt/PillagerBench.git\n ```\n3. Set-up your API keys:\n - Create a file named `api_keys.py` and add your API keys in this way:\n ```py\n openai_api_key = \"...\"\n deepseek_api_key = \"...\"\n openrouter_api_key = \"...\"\n ```\n - Place this file in the root of the project directory.\n4. Install NPM packages for Mineflayer:\n ```bash\n ./js_setup.sh\n ```\n5. Create a virtual environment:\n ```bash\n python -m venv venv\n source venv/bin/activate # On Windows, try venv\\Scripts\\activate\n ```\n6. Install the dependencies:\n ```bash\n pip install -r requirements.txt\n ```\n7. Run a benchmark from config\n ```bash\n python main.py -cn benchmark\n ```\n\n## QuickStart\n\n- Set-up [Hydra](https://hydra.cc/docs/intro/) test configs in the `configs` folder.\n- Run your test config:\n```bash\npython main.py -cn config_name\n```\n- Observe your test by joining the internal Minecraft server (requires Minecraft 1.19.4). The default address is `localhost:49172`, but the port is configurable.\n- Visualize results with `collate_results.py`: (requires editing the file to set options)\n```bash\npython collate_results.py\n```\n- Add additional test scenarios by adding classes to the `scenarios` folder that inherit from the `Scenario` base class. You can also add additional world saves to the `bench/mc_server` folder.\n- Add additional multi-agent systems by adding classes to the `agents` folder that inherit from the `Agent` base class.\n\n## Credits\n\nThis project is the Master's thesis of [Olivier Schipper](https://github.com/OliBomby), check out his other amazing projects!\n\n## Citation\n\nIf you find our work helpful, please leave us a star and cite our paper.\n\n```\n@INPROCEEDINGS{schipper2025pillagerbench,\n author={Schipper, Olivier and Zhang, Yudi and Du, Yali and Pechenizkiy, Mykola and Fang, Meng},\n booktitle={2025 IEEE Conference on Games (CoG)}, \n title={PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments}, \n year={2025},\n volume={},\n number={},\n pages={1-15},\n keywords={Adaptive learning;Games;Benchmark testing;Solids;Real-time systems;Cognition;Teamwork;Artificial intelligence;Multi-agent systems},\n doi={10.1109/CoG64752.2025.11114387},\n url={https://arxiv.org/abs/2509.06235}\n}\n```\n\n## License\n\nThis project is under the [MIT License](LICENSE).\n"}, "101ffb7d62236b35fcb21d2948c75fd27a00d9eb": {"t": 1785752851.3649287, "n": 3902, "h": "1940adb78f43e073", "c": "# PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments\n\nWe aim to investigate multi-agent systems in competitive team-vs-team scenarios in the Minecraft environment, and explore effective reinforcement learning techniques that enhance tactical play of Large Language Models.\n\nWelcome to PillagerBench, where the blocky world of Minecraft isn't just for fun and games, it's a warzone for the machines! Our benchmark suite is designed to push the boundaries of what virtual agents can achieve by adding dimensions of competition and team play to create complex and dynamic state spaces.\n\nCustomize your benchmark with our PillagerAgent extensible API, allowing you to add custom scenarios and new multi-agent systems.\n
\n\n---\n\n## Setup and Configuration\n\n### Requirements\n- **API Keys**: Obtain API keys from one or more of the following services:\n - OpenAI (for access to models like GPT-4o)\n - DeepSeek (for access to DeepSeek models)\n - OpenRouter (for access to a wide range of models)\n- **Ollama (optional)**: You have the option to use models from Ollama running locally.\n\n### Docker Installation Steps (recommended)\n\n1. Install [Docker](https://www.docker.com/) for your operating system.\n2. Clone the repository:\n```bash\ngit clone https://github.com/aialt/PillagerBench.git\ncd PillagerBench\n```\n3. Set-up your API keys:\n- Create a file named `api_keys.py` and add your API keys in this way:\n```py\nopenai_api_key = \"...\"\ndeepseek_api_key = \"...\"\nopenrouter_api_key = \"...\"\n```\n- Place this file in the root of the project directory.\n4. Build the Docker image:\n```bash\ndocker build -t PillagerBench .\n```\n5. Install NPM packages for Mineflayer:\n```bash\n./js_setup_docker.ps1\n``` \n6. Launch the Docker container:\n```bash\ndocker compose up -d\ndocker attach pillagerbench\n```\n7. Run a benchmark from config\n```bash\npython main.py -cn benchmark\n```\n\n### Local Installation Steps\n1. Install dependencies:\n - Python 3.10\n - Node.js 20 (with NPM)\n - Java 17\n2. Clone the repository:\n ```bash\n git clone https://github.com/aialt/PillagerBench.git\n ```\n3. Set-up your API keys:\n - Create a file named `api_keys.py` and add your API keys in this way:\n ```py\n openai_api_key = \"...\"\n deepseek_api_key = \"...\"\n openrouter_api_key = \"...\"\n ```\n - Place this file in the root of the project directory.\n4. Install NPM packages for Mineflayer:\n ```bash\n ./js_setup.sh\n ```\n5. Create a virtual environment:\n ```bash\n python -m venv venv\n source venv/bin/activate # On Windows, try venv\\Scripts\\activate\n ```\n6. Install the dependencies:\n ```bash\n pip install -r requirements.txt\n ```\n7. Run a benchmark from config\n ```bash\n python main.py -cn benchmark\n ```\n\n## QuickStart\n\n- Set-up [Hydra](https://hydra.cc/docs/intro/) test configs in the `configs` folder.\n- Run your test config:\n```bash\npython main.py -cn config_name\n```\n- Observe your test by joining the internal Minecraft server (requires Minecraft 1.19.4). The default address is `localhost:49172`, but the port is configurable.\n- Visualize results with `collate_results.py`: (requires editing the file to set options)\n```bash\npython collate_results.py\n```\n- Add additional test scenarios by adding classes to the `scenarios` folder that inherit from the `Scenario` base class. You can also add additional world saves to the `bench/mc_server` folder.\n- Add additional multi-agent systems by adding classes to the `agents` folder that inherit from the `Agent` base class.\n\n## Credits\n\nThis project is the Master's thesis of [Olivier Schipper](https://github.com/OliBomby), check out his other amazing projects!\n\n## Citation\n\nTODO\n\n## License\n\nThis project is under the [MIT License](LICENSE).\n"}, "7067dfefc8e1c7bac865684301acfeb174cdbfb0": {"t": 1785752852.8600233, "n": 15, "h": "b0a32443b056fb06", "c": "# PillagerBench"}}}}
\n \n