hack/tools/scripts/llm-key-hunter/results/_cache/content/3bai233_genai-group3.json

{"repo": "3bai233/genai-group3", "files": {"code/Agent/AgentS3+BAMI/README.md": {"e5d0e5df4595689b19d28fb92edbf1bd5cd61599": {"t": 1785768899.3346431, "n": 5091, "h": "ae705c5ea82343b2", "c": "# AgentS3-BAMI\n\n## 1. 环境依赖\n\n### 硬件要求\n| 组件 | 配置 | 说明 |\n|------|------|------|\n| GPU | NVIDIA GPU（24GB+ 显存） | BAMI 和 vLLM 推理加速 |\n| 内存 | 32GB+ | 8 个 Docker 容器并行 |\n| 存储 | 100GB+ | 模型文件 + 虚拟机镜像 |\n\n### 软件要求\n| 软件 | 版本 | 说明 |\n|------|------|------|\n| Python | 3.10+ | 运行环境 |\n| Docker | 20.10+ | OSWorld 容器运行 |\n| CUDA | 11.8+ | GPU 加速 |\n| NVIDIA Driver | 525+ | GPU 驱动 |\n\n### 外部服务\n| 服务 | 端口 | GPU | 说明 |\n|------|------|-----|------|\n| vLLM Server | 8000 | GPU 6,7 | UI-TARS-1.5-7B（fallback 定位） |\n| BAMI Server | 8001 | GPU 4 | Holo2-8B + 裁剪判别模型 |\n| Docker | - | CPU | OSWorld 容器（8 并行） |\n\n### Python 依赖\n\n| 依赖包 | 用途 |\n| :--- | :--- |\n| openai>=1.0.0 | GPT-4o 规划模型调用 |\n| pyautogui>=0.9.54 | GUI 鼠标键盘控制 |\n| docker>=6.1.0 | OSWorld 容器管理 |\n| vllm>=0.4.0 | UI-TARS 推理服务 |\n| torch>=2.0.0 | 深度学习框架 |\n| pillow>=10.0.0 | 图像处理 |\n| pytesseract>=0.3.10 | OCR 文字识别 |\n| anthropic>=0.18.0 | Claude 模型支持 |\n| backoff>=2.2.0 | API 重试机制 |\n\n### 模型文件\n\n| 模型 | 路径 | 大小 |\n| :--- | :--- | :--- |\n| UI-TARS-1.5-7B | vLLM 加载 | ~14GB |\n| Holo2-8B | /data/models/Holo2-8B | ~16GB |\n| 裁剪判别模型 | /data/models/two_box_judge_qwen3_8b/checkpoints/checkpoint-2712 | ~16GB |\n\n### 数据集\n\n| 文件 | 路径 |\n| :--- | :--- |\n| OSWorld 测试集 | evaluation_examples/test_nogdrive.json |\n| Ubuntu 虚拟机镜像 | docker_vm_data/Ubuntu.qcow2 |\n| Docker 镜像 | happysixd/osworld-docker |\n\n## 2. 代码仓库\n\n```\ns3_bami/\n├── __init__.py                         # 包初始化文件\n├── agents/                             # Agent 核心实现\n│   ├── __init__.py\n│   ├── agent_s.py                      # AgentS3 主代理类（规划+执行）\n│   ├── worker.py                       # Worker 执行代理（动作生成）\n│   ├── grounding.py                    # Grounding 层（BAMI + UI-TARS 调用）\n│   └── code_agent.py                   # 代码执行代理（Python/Bash）\n├── core/                               # 核心引擎\n│   ├── __init__.py\n│   ├── engine.py                       # LLM 引擎（OpenAI/Anthropic/vLLM）\n│   ├── mllm.py                         # 多模态 Agent 封装\n│   └── module.py                       # 基础模块类\n├── memory/                             # 记忆模块\n│   ├── __init__.py\n│   └── procedural_memory.py            # Prompt 模板库（所有系统提示词）\n├── bbon/                               # 行为分析模块\n│   ├── __init__.py\n│   ├── behavior_narrator.py            # 行为叙述器（动作前后对比）\n│   └── comparative_judge.py            # 比较评测器（多轨迹对比）\n├── utils/                              # 工具函数\n│   ├── __init__.py\n│   ├── common_utils.py                 # 通用工具（LLM调用、代码解析）\n│   ├── formatters.py                   # 响应格式检查器\n│   └── local_env.py                    # 本地代码执行环境\n└── cli_app.py                          # 命令行入口（交互式/单任务）\n```\n\n## 3.  如何运行\n\n### 1. 启动 Grounding 服务\n\n#### 启动 vLLM Server（GPU 6,7）- UI-TARS fallback\nCUDA_VISIBLE_DEVICES=6,7 vllm serve UI-TARS-1.5-7B \\\n  --host 127.0.0.1 \\\n  --port 8000 \\\n  --served-model-name ByteDance-Seed/UI-TARS-1.5-7B \\\n  --max-model-len 32768 \\\n  --tensor-parallel-size 2\n  \n#### 启动 BAMI Server（GPU 4）- 主要定位服务\nCUDA_VISIBLE_DEVICES=4 python scripts/bami_grounding_server.py \\\n  --holo2_model_path /data/models/Holo2-8B \\\n  --judge_model_path /data/models/two_box_judge_qwen3_8b/checkpoints/checkpoint-2712 \\\n  --port 8001\n  \n### 2. 运行 AgentS3-BAMI\n\n#### 交互式模式\npython cli_app.py \\\n  --provider openai \\\n  --model gpt-4o \\\n  --ground_provider openai \\\n  --ground_url \"http://localhost:8000/v1\" \\\n  --ground_api_key EMPTY \\\n  --ground_model ByteDance-Seed/UI-TARS-1.5-7B \\\n  --grounding_width 1920 \\\n  --grounding_height 1080\n  \n运行后输入任务，例如：\ntext\nQuery: Open Chrome and search for AgentS3\n\n#### 单任务模式\npython cli_app.py \\\n  --provider openai \\\n  --model gpt-4o \\\n  --model_url \"https://www.dmxapi.cn/v1\" \\\n  --model_api_key \"your-api-key\" \\\n  --ground_provider openai \\\n  --ground_url \"http://localhost:8000/v1\" \\\n  --ground_api_key EMPTY \\\n  --ground_model ByteDance-Seed/UI-TARS-1.5-7B \\\n  --grounding_width 1920 \\\n  --grounding_height 1080 \\\n  --task \"Open Chrome and search for AgentS3\"\n  \n#### 批量测试（使用监控脚本）\n##### 后台运行完整监控（推荐）\nnohup bash scripts/bash/monitor_agents3_bami.sh > logs/monitor_agents3_bami.log 2>&1 &\n\n#### 查看状态\ntail -f logs/monitor_agents3_bami.log\n\n### 3. 配置文件方式\n\n也可以通过 `args.json` 配置文件运行：\n\n```bash\npython cli_app.py --config args.json\n```\n\n`args.json` 配置示例：\n\n```json\n{\n  \"path_to_vm\": \"docker_vm_data/Ubuntu.qcow2\",\n  \"provider_name\": \"docker\",\n  \"headless\": true,\n  \"action_space\": \"pyautogui\",\n  \"observation_type\": \"screenshot\",\n  \"num_envs\": 8,\n  \"screen_width\": 1920,\n  \"screen_height\": 1080,\n  \"sleep_after_execution\": 3.0,\n  \"max_steps\": 50,\n  \"max_trajectory_length\": 8,\n  \"domain\": \"all\",\n  \"test_all_meta_path\": \"evaluation_examples/test_nogdrive.json\",\n  \"test_config_base_dir\": \"evaluation_examples\",\n  \"result_dir\": \"results_agents3_bami\",\n  \"model_provider\": \"openai\",\n  \"model\": \"gpt-4o\",\n  \"model_url\": \"https://www.dmxapi.cn/v1\",\n  \"model_api_key\": \"sk-YGmlJCXmOcpOiHcYQCL91RwuHI6MYYvTXk8XevZaYBAFWERh\",\n  \"model_temperature\": null,\n  \"ground_provider\": \"openai\",\n  \"ground_url\": \"http://localhost:8000/v1\",\n  \"ground_api_key\": \"EMPTY\",\n  \"ground_model\": \"ByteDance-Seed/UI-TARS-1.5-7B\",\n  \"grounding_width\": 1920,\n  \"grounding_height\": 1080,\n  \"bami_server_url\": \"http://127.0.0.1:8001\"\n}\n```\n\n\n"}}, "log/Agent/baseline/agents3/results/pyautogui/screenshot/gpt-4o/args.json": {"e5d0e5df4595689b19d28fb92edbf1bd5cd61599": {"t": 1785768911.0608013, "n": 909, "h": "51d0723515c8d906", "c": "{\n    \"path_to_vm\": \"docker_vm_data/Ubuntu.qcow2\",\n    \"provider_name\": \"docker\",\n    \"headless\": true,\n    \"action_space\": \"pyautogui\",\n    \"observation_type\": \"screenshot\",\n    \"num_envs\": 5,\n    \"screen_width\": 1920,\n    \"screen_height\": 1080,\n    \"sleep_after_execution\": 3.0,\n    \"max_steps\": 50,\n    \"max_trajectory_length\": 8,\n    \"domain\": \"all\",\n    \"test_all_meta_path\": \"evaluation_examples/test_nogdrive.json\",\n    \"test_config_base_dir\": \"evaluation_examples\",\n    \"result_dir\": \"./results_agents3\",\n    \"model_provider\": \"openai\",\n    \"model\": \"gpt-4o\",\n    \"model_url\": \"https://www.dmxapi.cn/v1\",\n    \"model_api_key\": \"<REDACTED_API_KEY>\",\n    \"model_temperature\": null,\n    \"ground_provider\": \"openai\",\n    \"ground_url\": \"http://localhost:8000/v1\",\n    \"ground_api_key\": \"EMPTY\",\n    \"ground_model\": \"ByteDance-Seed/UI-TARS-1.5-7B\",\n    \"grounding_width\": 1920,\n    \"grounding_height\": 1080\n}"}}, "code/Agent/baseline/agents3/gui_agents_s3/core/mllm.py": {"e5d0e5df4595689b19d28fb92edbf1bd5cd61599": {"t": 1786068482.5895712, "n": 14533, "h": "e76ac466290f77cc", "c": "import base64\n\nimport numpy as np\n\nfrom gui_agents.s3.core.engine import (\n    LMMEngineAnthropic,\n    LMMEngineAzureOpenAI,\n    LMMEngineHuggingFace,\n    LMMEngineOpenAI,\n    LMMEngineOpenRouter,\n    LMMEngineParasail,\n    LMMEnginevLLM,\n    LMMEngineGemini,\n)\n\n\nclass LMMAgent:\n    def __init__(self, engine_params=None, system_prompt=None, engine=None):\n        if engine is None:\n            if engine_params is not None:\n                engine_type = engine_params.get(\"engine_type\")\n                if engine_type == \"openai\":\n                    self.engine = LMMEngineOpenAI(**engine_params)\n                elif engine_type == \"anthropic\":\n                    self.engine = LMMEngineAnthropic(**engine_params)\n                elif engine_type == \"azure\":\n                    self.engine = LMMEngineAzureOpenAI(**engine_params)\n                elif engine_type == \"vllm\":\n                    self.engine = LMMEnginevLLM(**engine_params)\n                elif engine_type == \"huggingface\":\n                    self.engine = LMMEngineHuggingFace(**engine_params)\n                elif engine_type == \"gemini\":\n                    self.engine = LMMEngineGemini(**engine_params)\n                elif engine_type == \"open_router\":\n                    self.engine = LMMEngineOpenRouter(**engine_params)\n                elif engine_type == \"parasail\":\n                    self.engine = LMMEngineParasail(**engine_params)\n                elif engine_type == \"ollama\":\n                    # Reuse LMMEngineOpenAI for Ollama\n                    if not engine_params.get(\"base_url\"):\n                        import os\n\n                        base_url = os.getenv(\"OLLAMA_HOST\")\n                        if base_url:\n                            if not base_url.endswith(\"/v1\"):\n                                base_url = base_url.rstrip(\"/\") + \"/v1\"\n                            engine_params[\"base_url\"] = base_url\n                        else:\n                            # RAISE ERROR instead of default\n                            raise ValueError(\n                                \"Ollama endpoint must be provided via 'base_url' parameter or 'OLLAMA_HOST' environment variable.\"\n                            )\n                    if not engine_params.get(\"api_key\"):\n                        engine_params[\"api_key\"] = \"ollama\"\n                    self.engine = LMMEngineOpenAI(**engine_params)\n                elif engine_type == \"deepseek\":\n                    if \"base_url\" not in engine_params:\n                        import os\n\n                        base_url = os.getenv(\"DEEPSEEK_ENDPOINT_URL\")\n                        if not base_url:\n                            base_url = \"https://api.deepseek.com\"\n                        if not base_url.endswith(\"/v1\"):\n                            base_url = base_url.rstrip(\"/\") + \"/v1\"\n                        engine_params[\"base_url\"] = base_url\n\n                    if not engine_params.get(\"api_key\"):\n                        import os\n\n                        api_key = os.getenv(\"DEEPSEEK_API_KEY\")\n                        if not api_key:\n                            raise ValueError(\n                                \"DeepSeek API key must be provided via 'api_key' parameter or 'DEEPSEEK_API_KEY' environment variable.\"\n                            )\n                        engine_params[\"api_key\"] = api_key\n\n                    self.engine = LMMEngineOpenAI(**engine_params)\n                elif engine_type == \"qwen\":\n                    if not engine_params.get(\"base_url\"):\n                        import os\n\n                        base_url = os.getenv(\"QWEN_ENDPOINT_URL\")\n                        if not base_url:\n                            base_url = (\n                                \"https://dashscope.aliyuncs.com/compatible-mode/v1\"\n                            )\n                        if not base_url.endswith(\"/v1\"):\n                            base_url = base_url.rstrip(\"/\") + \"/v1\"\n                        engine_params[\"base_url\"] = base_url\n\n                    if not engine_params.get(\"api_key\"):\n                        import os\n\n                        api_key = os.getenv(\"QWEN_API_KEY\")\n                        if not api_key:\n                            raise ValueError(\n                                \"Qwen API key must be provided via 'api_key' parameter or 'QWEN_API_KEY' environment variable.\"\n                            )\n                        engine_params[\"api_key\"] = api_key\n                    self.engine = LMMEngineOpenAI(**engine_params)\n                else:\n                    raise ValueError(f\"engine_type '{engine_type}' is not supported\")\n            else:\n                raise ValueError(\"engine_params must be provided\")\n        else:\n            self.engine = engine\n\n        self.messages = []  # Empty messages\n\n        if system_prompt:\n            self.add_system_prompt(system_prompt)\n        else:\n            self.add_system_prompt(\"You are a helpful assistant.\")\n\n    def encode_image(self, image_content):\n        # if image_content is a path to an image file, check type of the image_content to verify\n        if isinstance(image_content, str):\n            with open(image_content, \"rb\") as image_file:\n                return base64.b64encode(image_file.read()).decode(\"utf-8\")\n        else:\n            return base64.b64encode(image_content).decode(\"utf-8\")\n\n    def reset(\n        self,\n    ):\n\n        self.messages = [\n            {\n                \"role\": \"system\",\n                \"content\": [{\"type\": \"text\", \"text\": self.system_prompt}],\n            }\n        ]\n\n    def add_system_prompt(self, system_prompt):\n        self.system_prompt = system_prompt\n        if len(self.messages) > 0:\n            self.messages[0] = {\n                \"role\": \"system\",\n                \"content\": [{\"type\": \"text\", \"text\": self.system_prompt}],\n            }\n        else:\n            self.messages.append(\n                {\n                    \"role\": \"system\",\n                    \"content\": [{\"type\": \"text\", \"text\": self.system_prompt}],\n                }\n            )\n\n    def remove_message_at(self, index):\n        \"\"\"Remove a message at a given index\"\"\"\n        if index < len(self.messages):\n            self.messages.pop(index)\n\n    def replace_message_at(\n        self, index, text_content, image_content=None, image_detail=\"high\"\n    ):\n        \"\"\"Replace a message at a given index\"\"\"\n        if index < len(self.messages):\n            self.messages[index] = {\n                \"role\": self.messages[index][\"role\"],\n                \"content\": [{\"type\": \"text\", \"text\": text_content}],\n            }\n            if image_content:\n                base64_image = self.encode_image(image_content)\n                self.messages[index][\"content\"].append(\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": f\"data:image/png;base64,{base64_image}\",\n                            \"detail\": image_detail,\n                        },\n                    }\n                )\n\n    def add_message(\n        self,\n        text_content,\n        image_content=None,\n        role=None,\n        image_detail=\"high\",\n        put_text_last=False,\n    ):\n        \"\"\"Add a new message to the list of messages\"\"\"\n\n        # API-style inference from OpenAI and AzureOpenAI\n        if isinstance(\n            self.engine,\n            (\n                LMMEngineOpenAI,\n                LMMEngineAzureOpenAI,\n                LMMEngineHuggingFace,\n                LMMEngineGemini,\n                LMMEngineOpenRouter,\n                LMMEngineParasail,\n            ),\n        ):\n            # infer role from previous message\n            if role != \"user\":\n                if self.messages[-1][\"role\"] == \"system\":\n                    role = \"user\"\n                elif self.messages[-1][\"role\"] == \"user\":\n                    role = \"assistant\"\n                elif self.messages[-1][\"role\"] == \"assistant\":\n                    role = \"user\"\n\n            message = {\n                \"role\": role,\n                \"content\": [{\"type\": \"text\", \"text\": text_content}],\n            }\n\n            if isinstance(image_content, np.ndarray) or image_content:\n                # Check if image_content is a list or a single image\n                if isinstance(image_content, list):\n                    # If image_content is a list of images, loop through each image\n                    for image in image_content:\n                        base64_image = self.encode_image(image)\n                        message[\"content\"].append(\n                            {\n                                \"type\": \"image_url\",\n                                \"image_url\": {\n                                    \"url\": f\"data:image/png;base64,{base64_image}\",\n                                    \"detail\": image_detail,\n                                },\n                            }\n                        )\n                else:\n                    # If image_content is a single image, handle it directly\n                    base64_image = self.encode_image(image_content)\n                    message[\"content\"].append(\n                        {\n                            \"type\": \"image_url\",\n                            \"image_url\": {\n                                \"url\": f\"data:image/png;base64,{base64_image}\",\n                                \"detail\": image_detail,\n                            },\n                        }\n                    )\n\n            # Rotate text to be the last message if desired\n            if put_text_last:\n                text_content = message[\"content\"].pop(0)\n                message[\"content\"].append(text_content)\n\n            self.messages.append(message)\n\n        # For API-style inference from Anthropic\n        elif isinstance(self.engine, LMMEngineAnthropic):\n            # infer role from previous message\n            if role != \"user\":\n                if self.messages[-1][\"role\"] == \"system\":\n                    role = \"user\"\n                elif self.messages[-1][\"role\"] == \"user\":\n                    role = \"assistant\"\n                elif self.messages[-1][\"role\"] == \"assistant\":\n                    role = \"user\"\n\n            message = {\n                \"role\": role,\n                \"content\": [{\"type\": \"text\", \"text\": text_content}],\n            }\n\n            if image_content:\n                # Check if image_content is a list or a single image\n                if isinstance(image_content, list):\n                    # If image_content is a list of images, loop through each image\n                    for image in image_content:\n                        base64_image = self.encode_image(image)\n                        message[\"content\"].append(\n                            {\n                                \"type\": \"image\",\n                                \"source\": {\n                                    \"type\": \"base64\",\n                                    \"media_type\": \"image/png\",\n                                    \"data\": base64_image,\n                                },\n                            }\n                        )\n                else:\n                    # If image_content is a single image, handle it directly\n                    base64_image = self.encode_image(image_content)\n                    message[\"content\"].append(\n                        {\n                            \"type\": \"image\",\n                            \"source\": {\n                                \"type\": \"base64\",\n                                \"media_type\": \"image/png\",\n                                \"data\": base64_image,\n                            },\n                        }\n                    )\n            self.messages.append(message)\n\n        # Locally hosted vLLM model inference\n        elif isinstance(self.engine, LMMEnginevLLM):\n            # infer role from previous message\n            if role != \"user\":\n                if self.messages[-1][\"role\"] == \"system\":\n                    role = \"user\"\n                elif self.messages[-1][\"role\"] == \"user\":\n                    role = \"assistant\"\n                elif self.messages[-1][\"role\"] == \"assistant\":\n                    role = \"user\"\n\n            message = {\n                \"role\": role,\n                \"content\": [{\"type\": \"text\", \"text\": text_content}],\n            }\n\n            if image_content:\n                # Check if image_content is a list or a single image\n                if isinstance(image_content, list):\n                    # If image_content is a list of images, loop through each image\n                    for image in image_content:\n                        base64_image = self.encode_image(image)\n                        message[\"content\"].append(\n                            {\n                                \"type\": \"image_url\",\n                                \"image_url\": {\n                                    \"url\": f\"data:image;base64,{base64_image}\"\n                                },\n                            }\n                        )\n                else:\n                    # If image_content is a single image, handle it directly\n                    base64_image = self.encode_image(image_content)\n                    message[\"content\"].append(\n                        {\n                            \"type\": \"image_url\",\n                            \"image_url\": {\"url\": f\"data:image;base64,{base64_image}\"},\n                        }\n                    )\n\n            self.messages.append(message)\n        else:\n            raise ValueError(\"engine_type is not supported\")\n\n    def get_response(\n        self,\n        user_message=None,\n        messages=None,\n        temperature=0.0,\n        max_new_tokens=None,\n        use_thinking=False,\n        **kwargs,\n    ):\n        \"\"\"Generate the next response based on previous messages\"\"\"\n        if messages is None:\n            messages = self.messages\n        if user_message:\n            messages.append(\n                {\"role\": \"user\", \"content\": [{\"type\": \"text\", \"text\": user_message}]}\n            )\n\n        # Regular generation\n        if use_thinking:\n            return self.engine.generate_with_thinking(\n                messages,\n                temperature=temperature,\n                max_new_tokens=max_new_tokens,\n                **kwargs,\n            )\n\n        return self.engine.generate(\n            messages,\n            temperature=temperature,\n            max_new_tokens=max_new_tokens,\n            **kwargs,\n        )\n"}}}}