1 line
29 KiB
JSON
1 line
29 KiB
JSON
{"repo": "aipythoner/pymidscene", "files": {"docs/MODELS.md": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674250.7292187, "n": 2437, "h": "3c738f8b0f15abe1", "c": "# 模型配置指南 - Model Configuration\n\nPyMidscene 支持多种视觉大模型，本文档介绍各模型的配置方法。\n\n## 支持的模型\n\n| 模型 | 提供商 | 模型家族 | 推荐场景 |\n|------|--------|----------|----------|\n| doubao-seed-1.6 | 字节跳动/火山引擎 | doubao-vision | 国内用户首选 |\n| qwen-vl-max | 阿里云 | qwen2.5-vl | 国内备选 |\n| gpt-4-vision | OpenAI | openai | 海外用户 |\n| claude-3 | Anthropic | claude | 海外用户 |\n\n## 配置方式\n\n### 方式一：环境变量（推荐）\n\n创建 `.env` 文件：\n\n```bash\n# 豆包模型\nMIDSCENE_MODEL_NAME=your-endpoint-id\nMIDSCENE_MODEL_API_KEY=your-api-key\nMIDSCENE_MODEL_BASE_URL=https://ark.cn-beijing.volces.com/api/v3\nMIDSCENE_MODEL_FAMILY=doubao-vision\n```\n\n代码中加载：\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nagent = PlaywrightAgent(page)\n```\n\n### 方式二：代码配置\n\n```python\nimport os\n\nos.environ[\"MIDSCENE_MODEL_NAME\"] = \"your-endpoint-id\"\nos.environ[\"MIDSCENE_MODEL_API_KEY\"] = \"your-api-key\"\nos.environ[\"MIDSCENE_MODEL_BASE_URL\"] = \"https://...\"\nos.environ[\"MIDSCENE_MODEL_FAMILY\"] = \"doubao-vision\"\n\nagent = PlaywrightAgent(page)\n```\n\n### 方式三：model_config 参数\n\n```python\nconfig = {\n \"MIDSCENE_MODEL_NAME\": \"your-endpoint-id\",\n \"MIDSCENE_MODEL_API_KEY\": \"your-api-key\",\n \"MIDSCENE_MODEL_BASE_URL\": \"https://...\",\n \"MIDSCENE_MODEL_FAMILY\": \"doubao-vision\",\n}\n\nagent = PlaywrightAgent(page, model_config=config)\n```\n\n## 各模型详细配置\n\n### 豆包 Doubao（火山引擎）\n\n1. 访问 [火山引擎控制台](https://console.volcengine.com/ark)\n2. 创建推理接入点，获取 Endpoint ID\n3. 获取 API Key\n\n```bash\nMIDSCENE_MODEL_NAME=ep-20250122xxxxxx # 你的 Endpoint ID\nMIDSCENE_MODEL_API_KEY=your-api-key\nMIDSCENE_MODEL_BASE_URL=https://ark.cn-beijing.volces.com/api/v3\nMIDSCENE_MODEL_FAMILY=doubao-vision\n```\n\n### 千问 Qwen（阿里云）\n\n1. 访问 [阿里云 DashScope](https://dashscope.console.aliyun.com/)\n2. 开通模型服务，获取 API Key\n\n```bash\nMIDSCENE_MODEL_NAME=qwen-vl-max\nMIDSCENE_MODEL_API_KEY=sk-xxxxxx\nMIDSCENE_MODEL_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1\nMIDSCENE_MODEL_FAMILY=qwen2.5-vl\n```\n\n### OpenAI GPT-4V\n\n```bash\nMIDSCENE_MODEL_NAME=gpt-4-vision-preview\nMIDSCENE_MODEL_API_KEY=sk-xxxxxx\nMIDSCENE_MODEL_BASE_URL=https://api.openai.com/v1\nMIDSCENE_MODEL_FAMILY=openai\n```\n\n## 坐标系统差异\n\n不同模型返回的坐标格式不同，PyMidscene 会自动处理：\n\n| 模型家族 | 坐标格式 | 说明 |\n|----------|----------|------|\n| doubao-vision | 归一化 0-1000 | 需要 `x * width / 1000` |\n| qwen2.5-vl | 像素坐标 | 直接使用 |\n| openai | 像素坐标 | 直接使用 |\n| gemini | [y1, x1, y2, x2] | 需要交换顺序 |\n\n## 常见问题\n\n### Q: 为什么点击位置不准确？\n\nA: 检查 `MIDSCENE_MODEL_FAMILY` 是否正确设置。不同模型返回的坐标格式不同，需要正确的模型家族才能正确转换坐标。\n\n### Q: 如何切换模型？\n\nA: 修改环境变量或 `.env` 文件即可，无需修改代码。\n"}}, "README_CN.md": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674253.6049097, "n": 5390, "h": "eb1472c5827567ea", "c": "# PyMidscene\n\n<p align=\"center\">\n <img src=\"https://midscenejs.com/midscene_with_text_light.png\" alt=\"Midscene Logo\" width=\"300\">\n</p>\n\n<p align=\"center\">\n <strong>Midscene.js 的 Python SDK - 使用自然语言进行 AI 驱动的 UI 自动化</strong>\n</p>\n\n<p align=\"center\">\n <a href=\"https://github.com/AIPythoner/pymidscene/blob/main/LICENSE\"><img src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"License\"></a>\n <a href=\"https://www.python.org/downloads/\"><img src=\"https://img.shields.io/badge/python-3.10+-blue.svg\" alt=\"Python\"></a>\n <a href=\"https://midscenejs.com\"><img src=\"https://img.shields.io/badge/docs-midscenejs.com-green.svg\" alt=\"Docs\"></a>\n</p>\n\n<p align=\"center\">\n <a href=\"#特性\">特性</a> |\n <a href=\"#安装\">安装</a> |\n <a href=\"#快速开始\">快速开始</a> |\n <a href=\"#文档\">文档</a> |\n <a href=\"./README.md\">English</a>\n</p>\n\n---\n\n## 什么是 PyMidscene？\n\nPyMidscene 是 [Midscene.js](https://midscenejs.com) 的 Python 版本 - 一个 AI 驱动的 UI 自动化框架。它允许你使用自然语言而不是 CSS 选择器或 XPath 来控制浏览器。\n\n**告别脆弱的选择器！** 只需描述你想要点击、输入或提取的内容：\n\n```python\n# 不再需要: page.click(\"#submit-btn-primary\")\nawait agent.ai_click(\"蓝色的提交按钮\")\n\n# 不再需要: page.fill(\"input[name='email']\", \"test@example.com\") \nawait agent.ai_input(\"邮箱输入框\", \"test@example.com\")\n\n# 使用自然语言提取结构化数据\nresult = await agent.ai_query({\n \"title\": \"页面标题\",\n \"price\": \"商品价格，数字类型\"\n})\n```\n\n## 特性\n\n- **自然语言自动化** - 用中文/英文描述元素，无需选择器\n- **多模型支持** - 支持豆包、千问、GPT-4V、Claude 等视觉大模型\n- **Playwright 集成** - 与 Playwright 无缝集成进行网页自动化\n- **XPath 缓存** - 与 Midscene.js 格式兼容的智能缓存系统\n- **可视化报告** - 生成精美的 HTML 报告用于调试和分享\n- **类型安全** - 完整的类型提示，提供优秀的 IDE 支持\n\n## 安装\n\n```bash\npip install pymidscene\n\n# 安装 Playwright 浏览器\nplaywright install chromium\n```\n\n或使用 Poetry：\n\n```bash\npoetry add pymidscene\nplaywright install chromium\n```\n\n## 快速开始\n\n### 1. 配置 API 密钥\n\n```bash\n# 豆包模型（推荐国内用户使用）\nexport MIDSCENE_MODEL_NAME=\"doubao-seed-1-6-251015\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://ark.cn-beijing.volces.com/api/v3\"\nexport MIDSCENE_MODEL_FAMILY=\"doubao-vision\"\n\n# 千问模型\nexport MIDSCENE_MODEL_NAME=\"qwen-vl-max\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://dashscope.aliyuncs.com/compatible-mode/v1\"\nexport MIDSCENE_MODEL_FAMILY=\"qwen2.5-vl\"\n```\n\n### 2. 编写自动化脚本\n\n```python\nimport asyncio\nimport os\nfrom playwright.async_api import async_playwright\nfrom pymidscene import PlaywrightAgent\n\nasync def main():\n # 配置模型（或使用环境变量）\n os.environ[\"MIDSCENE_MODEL_NAME\"] = \"doubao-seed-1-6-251015\"\n os.environ[\"MIDSCENE_MODEL_API_KEY\"] = \"your-api-key\"\n os.environ[\"MIDSCENE_MODEL_BASE_URL\"] = \"https://ark.cn-beijing.volces.com/api/v3\"\n os.environ[\"MIDSCENE_MODEL_FAMILY\"] = \"doubao-vision\"\n\n async with async_playwright() as p:\n browser = await p.chromium.launch(headless=False)\n page = await browser.new_page()\n\n # 创建 agent，可选启用缓存\n agent = PlaywrightAgent(page, cache_id=\"my_task\")\n\n # 导航到页面\n await page.goto(\"https://www.baidu.com\")\n\n # 使用自然语言进行交互\n await agent.ai_click(\"搜索输入框\")\n await agent.ai_input(\"搜索框\", \"Python 自动化\")\n await agent.ai_click(\"百度一下按钮\")\n\n # 提取数据\n result = await agent.ai_query({\n \"results_count\": \"搜索结果数量\",\n \"first_title\": \"第一条结果的标题\"\n })\n print(f\"结果: {result}\")\n\n # 断言页面状态\n await agent.ai_assert(\"页面显示了搜索结果\")\n\n # 生成可视化报告\n report_path = agent.finish()\n print(f\"报告已保存到: {report_path}\")\n\n await browser.close()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n## 文档\n\n### 核心 API\n\n| 方法 | 描述 |\n|------|------|\n| `ai_click(description)` | 使用自然语言描述点击元素 |\n| `ai_input(description, text)` | 在输入框中输入文本 |\n| `ai_locate(description)` | 定位元素并返回坐标 |\n| `ai_query(schema)` | 从页面提取结构化数据 |\n| `ai_assert(assertion)` | 断言某个条件为真 |\n| `ai_action(task)` | 执行复杂任务，AI 自动规划-执行-重规划循环 |\n| `ai_wait_for(assertion, timeout)` | 等待页面满足某个条件（轮询） |\n| `ai_scroll(direction, distance)` | AI 辅助滚动页面 |\n| `finish()` | 生成 HTML 报告并返回路径 |\n\n### 支持的模型\n\n| 模型 | 模型家族 | 提供商 |\n|------|----------|--------|\n| doubao-seed-1-6-251015 | doubao-vision | 字节跳动/火山引擎 |\n| qwen-vl-max | qwen2.5-vl | 阿里巴巴 |\n| gpt-4-vision-preview | openai | OpenAI |\n| claude-3-opus | claude | Anthropic |\n\n### 缓存系统\n\nPyMidscene 使用与 Midscene.js 兼容的基于 XPath 的缓存：\n\n```yaml\n# midscene_run/cache/my_task.cache.yaml\nmidsceneVersion: 1.0.0\ncacheId: my_task\ncaches:\n - type: locate\n prompt: 登录按钮\n cache:\n xpaths:\n - /html/body/div[1]/button[1]\n```\n\n这意味着：\n- 缓存文件可在 JS 和 Python 版本之间互换使用\n- 基于 XPath 的缓存在不同窗口大小下都能正常工作\n- 当元素移动时会自动使缓存失效\n\n## 示例\n\n查看 [examples/](examples/) 目录：\n\n- `basic_usage.py` - 入门示例\n- `login_demo.py` - 带可视化报告的登录自动化\n- `login_demo.html` - 登录演示的测试页面\n\n## 相关项目\n\n本项目是 [Midscene.js](https://midscenejs.com) 的 Python 实现。\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - 原版 JavaScript 版本\n- [官方文档](https://midscenejs.com)\n- [Awesome Midscene](https://midscenejs.com/awesome-midscene) - 社区项目\n\n## 贡献\n\n欢迎贡献！请查看 [CONTRIBUTING.md](CONTRIBUTING.md) 了解指南。\n\n```bash\n# 开发环境设置\ngit clone https://github.com/AIPythoner/pymidscene.git\ncd pymidscene\npip install -e \".[dev]\"\n\n# 运行测试\npytest\n\n# 格式化代码\nblack pymidscene tests\n```\n\n## 许可证\n\nMIT License - 详见 [LICENSE](LICENSE) 文件。\n\n## 致谢\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - 字节跳动开发的原版 JavaScript 框架\n- [Playwright](https://playwright.dev/) - 浏览器自动化库\n\n## 交流群\n\nQQ 群：**2156022195**\n\n---\n\n<p align=\"center\">\n 由 PyMidscene 社区用 ❤️ 打造\n</p>\n"}}, "README.md": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674254.2864528, "n": 8957, "h": "2aec168769f74c16", "c": "# PyMidscene\n\n<p align=\"center\">\n <img src=\"https://midscenejs.com/midscene_with_text_light.png\" alt=\"Midscene Logo\" width=\"300\">\n</p>\n\n<p align=\"center\">\n <strong>Python SDK for Midscene.js - AI-powered UI automation using natural language</strong>\n</p>\n\n<p align=\"center\">\n <a href=\"https://github.com/AIPythoner/pymidscene/blob/main/LICENSE\"><img src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"License\"></a>\n <a href=\"https://www.python.org/downloads/\"><img src=\"https://img.shields.io/badge/python-3.10+-blue.svg\" alt=\"Python\"></a>\n <a href=\"https://midscenejs.com\"><img src=\"https://img.shields.io/badge/docs-midscenejs.com-green.svg\" alt=\"Docs\"></a>\n</p>\n\n<p align=\"center\">\n <a href=\"#features\">Features</a> |\n <a href=\"#installation\">Installation</a> |\n <a href=\"#quick-start\">Quick Start</a> |\n <a href=\"#documentation\">Documentation</a> |\n <a href=\"./README_CN.md\">中文文档</a>\n</p>\n\n---\n\n## What is PyMidscene?\n\nPyMidscene is a Python port of [Midscene.js](https://midscenejs.com) - an AI-powered UI automation framework. It allows you to control web browsers using natural language instead of CSS selectors or XPath.\n\n**No more fragile selectors!** Just describe what you want to click, type, or extract:\n\n```python\n# Instead of: page.click(\"#submit-btn-primary\")\nawait agent.ai_click(\"the blue Submit button\")\n\n# Instead of: page.fill(\"input[name='email']\", \"test@example.com\") \nawait agent.ai_input(\"email input field\", \"test@example.com\")\n\n# Extract structured data with natural language\nresult = await agent.ai_query({\n \"title\": \"the page title\",\n \"price\": \"the product price as a number\"\n})\n```\n\n## Features\n\n- **Natural Language Automation** - Describe elements in plain English/Chinese, no selectors needed\n- **Multi-Model Support** - Works with Doubao, Qwen, GPT-4V, Claude, and other vision LLMs\n- **Playwright Integration** - Seamless integration with Playwright for web automation\n- **Android Integration** - Control real devices over ADB with `pymidscene[android]` (see [pymidscene/android/README.md](pymidscene/android/README.md))\n- **iOS Integration** - Drive iPhones / simulators through WebDriverAgent (see [pymidscene/ios/README.md](pymidscene/ios/README.md))\n- **XPath Caching** - Smart caching system compatible with Midscene.js format\n- **Visual Reports** - Generate beautiful HTML reports for debugging and sharing\n- **Type-Safe** - Full type hints for excellent IDE support\n\n## Installation\n\n```bash\npip install pymidscene\n\n# Install Playwright browsers\nplaywright install chromium\n```\n\nOr with Poetry:\n\n```bash\npoetry add pymidscene\nplaywright install chromium\n```\n\n## Quick Start\n\n### 1. Set up your API key\n\n```bash\n# For Doubao (recommended for Chinese users)\nexport MIDSCENE_MODEL_NAME=\"doubao-seed-1-6-251015\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://ark.cn-beijing.volces.com/api/v3\"\nexport MIDSCENE_MODEL_FAMILY=\"doubao-vision\"\n\n# For Qwen\nexport MIDSCENE_MODEL_NAME=\"qwen-vl-max\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://dashscope.aliyuncs.com/compatible-mode/v1\"\nexport MIDSCENE_MODEL_FAMILY=\"qwen2.5-vl\"\n```\n\n### 2. Write your automation script\n\n```python\nimport asyncio\nimport os\nfrom playwright.async_api import async_playwright\nfrom pymidscene import PlaywrightAgent\n\nasync def main():\n # Configure model (or use environment variables)\n os.environ[\"MIDSCENE_MODEL_NAME\"] = \"doubao-seed-1-6-251015\"\n os.environ[\"MIDSCENE_MODEL_API_KEY\"] = \"your-api-key\"\n os.environ[\"MIDSCENE_MODEL_BASE_URL\"] = \"https://ark.cn-beijing.volces.com/api/v3\"\n os.environ[\"MIDSCENE_MODEL_FAMILY\"] = \"doubao-vision\"\n\n async with async_playwright() as p:\n browser = await p.chromium.launch(headless=False)\n page = await browser.new_page()\n\n # Create agent with optional caching\n agent = PlaywrightAgent(page, cache_id=\"my_task\")\n\n # Navigate to page\n await page.goto(\"https://www.example.com\")\n\n # Use natural language to interact\n await agent.ai_click(\"the search box\")\n await agent.ai_input(\"search input\", \"Python automation\")\n await agent.ai_click(\"search button\")\n\n # Extract data\n result = await agent.ai_query({\n \"results_count\": \"number of search results\",\n \"first_title\": \"title of the first result\"\n })\n print(f\"Found: {result}\")\n\n # Assert page state\n await agent.ai_assert(\"search results are displayed\")\n\n # Generate visual report\n report_path = agent.finish()\n print(f\"Report saved to: {report_path}\")\n\n await browser.close()\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n## CLI\n\nRun automation written as YAML scripts — no Python needed — with the `pymidscene`\ncommand (installed with the package):\n\n```bash\npymidscene ./script.yaml # run a single script\npymidscene ./scripts/ # run every *.yaml in a directory\npymidscene --files a.yaml b.yaml --concurrent 2\npymidscene --config ./suite.yaml --continue-on-error\n```\n\nA script declares one platform target (`web` / `android` / `ios`) plus `tasks`:\n\n```yaml\nweb:\n url: https://www.bing.com\ntasks:\n - name: search\n flow:\n - aiInput: the search box\n value: midscene\n - aiKeyboardPress: Enter\n - aiWaitFor: search results are visible\n timeout: 15000\n - aiQuery: \"{ titles: string[] }\"\n name: results\n - aiAssert: a list of results is shown\n```\n\nSee [docs/CLI.md](docs/CLI.md) for the full flag and flow-item reference, and\n[examples/cli/](examples/cli/) for ready-to-run scripts.\n\n## Documentation\n\n### Core API\n\n| Method | Description |\n|--------|-------------|\n| `ai_click(description)` | Click an element described in natural language |\n| `ai_input(description, text)` | Type text into an input field |\n| `ai_locate(description)` | Locate an element and return its coordinates |\n| `ai_query(schema)` | Extract structured data from the page |\n| `ai_assert(assertion)` | Assert that a condition is true |\n| `ai_action(task)` | Execute a complex task with AI planning loop (plan-execute-replan) |\n| `ai_wait_for(assertion, timeout)` | Wait until a page condition is met (polling) |\n| `ai_scroll(direction, distance)` | Scroll the page with AI assistance |\n| `finish()` | Generate HTML report and return the path |\n\n### Supported Models\n\n| Model | Family | Provider |\n|-------|--------|----------|\n| doubao-seed-1-6-251015 | doubao-vision | Bytedance/Volcano |\n| qwen-vl-max | qwen2.5-vl | Alibaba |\n| gpt-4-vision-preview | openai | OpenAI |\n| claude-3-opus | claude | Anthropic |\n\n### Cache System\n\nPyMidscene uses XPath-based caching compatible with Midscene.js:\n\n```yaml\n# midscene_run/cache/my_task.cache.yaml\nmidsceneVersion: 1.0.0\ncacheId: my_task\ncaches:\n - type: locate\n prompt: the login button\n cache:\n xpaths:\n - /html/body/div[1]/button[1]\n```\n\nThis means:\n- Cache files are interchangeable between JS and Python versions\n- XPath-based caching works across different window sizes\n- Cache invalidation happens automatically when elements move\n\n## Examples\n\nCheck out the [examples/](examples/) directory:\n\n- `basic_usage.py` - Getting started\n- `login_demo.py` / `login_demo.html` - Login automation with a visual report\n- `android_basic.py` / `ios_basic.py` - Mobile automation\n- `cli/` - YAML scripts for the `pymidscene` CLI (web / android / suite)\n\n## Project Structure\n\n```\npymidscene/\n├── pymidscene/ # Main package\n│ ├── core/ # Core automation logic\n│ │ ├── agent/ # Agent implementation\n│ │ ├── ai_model/ # AI model integration\n│ │ └── dump.py # Report generation\n│ ├── web_integration/ # Browser integrations\n│ │ └── playwright/ # Playwright adapter\n│ └── shared/ # Shared utilities\n├── examples/ # Usage examples\n├── tests/ # Test suite\n└── docs/ # Documentation\n```\n\n## Related Projects\n\nThis is the Python implementation of [Midscene.js](https://midscenejs.com).\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - Original JavaScript version\n- [Official Documentation](https://midscenejs.com)\n- [Awesome Midscene](https://midscenejs.com/awesome-midscene) - Community projects\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n```bash\n# Development setup\ngit clone https://github.com/AIPythoner/pymidscene.git\ncd pymidscene\npip install -e \".[dev]\"\n\n# Run tests\npytest\n\n# Format code\nblack pymidscene tests\n```\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - The original JavaScript framework by Bytedance\n- [Playwright](https://playwright.dev/) - Browser automation library\n\n---\n\n<p align=\"center\">\n Made with love by the PyMidscene community\n</p>\n"}}, "pymidscene/core/ai_model/models/doubao.py": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674281.930608, "n": 7558, "h": "ff9e9aaa80656366", "c": "\"\"\"\n豆包（Doubao）视觉模型适配器\n\n支持豆包 Seed 1.6 和 UI-TARS 模型家族。\n火山引擎文档: https://www.volcengine.com/docs/82379/1298454\n\"\"\"\n\nimport os\nimport re\nfrom typing import List, Dict, Any, Optional, Tuple\nfrom openai import OpenAI\n\nfrom .base import BaseAIModel\nfrom ....shared.logger import logger\n\n\nclass DoubaoVisionModel(BaseAIModel):\n \"\"\"\n 豆包视觉模型适配器\n\n 支持的模型：\n - doubao-vision (Seed 1.6) - 推荐\n - vlm-ui-tars-doubao-1.5 (UI-TARS 1.5)\n - vlm-ui-tars-doubao (UI-TARS 默认版本)\n\n 环境变量配置：\n - MIDSCENE_DOUBAO_API_KEY: API 密钥\n - MIDSCENE_DOUBAO_BASE_URL: API 基础 URL（默认：https://ark.cn-beijing.volces.com/api/v3）\n - MIDSCENE_DOUBAO_MODEL_NAME: 推理接入点 ID（如 ep-20250122xxx）\n \"\"\"\n\n DEFAULT_BASE_URL = \"https://ark.cn-beijing.volces.com/api/v3\"\n\n SUPPORTED_MODELS = [\n \"doubao-vision\",\n \"vlm-ui-tars-doubao\",\n \"vlm-ui-tars-doubao-1.5\",\n ]\n\n # 豆包使用 0-1000 归一化坐标系统\n COORDINATE_SCALE = 1000\n\n def __init__(\n self,\n model_name: str = \"doubao-vision\",\n api_key: Optional[str] = None,\n base_url: Optional[str] = None,\n endpoint_id: Optional[str] = None,\n **kwargs\n ):\n \"\"\"\n 初始化豆包模型\n\n Args:\n model_name: 模型名称（doubao-vision 或 vlm-ui-tars-doubao-1.5）\n api_key: API 密钥\n base_url: API 基础 URL\n endpoint_id: 推理接入点 ID（ep-xxx）\n **kwargs: 其他参数（temperature, max_tokens 等）\n \"\"\"\n if model_name not in self.SUPPORTED_MODELS:\n logger.warning(\n f\"Model {model_name} not in supported list: {self.SUPPORTED_MODELS}. \"\n f\"Will try to use it anyway.\"\n )\n\n self.model_name = model_name\n self.endpoint_id = endpoint_id or os.getenv(\"MIDSCENE_DOUBAO_MODEL_NAME\", \"\")\n self.api_key = api_key or os.getenv(\"MIDSCENE_DOUBAO_API_KEY\", \"\")\n self.base_url = base_url or os.getenv(\n \"MIDSCENE_DOUBAO_BASE_URL\",\n self.DEFAULT_BASE_URL\n )\n self.model_kwargs = kwargs\n\n if not self.api_key:\n raise ValueError(\n \"Doubao API key is required. Set MIDSCENE_DOUBAO_API_KEY or pass api_key.\"\n )\n\n if not self.endpoint_id:\n raise ValueError(\n \"Doubao endpoint ID is required. Set MIDSCENE_DOUBAO_MODEL_NAME or pass endpoint_id.\"\n )\n\n # 初始化 OpenAI 客户端（豆包兼容 OpenAI API）\n self.client = OpenAI(\n api_key=self.api_key,\n base_url=self.base_url\n )\n\n logger.info(\n f\"Doubao model initialized: {model_name}, \"\n f\"endpoint={self.endpoint_id}\"\n )\n\n @classmethod\n def from_env(\n cls,\n model_name: str = \"doubao-vision\"\n ) -> 'DoubaoVisionModel':\n \"\"\"\n 从环境变量创建模型实例\n\n 环境变量：\n - MIDSCENE_DOUBAO_API_KEY\n - MIDSCENE_DOUBAO_BASE_URL (可选)\n - MIDSCENE_DOUBAO_MODEL_NAME (推理接入点 ID)\n\n Args:\n model_name: 模型名称\n\n Returns:\n DoubaoVisionModel 实例\n \"\"\"\n return cls(model_name=model_name)\n\n def call(\n self,\n messages: List[Dict[str, Any]],\n stream: bool = False,\n **kwargs\n ) -> Dict[str, Any]:\n \"\"\"\n 调用豆包模型\n\n Args:\n messages: 消息列表（OpenAI 格式）\n stream: 是否流式输出\n **kwargs: 其他参数\n\n Returns:\n 包含 content 和 usage 的字典\n \"\"\"\n # 合并参数\n request_params = {\n \"model\": self.endpoint_id, # 使用推理接入点 ID\n \"messages\": messages,\n \"stream\": stream,\n **self.model_kwargs,\n **kwargs\n }\n\n # 处理深度思考参数（豆包使用 thinking.type）\n deep_think_value = None\n if \"deep_think\" in request_params:\n deep_think_value = request_params.pop(\"deep_think\")\n elif \"deepThink\" in request_params:\n deep_think_value = request_params.pop(\"deepThink\")\n\n if deep_think_value is not None:\n request_params.setdefault(\"extra_body\", {})\n request_params[\"extra_body\"][\"config\"] = {\n \"thinking\": {\n \"type\": \"enabled\" if deep_think_value else \"disabled\"\n }\n }\n logger.debug(f\"Doubao deepThink mapped to thinking.type={request_params['extra_body']['config']['thinking']['type']}\")\n\n logger.debug(f\"Calling Doubao model: {self.model_name} (endpoint: {self.endpoint_id})\")\n\n try:\n response = self.client.chat.completions.create(**request_params)\n\n if stream:\n # 流式响应处理\n return {\"stream\": response}\n\n # 提取响应内容\n content = response.choices[0].message.content or \"\"\n\n # 预处理豆包 bbox 格式（空格分隔 -> 逗号分隔）\n content = self.preprocess_doubao_bbox_json(content)\n\n # 提取 usage 信息\n usage = None\n if hasattr(response, 'usage') and response.usage:\n usage = {\n \"prompt_tokens\": response.usage.prompt_tokens,\n \"completion_tokens\": response.usage.completion_tokens,\n \"total_tokens\": response.usage.total_tokens,\n }\n\n logger.debug(f\"Doubao response received: {len(content)} chars\")\n\n return {\n \"content\": content,\n \"usage\": usage,\n \"raw_response\": response\n }\n\n except Exception as e:\n logger.error(f\"Doubao API call failed: {e}\")\n raise\n\n @staticmethod\n def preprocess_doubao_bbox_json(input_str: str) -> str:\n \"\"\"\n 预处理豆包 bbox JSON 格式\n\n 豆包可能返回空格分隔的 bbox 值：\"940 445 969 490\"\n 需要转换为逗号分隔：\"940,445,969,490\"\n\n Args:\n input_str: 原始 JSON 字符串\n\n Returns:\n 处理后的 JSON 字符串\n \"\"\"\n if 'bbox' not in input_str:\n return input_str\n\n # 将 bbox 值中的空格替换为逗号\n # 匹配模式：数字 + 空格 + 数字\n while re.search(r'\\d+\\s+\\d+', input_str):\n input_str = re.sub(r'(\\d+)\\s+(\\d+)', r'\\1,\\2', input_str)\n\n return input_str\n\n @staticmethod\n def adapt_doubao_bbox(\n bbox: Any,\n width: int,\n height: int\n ) -> Tuple[int, int, int, int]:\n \"\"\"\n 适配豆包 bbox 格式到标准格式\n \n 对应 JS 版本: adaptDoubaoBbox (common.ts:108-192)\n\n 豆包支持多种 bbox 格式：\n 1. 字符串：\"x1 y1 x2 y2\" (空格分隔，0-1000 归一化坐标)\n 2. 字符串数组：[\"123 222\", \"789 100\"] 或 [\"123,222\", \"789,100\"]\n 3. 数组：[x1, y1, x2, y2] (0-1000 归一化坐标)\n 4. 数组：[x, y] (中心点坐标，自动创建小矩形)\n\n Args:\n bbox: bbox 值（字符串或数组）\n width: 图像宽度（像素）\n height: 图像高度（像素）\n\n Returns:\n 标准 bbox: (left, top, right, bottom) 像素坐标\n \"\"\"\n # 使用统一的工具函数\n from ....shared.utils import adapt_doubao_bbox as _adapt_doubao_bbox\n return _adapt_doubao_bbox(bbox, width, height)\n\n def get_model_name(self) -> str:\n \"\"\"获取模型名称\"\"\"\n return self.model_name\n\n def get_model_family(self) -> str:\n \"\"\"获取模型家族\"\"\"\n if \"ui-tars\" in self.model_name:\n return \"vlm-ui-tars-doubao\"\n return \"doubao-vision\"\n\n def validate_config(self) -> bool:\n \"\"\"\n 验证配置是否有效\n\n Returns:\n 配置是否有效\n \"\"\"\n if not self.api_key:\n logger.error(\"Doubao API key is missing\")\n return False\n if not self.endpoint_id:\n logger.error(\"Doubao endpoint ID is missing\")\n return False\n return True\n\n\n__all__ = [\"DoubaoVisionModel\"]\n"}}}} |