{"repo": "aipythoner/pymidscene", "files": {"docs/MODELS.md": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674250.7292187, "n": 2437, "h": "3c738f8b0f15abe1", "c": "# 模型配置指南 - Model Configuration\n\nPyMidscene 支持多种视觉大模型，本文档介绍各模型的配置方法。\n\n## 支持的模型\n\n| 模型 | 提供商 | 模型家族 | 推荐场景 |\n|------|--------|----------|----------|\n| doubao-seed-1.6 | 字节跳动/火山引擎 | doubao-vision | 国内用户首选 |\n| qwen-vl-max | 阿里云 | qwen2.5-vl | 国内备选 |\n| gpt-4-vision | OpenAI | openai | 海外用户 |\n| claude-3 | Anthropic | claude | 海外用户 |\n\n## 配置方式\n\n### 方式一：环境变量（推荐）\n\n创建 `.env` 文件：\n\n```bash\n# 豆包模型\nMIDSCENE_MODEL_NAME=your-endpoint-id\nMIDSCENE_MODEL_API_KEY=your-api-key\nMIDSCENE_MODEL_BASE_URL=https://ark.cn-beijing.volces.com/api/v3\nMIDSCENE_MODEL_FAMILY=doubao-vision\n```\n\n代码中加载：\n\n```python\nfrom dotenv import load_dotenv\nload_dotenv()\n\nagent = PlaywrightAgent(page)\n```\n\n### 方式二：代码配置\n\n```python\nimport os\n\nos.environ[\"MIDSCENE_MODEL_NAME\"] = \"your-endpoint-id\"\nos.environ[\"MIDSCENE_MODEL_API_KEY\"] = \"your-api-key\"\nos.environ[\"MIDSCENE_MODEL_BASE_URL\"] = \"https://...\"\nos.environ[\"MIDSCENE_MODEL_FAMILY\"] = \"doubao-vision\"\n\nagent = PlaywrightAgent(page)\n```\n\n### 方式三：model_config 参数\n\n```python\nconfig = {\n    \"MIDSCENE_MODEL_NAME\": \"your-endpoint-id\",\n    \"MIDSCENE_MODEL_API_KEY\": \"your-api-key\",\n    \"MIDSCENE_MODEL_BASE_URL\": \"https://...\",\n    \"MIDSCENE_MODEL_FAMILY\": \"doubao-vision\",\n}\n\nagent = PlaywrightAgent(page, model_config=config)\n```\n\n## 各模型详细配置\n\n### 豆包 Doubao（火山引擎）\n\n1. 访问 [火山引擎控制台](https://console.volcengine.com/ark)\n2. 创建推理接入点，获取 Endpoint ID\n3. 获取 API Key\n\n```bash\nMIDSCENE_MODEL_NAME=ep-20250122xxxxxx  # 你的 Endpoint ID\nMIDSCENE_MODEL_API_KEY=your-api-key\nMIDSCENE_MODEL_BASE_URL=https://ark.cn-beijing.volces.com/api/v3\nMIDSCENE_MODEL_FAMILY=doubao-vision\n```\n\n### 千问 Qwen（阿里云）\n\n1. 访问 [阿里云 DashScope](https://dashscope.console.aliyun.com/)\n2. 开通模型服务，获取 API Key\n\n```bash\nMIDSCENE_MODEL_NAME=qwen-vl-max\nMIDSCENE_MODEL_API_KEY=sk-xxxxxx\nMIDSCENE_MODEL_BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1\nMIDSCENE_MODEL_FAMILY=qwen2.5-vl\n```\n\n### OpenAI GPT-4V\n\n```bash\nMIDSCENE_MODEL_NAME=gpt-4-vision-preview\nMIDSCENE_MODEL_API_KEY=sk-xxxxxx\nMIDSCENE_MODEL_BASE_URL=https://api.openai.com/v1\nMIDSCENE_MODEL_FAMILY=openai\n```\n\n## 坐标系统差异\n\n不同模型返回的坐标格式不同，PyMidscene 会自动处理：\n\n| 模型家族 | 坐标格式 | 说明 |\n|----------|----------|------|\n| doubao-vision | 归一化 0-1000 | 需要 `x * width / 1000` |\n| qwen2.5-vl | 像素坐标 | 直接使用 |\n| openai | 像素坐标 | 直接使用 |\n| gemini | [y1, x1, y2, x2] | 需要交换顺序 |\n\n## 常见问题\n\n### Q: 为什么点击位置不准确？\n\nA: 检查 `MIDSCENE_MODEL_FAMILY` 是否正确设置。不同模型返回的坐标格式不同，需要正确的模型家族才能正确转换坐标。\n\n### Q: 如何切换模型？\n\nA: 修改环境变量或 `.env` 文件即可，无需修改代码。\n"}}, "README_CN.md": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674253.6049097, "n": 5390, "h": "eb1472c5827567ea", "c": "# PyMidscene\n\n<p align=\"center\">\n  <img src=\"https://midscenejs.com/midscene_with_text_light.png\" alt=\"Midscene Logo\" width=\"300\">\n</p>\n\n<p align=\"center\">\n  <strong>Midscene.js 的 Python SDK - 使用自然语言进行 AI 驱动的 UI 自动化</strong>\n</p>\n\n<p align=\"center\">\n  <a href=\"https://github.com/AIPythoner/pymidscene/blob/main/LICENSE\"><img src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"License\"></a>\n  <a href=\"https://www.python.org/downloads/\"><img src=\"https://img.shields.io/badge/python-3.10+-blue.svg\" alt=\"Python\"></a>\n  <a href=\"https://midscenejs.com\"><img src=\"https://img.shields.io/badge/docs-midscenejs.com-green.svg\" alt=\"Docs\"></a>\n</p>\n\n<p align=\"center\">\n  <a href=\"#特性\">特性</a> |\n  <a href=\"#安装\">安装</a> |\n  <a href=\"#快速开始\">快速开始</a> |\n  <a href=\"#文档\">文档</a> |\n  <a href=\"./README.md\">English</a>\n</p>\n\n---\n\n## 什么是 PyMidscene？\n\nPyMidscene 是 [Midscene.js](https://midscenejs.com) 的 Python 版本 - 一个 AI 驱动的 UI 自动化框架。它允许你使用自然语言而不是 CSS 选择器或 XPath 来控制浏览器。\n\n**告别脆弱的选择器！** 只需描述你想要点击、输入或提取的内容：\n\n```python\n# 不再需要: page.click(\"#submit-btn-primary\")\nawait agent.ai_click(\"蓝色的提交按钮\")\n\n# 不再需要: page.fill(\"input[name='email']\", \"test@example.com\")  \nawait agent.ai_input(\"邮箱输入框\", \"test@example.com\")\n\n# 使用自然语言提取结构化数据\nresult = await agent.ai_query({\n    \"title\": \"页面标题\",\n    \"price\": \"商品价格，数字类型\"\n})\n```\n\n## 特性\n\n- **自然语言自动化** - 用中文/英文描述元素，无需选择器\n- **多模型支持** - 支持豆包、千问、GPT-4V、Claude 等视觉大模型\n- **Playwright 集成** - 与 Playwright 无缝集成进行网页自动化\n- **XPath 缓存** - 与 Midscene.js 格式兼容的智能缓存系统\n- **可视化报告** - 生成精美的 HTML 报告用于调试和分享\n- **类型安全** - 完整的类型提示，提供优秀的 IDE 支持\n\n## 安装\n\n```bash\npip install pymidscene\n\n# 安装 Playwright 浏览器\nplaywright install chromium\n```\n\n或使用 Poetry：\n\n```bash\npoetry add pymidscene\nplaywright install chromium\n```\n\n## 快速开始\n\n### 1. 配置 API 密钥\n\n```bash\n# 豆包模型（推荐国内用户使用）\nexport MIDSCENE_MODEL_NAME=\"doubao-seed-1-6-251015\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://ark.cn-beijing.volces.com/api/v3\"\nexport MIDSCENE_MODEL_FAMILY=\"doubao-vision\"\n\n# 千问模型\nexport MIDSCENE_MODEL_NAME=\"qwen-vl-max\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://dashscope.aliyuncs.com/compatible-mode/v1\"\nexport MIDSCENE_MODEL_FAMILY=\"qwen2.5-vl\"\n```\n\n### 2. 编写自动化脚本\n\n```python\nimport asyncio\nimport os\nfrom playwright.async_api import async_playwright\nfrom pymidscene import PlaywrightAgent\n\nasync def main():\n    # 配置模型（或使用环境变量）\n    os.environ[\"MIDSCENE_MODEL_NAME\"] = \"doubao-seed-1-6-251015\"\n    os.environ[\"MIDSCENE_MODEL_API_KEY\"] = \"your-api-key\"\n    os.environ[\"MIDSCENE_MODEL_BASE_URL\"] = \"https://ark.cn-beijing.volces.com/api/v3\"\n    os.environ[\"MIDSCENE_MODEL_FAMILY\"] = \"doubao-vision\"\n\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(headless=False)\n        page = await browser.new_page()\n\n        # 创建 agent，可选启用缓存\n        agent = PlaywrightAgent(page, cache_id=\"my_task\")\n\n        # 导航到页面\n        await page.goto(\"https://www.baidu.com\")\n\n        # 使用自然语言进行交互\n        await agent.ai_click(\"搜索输入框\")\n        await agent.ai_input(\"搜索框\", \"Python 自动化\")\n        await agent.ai_click(\"百度一下按钮\")\n\n        # 提取数据\n        result = await agent.ai_query({\n            \"results_count\": \"搜索结果数量\",\n            \"first_title\": \"第一条结果的标题\"\n        })\n        print(f\"结果: {result}\")\n\n        # 断言页面状态\n        await agent.ai_assert(\"页面显示了搜索结果\")\n\n        # 生成可视化报告\n        report_path = agent.finish()\n        print(f\"报告已保存到: {report_path}\")\n\n        await browser.close()\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## 文档\n\n### 核心 API\n\n| 方法 | 描述 |\n|------|------|\n| `ai_click(description)` | 使用自然语言描述点击元素 |\n| `ai_input(description, text)` | 在输入框中输入文本 |\n| `ai_locate(description)` | 定位元素并返回坐标 |\n| `ai_query(schema)` | 从页面提取结构化数据 |\n| `ai_assert(assertion)` | 断言某个条件为真 |\n| `ai_action(task)` | 执行复杂任务，AI 自动规划-执行-重规划循环 |\n| `ai_wait_for(assertion, timeout)` | 等待页面满足某个条件（轮询） |\n| `ai_scroll(direction, distance)` | AI 辅助滚动页面 |\n| `finish()` | 生成 HTML 报告并返回路径 |\n\n### 支持的模型\n\n| 模型 | 模型家族 | 提供商 |\n|------|----------|--------|\n| doubao-seed-1-6-251015 | doubao-vision | 字节跳动/火山引擎 |\n| qwen-vl-max | qwen2.5-vl | 阿里巴巴 |\n| gpt-4-vision-preview | openai | OpenAI |\n| claude-3-opus | claude | Anthropic |\n\n### 缓存系统\n\nPyMidscene 使用与 Midscene.js 兼容的基于 XPath 的缓存：\n\n```yaml\n# midscene_run/cache/my_task.cache.yaml\nmidsceneVersion: 1.0.0\ncacheId: my_task\ncaches:\n  - type: locate\n    prompt: 登录按钮\n    cache:\n      xpaths:\n        - /html/body/div[1]/button[1]\n```\n\n这意味着：\n- 缓存文件可在 JS 和 Python 版本之间互换使用\n- 基于 XPath 的缓存在不同窗口大小下都能正常工作\n- 当元素移动时会自动使缓存失效\n\n## 示例\n\n查看 [examples/](examples/) 目录：\n\n- `basic_usage.py` - 入门示例\n- `login_demo.py` - 带可视化报告的登录自动化\n- `login_demo.html` - 登录演示的测试页面\n\n## 相关项目\n\n本项目是 [Midscene.js](https://midscenejs.com) 的 Python 实现。\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - 原版 JavaScript 版本\n- [官方文档](https://midscenejs.com)\n- [Awesome Midscene](https://midscenejs.com/awesome-midscene) - 社区项目\n\n## 贡献\n\n欢迎贡献！请查看 [CONTRIBUTING.md](CONTRIBUTING.md) 了解指南。\n\n```bash\n# 开发环境设置\ngit clone https://github.com/AIPythoner/pymidscene.git\ncd pymidscene\npip install -e \".[dev]\"\n\n# 运行测试\npytest\n\n# 格式化代码\nblack pymidscene tests\n```\n\n## 许可证\n\nMIT License - 详见 [LICENSE](LICENSE) 文件。\n\n## 致谢\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - 字节跳动开发的原版 JavaScript 框架\n- [Playwright](https://playwright.dev/) - 浏览器自动化库\n\n## 交流群\n\nQQ 群：**2156022195**\n\n---\n\n<p align=\"center\">\n  由 PyMidscene 社区用 ❤️ 打造\n</p>\n"}}, "README.md": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674254.2864528, "n": 8957, "h": "2aec168769f74c16", "c": "# PyMidscene\n\n<p align=\"center\">\n  <img src=\"https://midscenejs.com/midscene_with_text_light.png\" alt=\"Midscene Logo\" width=\"300\">\n</p>\n\n<p align=\"center\">\n  <strong>Python SDK for Midscene.js - AI-powered UI automation using natural language</strong>\n</p>\n\n<p align=\"center\">\n  <a href=\"https://github.com/AIPythoner/pymidscene/blob/main/LICENSE\"><img src=\"https://img.shields.io/badge/license-MIT-blue.svg\" alt=\"License\"></a>\n  <a href=\"https://www.python.org/downloads/\"><img src=\"https://img.shields.io/badge/python-3.10+-blue.svg\" alt=\"Python\"></a>\n  <a href=\"https://midscenejs.com\"><img src=\"https://img.shields.io/badge/docs-midscenejs.com-green.svg\" alt=\"Docs\"></a>\n</p>\n\n<p align=\"center\">\n  <a href=\"#features\">Features</a> |\n  <a href=\"#installation\">Installation</a> |\n  <a href=\"#quick-start\">Quick Start</a> |\n  <a href=\"#documentation\">Documentation</a> |\n  <a href=\"./README_CN.md\">中文文档</a>\n</p>\n\n---\n\n## What is PyMidscene?\n\nPyMidscene is a Python port of [Midscene.js](https://midscenejs.com) - an AI-powered UI automation framework. It allows you to control web browsers using natural language instead of CSS selectors or XPath.\n\n**No more fragile selectors!** Just describe what you want to click, type, or extract:\n\n```python\n# Instead of: page.click(\"#submit-btn-primary\")\nawait agent.ai_click(\"the blue Submit button\")\n\n# Instead of: page.fill(\"input[name='email']\", \"test@example.com\")  \nawait agent.ai_input(\"email input field\", \"test@example.com\")\n\n# Extract structured data with natural language\nresult = await agent.ai_query({\n    \"title\": \"the page title\",\n    \"price\": \"the product price as a number\"\n})\n```\n\n## Features\n\n- **Natural Language Automation** - Describe elements in plain English/Chinese, no selectors needed\n- **Multi-Model Support** - Works with Doubao, Qwen, GPT-4V, Claude, and other vision LLMs\n- **Playwright Integration** - Seamless integration with Playwright for web automation\n- **Android Integration** - Control real devices over ADB with `pymidscene[android]` (see [pymidscene/android/README.md](pymidscene/android/README.md))\n- **iOS Integration** - Drive iPhones / simulators through WebDriverAgent (see [pymidscene/ios/README.md](pymidscene/ios/README.md))\n- **XPath Caching** - Smart caching system compatible with Midscene.js format\n- **Visual Reports** - Generate beautiful HTML reports for debugging and sharing\n- **Type-Safe** - Full type hints for excellent IDE support\n\n## Installation\n\n```bash\npip install pymidscene\n\n# Install Playwright browsers\nplaywright install chromium\n```\n\nOr with Poetry:\n\n```bash\npoetry add pymidscene\nplaywright install chromium\n```\n\n## Quick Start\n\n### 1. Set up your API key\n\n```bash\n# For Doubao (recommended for Chinese users)\nexport MIDSCENE_MODEL_NAME=\"doubao-seed-1-6-251015\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://ark.cn-beijing.volces.com/api/v3\"\nexport MIDSCENE_MODEL_FAMILY=\"doubao-vision\"\n\n# For Qwen\nexport MIDSCENE_MODEL_NAME=\"qwen-vl-max\"\nexport MIDSCENE_MODEL_API_KEY=\"your-api-key\"\nexport MIDSCENE_MODEL_BASE_URL=\"https://dashscope.aliyuncs.com/compatible-mode/v1\"\nexport MIDSCENE_MODEL_FAMILY=\"qwen2.5-vl\"\n```\n\n### 2. Write your automation script\n\n```python\nimport asyncio\nimport os\nfrom playwright.async_api import async_playwright\nfrom pymidscene import PlaywrightAgent\n\nasync def main():\n    # Configure model (or use environment variables)\n    os.environ[\"MIDSCENE_MODEL_NAME\"] = \"doubao-seed-1-6-251015\"\n    os.environ[\"MIDSCENE_MODEL_API_KEY\"] = \"your-api-key\"\n    os.environ[\"MIDSCENE_MODEL_BASE_URL\"] = \"https://ark.cn-beijing.volces.com/api/v3\"\n    os.environ[\"MIDSCENE_MODEL_FAMILY\"] = \"doubao-vision\"\n\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(headless=False)\n        page = await browser.new_page()\n\n        # Create agent with optional caching\n        agent = PlaywrightAgent(page, cache_id=\"my_task\")\n\n        # Navigate to page\n        await page.goto(\"https://www.example.com\")\n\n        # Use natural language to interact\n        await agent.ai_click(\"the search box\")\n        await agent.ai_input(\"search input\", \"Python automation\")\n        await agent.ai_click(\"search button\")\n\n        # Extract data\n        result = await agent.ai_query({\n            \"results_count\": \"number of search results\",\n            \"first_title\": \"title of the first result\"\n        })\n        print(f\"Found: {result}\")\n\n        # Assert page state\n        await agent.ai_assert(\"search results are displayed\")\n\n        # Generate visual report\n        report_path = agent.finish()\n        print(f\"Report saved to: {report_path}\")\n\n        await browser.close()\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## CLI\n\nRun automation written as YAML scripts — no Python needed — with the `pymidscene`\ncommand (installed with the package):\n\n```bash\npymidscene ./script.yaml                  # run a single script\npymidscene ./scripts/                      # run every *.yaml in a directory\npymidscene --files a.yaml b.yaml --concurrent 2\npymidscene --config ./suite.yaml --continue-on-error\n```\n\nA script declares one platform target (`web` / `android` / `ios`) plus `tasks`:\n\n```yaml\nweb:\n  url: https://www.bing.com\ntasks:\n  - name: search\n    flow:\n      - aiInput: the search box\n        value: midscene\n      - aiKeyboardPress: Enter\n      - aiWaitFor: search results are visible\n        timeout: 15000\n      - aiQuery: \"{ titles: string[] }\"\n        name: results\n      - aiAssert: a list of results is shown\n```\n\nSee [docs/CLI.md](docs/CLI.md) for the full flag and flow-item reference, and\n[examples/cli/](examples/cli/) for ready-to-run scripts.\n\n## Documentation\n\n### Core API\n\n| Method | Description |\n|--------|-------------|\n| `ai_click(description)` | Click an element described in natural language |\n| `ai_input(description, text)` | Type text into an input field |\n| `ai_locate(description)` | Locate an element and return its coordinates |\n| `ai_query(schema)` | Extract structured data from the page |\n| `ai_assert(assertion)` | Assert that a condition is true |\n| `ai_action(task)` | Execute a complex task with AI planning loop (plan-execute-replan) |\n| `ai_wait_for(assertion, timeout)` | Wait until a page condition is met (polling) |\n| `ai_scroll(direction, distance)` | Scroll the page with AI assistance |\n| `finish()` | Generate HTML report and return the path |\n\n### Supported Models\n\n| Model | Family | Provider |\n|-------|--------|----------|\n| doubao-seed-1-6-251015 | doubao-vision | Bytedance/Volcano |\n| qwen-vl-max | qwen2.5-vl | Alibaba |\n| gpt-4-vision-preview | openai | OpenAI |\n| claude-3-opus | claude | Anthropic |\n\n### Cache System\n\nPyMidscene uses XPath-based caching compatible with Midscene.js:\n\n```yaml\n# midscene_run/cache/my_task.cache.yaml\nmidsceneVersion: 1.0.0\ncacheId: my_task\ncaches:\n  - type: locate\n    prompt: the login button\n    cache:\n      xpaths:\n        - /html/body/div[1]/button[1]\n```\n\nThis means:\n- Cache files are interchangeable between JS and Python versions\n- XPath-based caching works across different window sizes\n- Cache invalidation happens automatically when elements move\n\n## Examples\n\nCheck out the [examples/](examples/) directory:\n\n- `basic_usage.py` - Getting started\n- `login_demo.py` / `login_demo.html` - Login automation with a visual report\n- `android_basic.py` / `ios_basic.py` - Mobile automation\n- `cli/` - YAML scripts for the `pymidscene` CLI (web / android / suite)\n\n## Project Structure\n\n```\npymidscene/\n├── pymidscene/           # Main package\n│   ├── core/             # Core automation logic\n│   │   ├── agent/        # Agent implementation\n│   │   ├── ai_model/     # AI model integration\n│   │   └── dump.py       # Report generation\n│   ├── web_integration/  # Browser integrations\n│   │   └── playwright/   # Playwright adapter\n│   └── shared/           # Shared utilities\n├── examples/             # Usage examples\n├── tests/                # Test suite\n└── docs/                 # Documentation\n```\n\n## Related Projects\n\nThis is the Python implementation of [Midscene.js](https://midscenejs.com).\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - Original JavaScript version\n- [Official Documentation](https://midscenejs.com)\n- [Awesome Midscene](https://midscenejs.com/awesome-midscene) - Community projects\n\n## Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.\n\n```bash\n# Development setup\ngit clone https://github.com/AIPythoner/pymidscene.git\ncd pymidscene\npip install -e \".[dev]\"\n\n# Run tests\npytest\n\n# Format code\nblack pymidscene tests\n```\n\n## License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n## Acknowledgments\n\n- [Midscene.js](https://github.com/web-infra-dev/midscene) - The original JavaScript framework by Bytedance\n- [Playwright](https://playwright.dev/) - Browser automation library\n\n---\n\n<p align=\"center\">\n  Made with love by the PyMidscene community\n</p>\n"}}, "pymidscene/core/ai_model/models/doubao.py": {"332e0fc76b93c135654d74f5998f5d4b4e41ff3b": {"t": 1785674281.930608, "n": 7558, "h": "ff9e9aaa80656366", "c": "\"\"\"\n豆包（Doubao）视觉模型适配器\n\n支持豆包 Seed 1.6 和 UI-TARS 模型家族。\n火山引擎文档: https://www.volcengine.com/docs/82379/1298454\n\"\"\"\n\nimport os\nimport re\nfrom typing import List, Dict, Any, Optional, Tuple\nfrom openai import OpenAI\n\nfrom .base import BaseAIModel\nfrom ....shared.logger import logger\n\n\nclass DoubaoVisionModel(BaseAIModel):\n    \"\"\"\n    豆包视觉模型适配器\n\n    支持的模型：\n    - doubao-vision (Seed 1.6) - 推荐\n    - vlm-ui-tars-doubao-1.5 (UI-TARS 1.5)\n    - vlm-ui-tars-doubao (UI-TARS 默认版本)\n\n    环境变量配置：\n    - MIDSCENE_DOUBAO_API_KEY: API 密钥\n    - MIDSCENE_DOUBAO_BASE_URL: API 基础 URL（默认：https://ark.cn-beijing.volces.com/api/v3）\n    - MIDSCENE_DOUBAO_MODEL_NAME: 推理接入点 ID（如 ep-20250122xxx）\n    \"\"\"\n\n    DEFAULT_BASE_URL = \"https://ark.cn-beijing.volces.com/api/v3\"\n\n    SUPPORTED_MODELS = [\n        \"doubao-vision\",\n        \"vlm-ui-tars-doubao\",\n        \"vlm-ui-tars-doubao-1.5\",\n    ]\n\n    # 豆包使用 0-1000 归一化坐标系统\n    COORDINATE_SCALE = 1000\n\n    def __init__(\n        self,\n        model_name: str = \"doubao-vision\",\n        api_key: Optional[str] = None,\n        base_url: Optional[str] = None,\n        endpoint_id: Optional[str] = None,\n        **kwargs\n    ):\n        \"\"\"\n        初始化豆包模型\n\n        Args:\n            model_name: 模型名称（doubao-vision 或 vlm-ui-tars-doubao-1.5）\n            api_key: API 密钥\n            base_url: API 基础 URL\n            endpoint_id: 推理接入点 ID（ep-xxx）\n            **kwargs: 其他参数（temperature, max_tokens 等）\n        \"\"\"\n        if model_name not in self.SUPPORTED_MODELS:\n            logger.warning(\n                f\"Model {model_name} not in supported list: {self.SUPPORTED_MODELS}. \"\n                f\"Will try to use it anyway.\"\n            )\n\n        self.model_name = model_name\n        self.endpoint_id = endpoint_id or os.getenv(\"MIDSCENE_DOUBAO_MODEL_NAME\", \"\")\n        self.api_key = api_key or os.getenv(\"MIDSCENE_DOUBAO_API_KEY\", \"\")\n        self.base_url = base_url or os.getenv(\n            \"MIDSCENE_DOUBAO_BASE_URL\",\n            self.DEFAULT_BASE_URL\n        )\n        self.model_kwargs = kwargs\n\n        if not self.api_key:\n            raise ValueError(\n                \"Doubao API key is required. Set MIDSCENE_DOUBAO_API_KEY or pass api_key.\"\n            )\n\n        if not self.endpoint_id:\n            raise ValueError(\n                \"Doubao endpoint ID is required. Set MIDSCENE_DOUBAO_MODEL_NAME or pass endpoint_id.\"\n            )\n\n        # 初始化 OpenAI 客户端（豆包兼容 OpenAI API）\n        self.client = OpenAI(\n            api_key=self.api_key,\n            base_url=self.base_url\n        )\n\n        logger.info(\n            f\"Doubao model initialized: {model_name}, \"\n            f\"endpoint={self.endpoint_id}\"\n        )\n\n    @classmethod\n    def from_env(\n        cls,\n        model_name: str = \"doubao-vision\"\n    ) -> 'DoubaoVisionModel':\n        \"\"\"\n        从环境变量创建模型实例\n\n        环境变量：\n        - MIDSCENE_DOUBAO_API_KEY\n        - MIDSCENE_DOUBAO_BASE_URL (可选)\n        - MIDSCENE_DOUBAO_MODEL_NAME (推理接入点 ID)\n\n        Args:\n            model_name: 模型名称\n\n        Returns:\n            DoubaoVisionModel 实例\n        \"\"\"\n        return cls(model_name=model_name)\n\n    def call(\n        self,\n        messages: List[Dict[str, Any]],\n        stream: bool = False,\n        **kwargs\n    ) -> Dict[str, Any]:\n        \"\"\"\n        调用豆包模型\n\n        Args:\n            messages: 消息列表（OpenAI 格式）\n            stream: 是否流式输出\n            **kwargs: 其他参数\n\n        Returns:\n            包含 content 和 usage 的字典\n        \"\"\"\n        # 合并参数\n        request_params = {\n            \"model\": self.endpoint_id,  # 使用推理接入点 ID\n            \"messages\": messages,\n            \"stream\": stream,\n            **self.model_kwargs,\n            **kwargs\n        }\n\n        # 处理深度思考参数（豆包使用 thinking.type）\n        deep_think_value = None\n        if \"deep_think\" in request_params:\n            deep_think_value = request_params.pop(\"deep_think\")\n        elif \"deepThink\" in request_params:\n            deep_think_value = request_params.pop(\"deepThink\")\n\n        if deep_think_value is not None:\n            request_params.setdefault(\"extra_body\", {})\n            request_params[\"extra_body\"][\"config\"] = {\n                \"thinking\": {\n                    \"type\": \"enabled\" if deep_think_value else \"disabled\"\n                }\n            }\n            logger.debug(f\"Doubao deepThink mapped to thinking.type={request_params['extra_body']['config']['thinking']['type']}\")\n\n        logger.debug(f\"Calling Doubao model: {self.model_name} (endpoint: {self.endpoint_id})\")\n\n        try:\n            response = self.client.chat.completions.create(**request_params)\n\n            if stream:\n                # 流式响应处理\n                return {\"stream\": response}\n\n            # 提取响应内容\n            content = response.choices[0].message.content or \"\"\n\n            # 预处理豆包 bbox 格式（空格分隔 -> 逗号分隔）\n            content = self.preprocess_doubao_bbox_json(content)\n\n            # 提取 usage 信息\n            usage = None\n            if hasattr(response, 'usage') and response.usage:\n                usage = {\n                    \"prompt_tokens\": response.usage.prompt_tokens,\n                    \"completion_tokens\": response.usage.completion_tokens,\n                    \"total_tokens\": response.usage.total_tokens,\n                }\n\n            logger.debug(f\"Doubao response received: {len(content)} chars\")\n\n            return {\n                \"content\": content,\n                \"usage\": usage,\n                \"raw_response\": response\n            }\n\n        except Exception as e:\n            logger.error(f\"Doubao API call failed: {e}\")\n            raise\n\n    @staticmethod\n    def preprocess_doubao_bbox_json(input_str: str) -> str:\n        \"\"\"\n        预处理豆包 bbox JSON 格式\n\n        豆包可能返回空格分隔的 bbox 值：\"940 445 969 490\"\n        需要转换为逗号分隔：\"940,445,969,490\"\n\n        Args:\n            input_str: 原始 JSON 字符串\n\n        Returns:\n            处理后的 JSON 字符串\n        \"\"\"\n        if 'bbox' not in input_str:\n            return input_str\n\n        # 将 bbox 值中的空格替换为逗号\n        # 匹配模式：数字 + 空格 + 数字\n        while re.search(r'\\d+\\s+\\d+', input_str):\n            input_str = re.sub(r'(\\d+)\\s+(\\d+)', r'\\1,\\2', input_str)\n\n        return input_str\n\n    @staticmethod\n    def adapt_doubao_bbox(\n        bbox: Any,\n        width: int,\n        height: int\n    ) -> Tuple[int, int, int, int]:\n        \"\"\"\n        适配豆包 bbox 格式到标准格式\n        \n        对应 JS 版本: adaptDoubaoBbox (common.ts:108-192)\n\n        豆包支持多种 bbox 格式：\n        1. 字符串：\"x1 y1 x2 y2\" (空格分隔，0-1000 归一化坐标)\n        2. 字符串数组：[\"123 222\", \"789 100\"] 或 [\"123,222\", \"789,100\"]\n        3. 数组：[x1, y1, x2, y2] (0-1000 归一化坐标)\n        4. 数组：[x, y] (中心点坐标，自动创建小矩形)\n\n        Args:\n            bbox: bbox 值（字符串或数组）\n            width: 图像宽度（像素）\n            height: 图像高度（像素）\n\n        Returns:\n            标准 bbox: (left, top, right, bottom) 像素坐标\n        \"\"\"\n        # 使用统一的工具函数\n        from ....shared.utils import adapt_doubao_bbox as _adapt_doubao_bbox\n        return _adapt_doubao_bbox(bbox, width, height)\n\n    def get_model_name(self) -> str:\n        \"\"\"获取模型名称\"\"\"\n        return self.model_name\n\n    def get_model_family(self) -> str:\n        \"\"\"获取模型家族\"\"\"\n        if \"ui-tars\" in self.model_name:\n            return \"vlm-ui-tars-doubao\"\n        return \"doubao-vision\"\n\n    def validate_config(self) -> bool:\n        \"\"\"\n        验证配置是否有效\n\n        Returns:\n            配置是否有效\n        \"\"\"\n        if not self.api_key:\n            logger.error(\"Doubao API key is missing\")\n            return False\n        if not self.endpoint_id:\n            logger.error(\"Doubao endpoint ID is missing\")\n            return False\n        return True\n\n\n__all__ = [\"DoubaoVisionModel\"]\n"}}}}