1 line
33 KiB
JSON
1 line
33 KiB
JSON
{"repo": "benjiyaya/ominiui", "files": {"README.md": {"5e07675d47b2115896513d408c9609fbfac1bc0d": {"t": 1785674374.7877314, "n": 12454, "h": "7c5bbf959b48b1a7", "c": "# OminiUI\n\nA conversational AI interface for image and video generation, powered by **MCP (Model Context Protocol)** servers and any **OpenAI-compatible LLM**.\n\nDescribe what you want in natural language. The LLM reasons through your request, then calls the appropriate image/video tool automatically via MCP servers.\n\nMore Ongoing update and other development, feel free to check out in Patreon: https://patreon.com/aifuturetech\n\n<img width=\"1595\" height=\"1061\" alt=\"Screenshot 2026-05-12 060615\" src=\"https://github.com/user-attachments/assets/4d540ff2-51d5-4673-85d9-b1086a1d5f9e\" />\n\n<img width=\"1878\" height=\"1056\" alt=\"Screenshot 2026-05-12 060456\" src=\"https://github.com/user-attachments/assets/69e9ac61-507b-41e2-a38b-81c2a9b5d86b\" />\n\n\n## Features\n\n- **Multi-Engine Support** — Connect multiple MCP servers (HiDream, Flux, Kling, etc.) for different generation tasks\n- **Text-to-Image** — Generate images from detailed text prompts\n- **Image Editing** — Upload a photo and describe modifications\n- **Tool Calling** — LLM decides when to invoke tools based on conversation context\n- **Agent Workflow** — Multi-step creative tasks with planning and execution (stories, character references, scene composition)\n- **Enhance Prompt** — Toggle to refine prompts via OpenAI-compatible API before sending\n- **File Attachments** — Attach images, audio, and video files to your messages\n- **Add to Attachment** — Attach generated outputs back to composer for iterative editing\n- **Real-time Progress** — SSE streaming shows generation steps and preview thumbnails\n- **Auto-save** — Generated images are saved with timestamped filenames\n- **Chat History** — SQLite database persists all sessions, messages, and image references\n- **Model Auto-unload** — MCP server models are released from GPU after idle timeout, reloaded on demand\n- **MCP Settings Page** — Dedicated settings page with expandable server panels showing tool details\n- **Multi-session** — Sidebar manages multiple independent conversations\n- **Lightbox** — Click any generated image for full-size viewing and download\n\n## Architecture\n\n```\nBrowser (React + TypeScript)\n │\n ├── /api/chat → Express server → OpenAI-compatible LLM (tool calling)\n │\n ├── /api/workflow → Express server → DeepAgent (multi-step creative tasks)\n │\n ├── /api/tools/execute → Express server → MCP Client → MCP Servers (HiDream, Flux, etc.)\n │ ↕\n │ Tool Registry\n │\n ├── /api/mcp-servers → Express server → MCP Client Manager (connect/reconnect/status)\n │\n ├── /api/refine → Express server → OpenAI-compatible API (prompt enhancement)\n └── /api/sessions → Express server → SQLite (chat history)\n```\n\nThe Express backend orchestrates between the LLM and MCP servers. It sends user messages to the LLM with tool schemas, then routes any tool calls to the appropriate MCP server. Multiple MCP servers can be connected simultaneously.\n\n## Prerequisites\n\n| Requirement | Version | Notes |\n|-------------|---------|-------|\n| Node.js | 18+ | For the OminiUI server and frontend |\n| VRAM | varies | Depends on which MCP server models you run |\n\n> **Note:** You no longer need Python or CUDA locally. Image/video generation runs on separate MCP servers that can be on the same machine or remote.\n\n## Setup\n\n### 1. Install OminiUI\n\n```bash\ncd OminiUi\nnpm install\n```\n\n### 2. Configure environment\n\nCopy the example and edit `.env`:\n\n```bash\ncp .env.example .env\n```\n\nSet your LLM endpoint (any OpenAI-compatible API):\n\n```env\n# Examples:\n# OpenAI\nLLM_BASE_URL=https://api.openai.com/v1\nLLM_API_KEY=sk-...\nLLM_MODEL=gpt-4o\n\n# Ollama (local)\nLLM_BASE_URL=http://localhost:11434/v1\nLLM_API_KEY=ollama\nLLM_MODEL=qwen3:8b\n\n# Groq\nLLM_BASE_URL=https://api.groq.com/openai/v1\nLLM_API_KEY=gsk_...\nLLM_MODEL=llama-3.3-70b-versatile\n\n# Together\nLLM_BASE_URL=https://api.together.xyz/v1\nLLM_API_KEY=...\nLLM_MODEL=meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\n```\n\nSee [Configuration](#configuration) below for all options.\n\n### 3. Start OminiUI\n\n```bash\ncd OminiUi\nnpm run dev\n```\n\nThis starts both:\n- **Vite dev server** on `http://localhost:5173` (frontend with hot reload)\n- **Express API server** on `http://localhost:3001` (backend)\n\nOpen `http://localhost:5173` in your browser.\n\n### 4. Connect MCP Servers\n\n1. Click **Settings** in the sidebar\n2. Click **+ Add MCP Server**\n3. Enter the server details:\n - **Name**: unique identifier (e.g., `hidream`, `flux`, `kling`)\n - **Display Name**: human-readable name (e.g., `HiDream-O1-Image`)\n - **MCP URL**: the server's MCP endpoint (e.g., `http://localhost:8085/mcp`)\n4. Click **Connect**\n\nThe server will connect and list its available tools. You can expand the server panel to see all tools with their parameters.\n\n### MCP Server Examples\n\nAny MCP-compatible image/video generation server works. Common options:\n\n| Server | Description | Default Port |\n|--------|-------------|--------------|\n| HiDream-O1-Image | Text-to-image, image editing | 8085 |\n| Flux | Fast image generation | 8085 |\n| Kling | Video generation | 8085 |\n\n> **Note:** Each MCP server runs its own model. Make sure you have enough VRAM for the models you connect.\n\n## Configuration\n\nAll settings are in `.env` at the project root.\n\n### LLM Settings\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `LLM_BASE_URL` | OpenAI-compatible endpoint | `http://localhost:11434/v1` |\n| `LLM_API_KEY` | API key | `ollama` |\n| `LLM_MODEL` | Model for chat and tool calling | `qwen3:8b` |\n\n### Prompt Enhancement (Optional)\n\nUsed when the **Enhance Prompt** toggle is ON. Rewrites prompts into detailed image generation prompts.\n\n| Variable | Description |\n|----------|-------------|\n| `OPENAI_BASE_URL` | Any OpenAI-compatible API endpoint |\n| `OPENAI_API_KEY` | Your API key |\n| `OPENAI_MODEL` | Model name on the endpoint |\n\nLeave empty to disable prompt enhancement.\n\n### MCP Server Settings\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `MCP_CONFIG_PATH` | Path to MCP servers config file | `./mcp-servers.json` |\n| `MODEL_IDLE_TIMEOUT_MS` | Unload model after ms of inactivity | `600000` (10 min) |\n\n### Server Settings\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `PORT` | Express backend port | `3001` |\n| `RESULTS_DIR` | Directory to save generated images | `results` |\n\n## Usage\n\n### Generate an image\n\nType a description in the chat input:\n\n```\nA photorealistic cat wearing an astronaut helmet, floating in deep space\nwith Earth visible in the background. Cinematic lighting, 50mm lens.\n```\n\nThe LLM will call the appropriate image generation tool automatically.\n\n### Edit an image\n\n1. Click **Attach files** and upload a source photo\n2. Describe the edits you want:\n\n```\nMake this photo look like an oil painting in the style of Van Gogh\n```\n\nThe LLM will call the edit tool with the attached image.\n\n### Agent Workflow\n\nClick **Workflow** in the input toolbar to enable agent mode. This is for complex, multi-step creative tasks:\n\n```\nCreate a story with 3 scenes: a knight in a forest, then fighting a dragon,\nthen celebrating in a castle. Generate reference images for each scene.\n```\n\nThe agent will plan the workflow, generate character references, and compose each scene step by step.\n\n### Enhance Prompt\n\nClick **Enhance Prompt** in the input toolbar. Your prompt is rewritten into a detailed image generation prompt following the SCALIST framework (Subject, Composition, Action, Location, Image style, Specs, Text rendering).\n\nRequires `OPENAI_API_KEY` to be set.\n\n### Attach Files\n\nClick **Attach files** to upload images, audio, or video. You can attach up to 6 files. These are sent along with your message to the LLM for context.\n\n### Add Generated Output to Attachment\n\nWhen an image is generated, click **Add to attachment** to attach it to the composer for iterative editing.\n\n## MCP Settings Page\n\nClick **Settings** in the sidebar to open the MCP Settings page. Here you can:\n\n- **View all connected servers** with their status (connected/connecting/error/disconnected)\n- **Expand server panels** to see individual tools, descriptions, and input parameters\n- **Add/remove servers** dynamically\n- **Toggle servers** on/off without removing them\n- **Reconnect** failed servers\n\n## Project Structure\n\n```\nOminiUi/\n├── .env.example # Environment configuration template\n├── .gitignore # Git ignore rules\n├── mcp-servers.json # MCP server configurations\n├── package.json # Dependencies and scripts\n├── index.html # Vite entry HTML\n├── vite.config.ts # Vite config with API proxy\n├── tsconfig.json # TypeScript config (frontend)\n├── tsconfig.node.json # TypeScript config (server)\n├── server/\n│ ├── index.ts # Express server with all routes\n│ ├── chat.ts # Chat endpoint — forwards to LLM with tools\n│ ├── generate.ts # Tool execution — routes to MCP servers\n│ ├── agent.ts # DeepAgent workflow (multi-step creative tasks)\n│ ├── mcp-client.ts # MCP client manager (connections, tool registry)\n│ ├── mcp-config.ts # MCP server config persistence\n│ ├── model-tracker.ts # Model idle tracking and auto-unload\n│ ├── db.ts # SQLite database layer\n│ └── prompt-refine.ts # Prompt enhancement via OpenAI API\n└── src/\n ├── main.tsx # React entry point\n ├── App.tsx # Root component — view routing + session management\n ├── index.css # Global styles (dark theme)\n └── components/\n ├── Chat.tsx # Main chat interface + tool orchestration\n ├── MessageBubble.tsx # Renders text, tool calls, images\n ├── Sidebar.tsx # Conversation list + engine status\n ├── McpSettings.tsx # MCP Settings page with expandable panels\n └── Settings.tsx # Legacy settings dialog (deprecated)\n```\n\n## API Endpoints\n\n### Chat & Generation\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/chat` | Send messages to LLM with tool definitions |\n| `POST` | `/api/workflow` | Execute multi-step agent workflow |\n| `POST` | `/api/tools/execute` | Execute a tool call via MCP server |\n| `POST` | `/api/refine` | Enhance a prompt via OpenAI API |\n\n### MCP Servers\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/mcp-servers` | List all servers and their statuses |\n| `POST` | `/api/mcp-servers` | Add a new MCP server |\n| `PUT` | `/api/mcp-servers/:name` | Update server config (enable/disable, URL) |\n| `DELETE` | `/api/mcp-servers/:name` | Remove a server |\n| `POST` | `/api/mcp-servers/:name/reconnect` | Force reconnect a server |\n| `GET` | `/api/mcp-servers/:name/tools` | Get tools for a specific server |\n\n### Sessions\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/sessions` | List all sessions |\n| `POST` | `/api/sessions` | Create a session |\n| `PUT` | `/api/sessions/:id` | Rename a session |\n| `DELETE` | `/api/sessions/:id` | Delete a session |\n| `GET` | `/api/sessions/:id/messages` | Get messages for a session |\n| `POST` | `/api/sessions/:id/messages` | Save a message |\n\n## Troubleshooting\n\n| Problem | Solution |\n|---------|----------|\n| MCP server won't connect | Check the MCP URL and ensure the server is running |\n| `MCP error -32001: Request timed out` | Tool call exceeded 10min timeout — check MCP server logs |\n| Tool not found | Ensure the MCP server exposes the tool — check Settings page |\n| No tools available | Connect at least one MCP server in Settings |\n| Generation hangs | Check MCP server terminal for GPU errors |\n| Model stays in VRAM | Agent workflow auto-unloads; MCP servers unload after idle timeout |\n| Prompt enhancement fails | Check `OPENAI_API_KEY` is set in `.env` |\n| SSE reconnection spam | Already fixed — SSE retries disabled in transport config |\n\n## Scripts\n\n| Command | Description |\n|---------|-------------|\n| `npm run dev` | Start both frontend (5173) and backend (3001) |\n| `npm run dev:client` | Start only the Vite frontend |\n| `npm run dev:server` | Start only the Express backend |\n| `npm run build` | Production build to `dist/` |\n"}}, "HOW-TO.md": {"5e07675d47b2115896513d408c9609fbfac1bc0d": {"t": 1785674377.5018673, "n": 19109, "h": "f8f70e3d88b4957f", "c": "# OminiUI — How-To Guide\n\nA complete guide to installing, configuring, and using OminiUI with its MCP plugin architecture, DeepAgents workflow mode, and multiple image/video model support.\n\n---\n\n## Table of Contents\n\n1. [What is OminiUI](#what-is-ominiui)\n2. [Prerequisites](#prerequisites)\n3. [Installation](#installation)\n4. [Configuration](#configuration)\n5. [Starting the App](#starting-the-app)\n6. [Using the App](#using-the-app)\n7. [MCP Plugin System](#mcp-plugin-system)\n8. [Writing Your Own MCP Model Server](#writing-your-own-mcp-model-server)\n9. [DeepAgents Workflow Mode](#deepagents-workflow-mode)\n11. [Configuration Reference](#configuration-reference)\n12. [API Reference](#api-reference)\n13. [Troubleshooting](#troubleshooting)\n14. [Project Structure](#project-structure)\n\n---\n\n## What is OminiUI\n\nOminiUI is a modular AI creative studio. It connects to any number of image and video generation models through the **MCP (Model Context Protocol)** plugin system. Each model runs as an independent MCP server — adding a new model means spinning up a new MCP service and registering it, no code changes needed.\n\nThe chat is powered by an LLM (Ollama by default, configurable to any OpenAI-compatible endpoint) that decides which tool to call based on your request.\n\n**Core features:**\n- Text-to-image, image editing, subject-driven personalization\n- Multiple model support via MCP plugins\n- Workflow mode for multi-step creative projects (stories, character references, scene composition)\n- Prompt enhancement via OpenAI-compatible API\n- Real-time generation progress with SSE streaming\n- Auto-save, chat history, multi-session\n\n---\n\n## Prerequisites\n\n| Requirement | Version | Notes |\n|-------------|---------|-------|\n| Node.js | 18+ | For the Express server and React frontend |\n| LLM API key | — | Any OpenAI-compatible API (OpenAI, Groq, Together, Ollama, etc.) |\n\n---\n\n## Installation\n\n### 1. Clone and install\n\n```bash\ngit clone https://github.com/benjiyaya/HiDream-O1-Image-OminiUI.git\ncd HiDream-O1-Image-OminiUI\nnpm install\n```\n\n### 2. Configure\n\n```bash\ncp .env.example .env\n```\n\nSet your LLM. Works with any OpenAI-compatible API:\n\n```bash\n# OpenAI\nLLM_BASE_URL=https://api.openai.com/v1\nLLM_API_KEY=sk-your-key\nLLM_MODEL=gpt-4o\n\n# Groq (fast, free tier available)\nLLM_BASE_URL=https://api.groq.com/openai/v1\nLLM_API_KEY=gsk_...\nLLM_MODEL=llama-3.3-70b-versatile\n\n# Ollama (local, free)\nLLM_BASE_URL=http://localhost:11434/v1\nLLM_API_KEY=ollama\nLLM_MODEL=qwen3:8b\n\n# Together, Fireworks, LM Studio, vLLM — same pattern\n```\n\n### 3. Start\n\n```bash\nnpm run dev\n```\n\nOpen http://localhost:5173. The app works immediately for text chat. To enable image/video generation, add an MCP server via the Settings panel (gear icon in the sidebar).\n\n---\n\n## Configuration\n\nAll settings are in `.env` at the project root.\n\n### LLM (any OpenAI-compatible API)\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `LLM_BASE_URL` | API endpoint | `http://localhost:11434/v1` |\n| `LLM_API_KEY` | API key | `ollama` |\n| `LLM_MODEL` | Model name | — (required) |\n\nFalls back to `OLLAMA_*` env vars if `LLM_*` not set.\n\n### Prompt Enhancement (optional, separate from chat LLM)\n\n| Variable | Description |\n|----------|-------------|\n| `OPENAI_BASE_URL` | Any OpenAI-compatible endpoint |\n| `OPENAI_API_KEY` | API key |\n| `OPENAI_MODEL` | Model name |\n\nLeave empty to disable.\n\n### Server\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `PORT` | Express backend port | `3001` |\n| `RESULTS_DIR` | Directory to save generated images | `results` |\n\n### MCP Servers\n\n| Variable | Description | Default |\n|----------|-------------|---------|\n| `MCP_CONFIG_PATH` | Path to `mcp-servers.json` | `./mcp-servers.json` |\n\n---\n\n## Starting the App\n\n### Full stack (recommended)\n\n```bash\nnpm run dev\n```\n\nRuns both Vite frontend and Express backend concurrently.\n\n### Individual services\n\n```bash\n# Frontend only (hot reload)\nnpm run dev:client\n\n# Backend only (auto-restart on changes)\nnpm run dev:server\n```\n\n### Production build\n\n```bash\nnpm run build\nnpm run preview\n```\n\n---\n\n## Using the App\n\n### Generate an image\n\nType a description in the chat:\n\n```\nA photorealistic cat wearing an astronaut helmet, floating in deep space\nwith Earth visible in the background. Cinematic lighting, 50mm lens.\n```\n\nThe LLM automatically calls the appropriate image generation tool.\n\n### Edit an image\n\n1. Click **Attach images** and upload a source photo\n2. Describe the edits:\n\n```\nMake this photo look like an oil painting in the style of Van Gogh\n```\n\nThe LLM calls `edit_image` with your attached image.\n\n### Subject-driven generation\n\n1. Attach 2-6 reference images\n2. Describe the output:\n\n```\nGenerate a new portrait of this person in a cyberpunk city, neon lighting\n```\n\nThe LLM calls `subject_driven_image` with your references.\n\n### Prompt Enhancement\n\nClick **Enhance Prompt** in the input toolbar. When enabled, your prompt is sent to an OpenAI-compatible API that rewrites it following the SCALIST framework (Subject, Composition, Action, Location, Image style, Specs, Text rendering).\n\nRequires `OPENAI_API_KEY` in `.env`.\n\n### Workflow Mode\n\nClick the **Workflow** button (robot icon) to enable agent workflow mode. In this mode, the DeepAgents SDK takes over for complex multi-step creative tasks:\n\n- Story planning with scene breakdowns\n- Character reference generation\n- Background and environment creation\n- Scene composition with references\n- Video generation from image references\n\nSee [DeepAgents Workflow Mode](#deepagents-workflow-mode) for details.\n\n---\n\n## MCP Plugin System\n\nOminiUI uses MCP (Model Context Protocol) to discover and communicate with image/video model servers. Each model runs as an independent MCP server using streamable HTTP transport.\n\n### How it works\n\n1. On startup, OminiUI reads `mcp-servers.json` and connects to each enabled server\n2. Each server exposes tools via MCP's `tools/list` endpoint\n3. Tools are namespaced as `{serverName}_{toolName}` (e.g., `hidream_create_image`)\n4. The LLM receives all discovered tools and picks the right one\n5. Tool calls are routed to the correct MCP server\n6. Results (images, videos) flow back through the MCP protocol\n\n**Note:** MCP server adapters live in their respective model projects (e.g., HiDream's adapter is in the HiDream repo), not in OminiUI. This project is the MCP client only.\n\n### Managing MCP servers\n\n#### Via the UI\n\n1. Open the **Settings** panel (gear icon in the sidebar footer)\n2. Click **+ Add MCP Server**\n3. Fill in:\n - **Name**: Short identifier (alphanumeric, used as tool prefix) — e.g., `flux`\n - **Display Name**: Human-readable name — e.g., `FLUX.1`\n - **MCP URL**: Server endpoint — e.g., `http://localhost:8081/mcp`\n - **Idle timeout**: Auto-unload timeout in ms (0 = disabled)\n4. Click **Connect**\n\nThe server appears in the list with a status indicator:\n- Green = connected\n- Yellow = connecting\n- Red = error (click Reconnect)\n- Gray = disconnected\n\n#### Via config file\n\nEdit `mcp-servers.json`:\n\n```json\n{\n \"servers\": [\n {\n \"name\": \"hidream\",\n \"displayName\": \"HiDream-O1-Image\",\n \"url\": \"http://localhost:8080/mcp\",\n \"enabled\": true,\n \"idleTimeoutMs\": 600000\n },\n {\n \"name\": \"flux\",\n \"displayName\": \"FLUX.1\",\n \"url\": \"http://localhost:8081/mcp\",\n \"enabled\": true\n }\n ]\n}\n```\n\nRestart the app after editing the config file, or use the Settings UI for live changes.\n\n#### Via API\n\n```bash\n# List servers\ncurl http://localhost:3001/api/mcp-servers\n\n# Add a server\ncurl -X POST http://localhost:3001/api/mcp-servers \\\n -H 'Content-Type: application/json' \\\n -d '{\"name\":\"flux\",\"displayName\":\"FLUX.1\",\"url\":\"http://localhost:8081/mcp\",\"enabled\":true}'\n\n# Reconnect a server\ncurl -X POST http://localhost:3001/api/mcp-servers/flux/reconnect\n\n# Remove a server\ncurl -X DELETE http://localhost:3001/api/mcp-servers/flux\n\n# List tools for a server\ncurl http://localhost:3001/api/mcp-servers/hidream/tools\n```\n\n### Config fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `name` | string | Yes | Short unique ID, used as tool namespace prefix. Alphanumeric, hyphens, underscores. |\n| `displayName` | string | Yes | Human-readable name shown in the UI |\n| `url` | string | Yes | MCP server endpoint URL (streamable HTTP) |\n| `enabled` | boolean | No | Whether to connect on startup (default: `true`) |\n| `idleTimeoutMs` | number | No | Auto-unload timeout in ms. Only for servers that support load/unload. |\n\n---\n\n## Writing Your Own MCP Model Server\n\nAny image or video generation pipeline can be exposed as an MCP server. Here's how.\n\n### Python (using `mcp` package)\n\n```bash\npip install mcp uvicorn starlette\n```\n\nMinimal example:\n\n```python\nimport json\nimport httpx\nfrom mcp.server import Server\nfrom mcp.server.streamable_http import StreamableHTTPServerTransport\nfrom mcp.types import Tool, TextContent, ImageContent, CallToolResult\nfrom starlette.applications import Starlette\nfrom starlette.routing import Route\n\nserver = Server(\"my-image-model\")\n\n@server.list_tools()\nasync def list_tools():\n return [\n Tool(\n name=\"generate_image\",\n description=\"Generate an image from a text prompt\",\n inputSchema={\n \"type\": \"object\",\n \"properties\": {\n \"prompt\": {\"type\": \"string\", \"description\": \"Image description\"},\n \"width\": {\"type\": \"number\", \"description\": \"Width in pixels\"},\n \"height\": {\"type\": \"number\", \"description\": \"Height in pixels\"},\n },\n \"required\": [\"prompt\"],\n },\n )\n ]\n\n@server.call_tool()\nasync def call_tool(name: str, arguments: dict):\n if name == \"generate_image\":\n # Call your model's inference API here\n # image_base64 = your_model.generate(arguments[\"prompt\"])\n # return CallToolResult(\n # content=[ImageContent(type=\"image\", data=image_base64, mimeType=\"image/png\")]\n # )\n pass\n\n# ASGI app\ntransport = StreamableHTTPServerTransport(mcp_session_id=None)\n\nasync def handle_mcp(scope, receive, send):\n await transport.handle_request(scope, receive, send)\n\napp = Starlette(routes=[Route(\"/mcp\", endpoint=handle_mcp, methods=[\"GET\", \"POST\", \"DELETE\"])])\n\nif __name__ == \"__main__\":\n import uvicorn\n uvicorn.run(app, host=\"0.0.0.0\", port=8081)\n```\n\n### Node.js (using `@modelcontextprotocol/sdk`)\n\n```bash\nnpm install @modelcontextprotocol/sdk\n```\n\n```typescript\nimport { McpServer } from \"@modelcontextprotocol/sdk/server/mcp.js\";\nimport { StreamableHTTPServerTransport } from \"@modelcontextprotocol/sdk/server/streamableHttp.js\";\n\nconst server = new McpServer({ name: \"my-image-model\", version: \"1.0.0\" });\n\nserver.tool(\"generate_image\", \"Generate an image from a text prompt\", {\n prompt: { type: \"string\", description: \"Image description\" },\n}, async ({ prompt }) => {\n // Call your model's inference API here\n // const imageBase64 = await myModel.generate(prompt);\n return { content: [{ type: \"image\", data: imageBase64, mimeType: \"image/png\" }] };\n});\n\n// Set up streamable HTTP transport on your HTTP server\n```\n\n### Requirements for MCP servers\n\n1. **Streamable HTTP transport** at the `/mcp` path (or custom path)\n2. **`tools/list`** — return tool definitions with JSON Schema input\n3. **`tools/call`** — execute tools and return results\n4. **Image results** — return as `ImageContent` with base64 data\n5. **Progress notifications** — optional, for long-running generations\n\n### Tool naming convention\n\nThe MCP server's tool names become `{serverName}_{toolName}` in OminiUI. For example, if your server exposes `generate_image` and you name it `flux`, the LLM sees `flux_generate_image`.\n\n---\n\n## DeepAgents Workflow Mode\n\nOminiUI integrates [DeepAgents](https://www.npmjs.com/package/deepagents) (by LangChain) for complex multi-step creative workflows. This mode is designed for tasks that require planning, multiple generations, and iterative refinement.\n\n### When to use workflow mode\n\n- **Story illustration**: Plan scenes, generate character references, then compose each scene\n- **Character design**: Create reference sheets with multiple angles, then generate scenes\n- **Scene composition**: Generate backgrounds, characters, and props separately, then composite\n- **Video production**: Generate keyframes/image references, then use for video generation\n\n### How it works\n\n1. Enable **Workflow** mode (robot icon in the input toolbar)\n2. Describe your project in natural language\n3. The DeepAgent:\n - Plans the work using `write_todos` (task breakdown)\n - Generates character reference images\n - Generates background/location images\n - Composes final scenes using references\n - Tracks progress across steps\n4. Results are returned in the chat\n\n### Example: Story illustration\n\n```\nEnable Workflow, then type:\n\nCreate a short illustrated story about a fox who finds a magical forest.\nPlan 3 scenes, generate character references for the fox first,\nthen generate each scene with consistent character appearance.\n```\n\nThe agent will:\n1. Break the story into 3 scenes\n2. Generate a fox character reference image\n3. Generate Scene 1 with the fox reference\n4. Generate Scene 2 with the fox reference\n5. Generate Scene 3 with the fox reference\n\n### Architecture\n\n```\nUser message\n │\n ▼\nDeepAgent (LangGraph)\n │\n ├── write_todos (task planning)\n ├── read_file / write_file (working memory)\n ├── task (sub-agent delegation)\n │\n └── MCP tools (image/video generation)\n │\n ├── hidream_create_image\n ├── hidream_edit_image\n ├── flux_create_image\n └── ... (any registered MCP tools)\n```\n\n---\n\n## Configuration Reference\n\n### `.env` — all options\n\n```bash\n# ── LLM (any OpenAI-compatible API) ──\nLLM_BASE_URL=https://api.openai.com/v1\nLLM_API_KEY=sk-your-key\nLLM_MODEL=gpt-4o\n\n# ── Prompt Enhancement (optional) ──\nOPENAI_BASE_URL=\nOPENAI_API_KEY=\nOPENAI_MODEL=\n\n# ── Results directory ──\nRESULTS_DIR=results\n\n# ── MCP Servers Config ──\nMCP_CONFIG_PATH=./mcp-servers.json\n\n# ── Server ──\nPORT=3001\n```\n\n### `mcp-servers.json` — MCP server registry\n\n```json\n{\n \"servers\": [\n {\n \"name\": \"hidream\",\n \"displayName\": \"HiDream-O1-Image\",\n \"url\": \"http://localhost:8080/mcp\",\n \"enabled\": true,\n \"idleTimeoutMs\": 600000\n }\n ]\n}\n```\n\n---\n\n## API Reference\n\n### Chat & Generation\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `POST` | `/api/chat` | Send messages to LLM with tool definitions |\n| `POST` | `/api/tools/execute` | Execute a tool via MCP client |\n| `GET` | `/api/tools/stream/:jobId` | SSE progress stream for tool execution |\n| `POST` | `/api/workflow` | Run a DeepAgent workflow |\n| `POST` | `/api/refine` | Enhance a prompt via OpenAI-compatible API |\n| `POST` | `/api/generate` | Direct generation (backward-compatible alias) |\n\n### MCP Server Management\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/mcp-servers` | List all servers with status |\n| `POST` | `/api/mcp-servers` | Add a new server |\n| `PUT` | `/api/mcp-servers/:name` | Update server config |\n| `DELETE` | `/api/mcp-servers/:name` | Remove a server |\n| `POST` | `/api/mcp-servers/:name/reconnect` | Force reconnect |\n| `GET` | `/api/mcp-servers/:name/tools` | List tools for a server |\n\n### Model Info\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/model` | LLM model name + MCP server statuses |\n| `GET` | `/api/model/status` | MCP server statuses + idle tracker info |\n\n### Sessions\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| `GET` | `/api/sessions` | List all sessions |\n| `POST` | `/api/sessions` | Create a session |\n| `PUT` | `/api/sessions/:id` | Rename a session |\n| `DELETE` | `/api/sessions/:id` | Delete a session |\n| `GET` | `/api/sessions/:id/messages` | Get messages for a session |\n| `POST` | `/api/sessions/:id/messages` | Save a message |\n| `DELETE` | `/api/sessions/:id/messages` | Clear all messages in a session |\n\n---\n\n## Troubleshooting\n\n### Common issues\n\n| Problem | Solution |\n|---------|----------|\n| LLM connection refused | Check `LLM_BASE_URL` and `LLM_API_KEY` in `.env` |\n| Model not found | Verify `LLM_MODEL` matches your API provider |\n| HiDream API not reachable | Start the Flask server on port 7860 |\n| Generation hangs | Check the MCP server / Flask terminal for GPU errors |\n| Prompt enhancement fails | Set `OPENAI_API_KEY` in `.env` |\n| MCP server shows red status | Click Reconnect in Settings, or check the server is running |\n| No tools available | No MCP servers connected — add one in Settings |\n\n### Checking MCP server status\n\n```bash\n# Via API\ncurl http://localhost:3001/api/mcp-servers | python -m json.tool\n\n# Via the UI\n# Click the gear icon in the sidebar footer\n```\n\n### Logs\n\nThe Express server logs to stdout. Look for:\n- `[startup]` — MCP server connections on boot\n- `[mcp-client]` — tool discovery and server status\n- `[generate]` — tool execution requests\n- `[chat]` — LLM communication\n- `[workflow]` — DeepAgent execution\n\n---\n\n## Project Structure\n\n```\nOminiUI/\n├── .env.example # Environment config template\n├── .env # Your local config (not committed)\n├── mcp-servers.json # MCP server registry\n├── app.py # Patched HiDream Flask API\n├── package.json # Node.js dependencies\n├── index.html # Vite entry point\n├── vite.config.ts # Vite dev server + API proxy\n├── tsconfig.json # TypeScript config (frontend)\n├── tsconfig.node.json # TypeScript config (server)\n│\n├── server/ # Express backend (TypeScript)\n│ ├── index.ts # Entry point, all routes\n│ ├── chat.ts # Chat handler (LLM + dynamic tools)\n│ ├── generate.ts # Generic tool execution router\n│ ├── agent.ts # DeepAgents workflow integration\n│ ├── mcp-client.ts # MCP client manager\n│ ├── mcp-config.ts # Config file management\n│ ├── model-tracker.ts # Multi-server idle tracking\n│ ├── prompt-refine.ts # Prompt enhancement\n│ └── db.ts # SQLite database layer\n│\n├── src/ # React frontend (TypeScript)\n│ ├── main.tsx # React entry point\n│ ├── App.tsx # Root component, session management\n│ ├── index.css # Global dark-theme styles\n│ └── components/\n│ ├── Chat.tsx # Chat UI + tool execution + workflow toggle\n│ ├── MessageBubble.tsx # Message rendering (text, tools, images)\n│ ├── Sidebar.tsx # Session list + model info + settings\n│ └── Settings.tsx # MCP server management UI\n│\n└── data/\n └── ominiui.db # SQLite database (auto-created)\n```\n"}}}} |