1 line
19 KiB
JSON
1 line
19 KiB
JSON
{"repo": "armaggheddon/gstvlmvision", "files": {"README.md": {"0e5648e737643c6e0aeb93616856c929c718d43a": {"t": 1785674454.048284, "n": 18423, "h": "b247a0a3cbc742b5", "c": "<div id=\"top\"></div>\n<br/>\n<br/>\n<br/>\n\n<p align=\"center\">\n <img width=\"400\" src=\"docs/images/gstvlmvision_logo.png\">\n</p>\n<h1 align=\"center\">\n <a href=\"https://github.com/Armaggheddon/GstVlmVision\">GstVlmVision</a>\n</h1>\n<p align=\"center\">\n <a href=\"https://github.com/Armaggheddon/GstVlmVision/commits/master\">\n <img src=\"https://img.shields.io/github/last-commit/Armaggheddon/GstVlmVision\">\n </a>\n <a href=\"https://github.com/Armaggheddon/GstVlmVision\">\n <img src=\"https://img.shields.io/badge/Maintained-yes-green.svg\">\n </a>\n <a href=\"https://github.com/Armaggheddon/GstVlmVision/issues\">\n <img src=\"https://img.shields.io/github/issues/Armaggheddon/GstVlmVision\">\n </a>\n <a href=\"https://github.com/Armaggheddon/GstVlmVision/blob/master/LICENSE\">\n <img src=\"https://img.shields.io/github/license/Armaggheddon/GstVlmVision\">\n </a>\n</p>\n<p align=\"center\">\n Transform your media pipelines with AI-powered insights! 🎥🤖\n <br/>\n <br/>\n <a href=\"https://github.com/Armaggheddon/GstVlmVision/issues\">Report Bug</a>\n •\n <a href=\"https://github.com/Armaggheddon/GstVlmVision/issues\">Request Feature</a>\n</p>\n\n---\n\nEver wondered what a GStreamer pipeline would say if it could talk? Now it can (almost)! With **GstVlmVision** (formerly GstGeminiVision), you can inject the power of any OpenAI-compatible vision-language model directly into your GStreamer media pipelines. Turn your video streams into insightful descriptions, automate content analysis, or just have some fun making your videos self-aware! 🤖🎬\n\nhttps://github.com/user-attachments/assets/91c23d18-2409-410c-a8c1-2cc841419003\n\n---\n\n## 🤨 What is GstVlmVision?\n\nGstVlmVision is a GStreamer plugin that acts as a bridge between your live video or image streams and any OpenAI-compatible vision-language model API. It periodically captures frames, sends them to a VLM endpoint via `/v1/chat/completions`, and then makes the generated description available through GstMeta, GObject signals, or GstBus messages.\n\nSupported providers include OpenAI, Google Gemini (via OpenAI compatibility), vLLM, Ollama, and any `/v1/chat/completions`-compatible server.\n\nImagine:\n* Generating visual descriptions for accessibility purposes.\n* Creating a security camera that describes what it sees.\n* Building interactive art installations that react to visual input.\n* Automating video content moderation and tagging.\n* ...the possibilities are as vast as your imagination (and the VLM's capabilities)!\n\n---\n\n## 🌟 Features\n\n- **OpenAI Chat Completions Multimodal** -- standard `/v1/chat/completions` with text + image_url content blocks\n- **Multiple Output Modes** -- bitmask-selectable: GstMeta, GObject signal, GstBus message ([details](docs/output-modes.md))\n- **Template Override Mode** -- `{{PLACEHOLDER}}` templates for nonstandard API providers ([details](docs/template-mode.md))\n- **Async HTTP with Concurrency Control** -- threaded dispatch with configurable `max-inflight` and `timeout`\n- **JPEG Frame Encoding** -- raw video frames encoded to JPEG before sending\n- **Docker Support** -- pre-configured build environment ([details](#-docker))\n- **Python Bindings** -- GObject Introspection via `gi.repository.GstVlmVision`\n\n---\n\n## 🎬 See it in Action!\n\n* Show a `videotestsrc` pipeline running with the plugin.\n* Display the console output from `vlm_vision_example.py` or `vlm_vision_example.c` showing the descriptions.\n* *Bonus:* If you have a more complex demo (e.g., overlaying text on video), showcase that!\n\n\n\n\n```bash\ngst-launch-1.0 -m videotestsrc ! videoconvert ! \\\n vlmvision base-url=\"https://api.openai.com\" api-key=\"sk-...\" \\\n model=\"gpt-4o\" profile=\"openai\" \\\n user-prompt=\"What do you see?\" output-mode=4 \\\n ! videoconvert ! autovideosink\n```\n```console\nSetting pipeline to PLAYING state...\nPipeline running...\nPress Ctrl+C to quit\nPipeline state changed from NULL to READY\nPipeline state changed from READY to PAUSED\nPipeline state changed from PAUSED to PLAYING\n\n=================================\nFrame time: 0.000000000 (PTS: 0)\nDescription: That's a color bars test pattern, used to adjust color settings on television screens.\n\n=================================\n\n^CInterrupt received, stopping...\nCleaning up...\nPipeline stopped.\n```\n\n---\n\n## 🚀 Quick Start\n\n```bash\ngst-launch-1.0 -m videotestsrc ! videoconvert ! \\\n vlmvision base-url=\"http://localhost:8765\" api-key=\"dummy\" \\\n model=\"gpt-4o\" profile=\"openai\" \\\n user-prompt=\"What do you see?\" output-mode=4 \\\n ! videoconvert ! autovideosink\n```\n\n---\n\n## 🛠️ Building from Source\n\nReady to give your GStreamer pipelines a voice? Let's go!\n\n### Prerequisites\n\n- **GStreamer:** Core GStreamer libraries and development files (version 1.16+ recommended).\n- **Build Tools:** `meson`, `ninja`, `gcc` (or your C compiler), `pkg-config`.\n- **Dependencies for the Plugin:**\n - `libglib2.0-dev`\n - `libgstreamer-plugins-base1.0-dev`\n - `libcurl4-openssl-dev` (or your system's cURL dev package)\n - `libjson-c-dev`\n - `libjpeg-dev`\n - `libgirepository1.0-dev` & `gobject-introspection` (for GObject Introspection, used by Python bindings)\n- **Python 3** (for the Python example):\n - `python3-gi`\n - `python3-gst-1.0`\n\n### Building the Plugin\n\n1. **Clone the repository (if you haven't already):**\n ```bash\n git clone https://github.com/Armaggheddon/GstVlmVision.git\n cd GstVlmVision\n ```\n\n2. **Navigate to the plugin directory:**\n ```bash\n cd gst-vlm-plugin\n ```\n\n3. **Configure and build with Meson & Ninja:**\n First, set up the build directory using Meson. The `--prefix=/usr` ensures that a subsequent install places files in standard system locations.\n ```bash\n meson setup build --prefix=/usr --buildtype=release\n ```\n Then, compile the plugin:\n ```bash\n ninja -C build\n ```\n Your compiled plugin shared object (e.g., `libgstvlmvision.so`) will be located in the `gst-vlm-plugin/build/src/` directory.\n\n4. **Install the Plugin (Optional, but Recommended for System-Wide Access):**\n To make the plugin and its development files available system-wide, run the install command (this usually requires root privileges):\n ```bash\n sudo ninja -C build install\n ```\n This command will copy the necessary files to standard system locations. Based on a typical installation with `--prefix=/usr`, the files will be placed as follows:\n * The plugin library: `libgstvlmvision.so` to `/usr/lib/x86_64-linux-gnu/gstreamer-1.0/`\n * GObject Introspection data:\n * `GstVlmVision-1.0.gir` to `/usr/share/gir-1.0/`\n * `GstVlmVision-1.0.typelib` to `/usr/lib/x86_64-linux-gnu/girepository-1.0/`\n * Pkg-config file: `gstvlmvision.pc` to `/usr/lib/x86_64-linux-gnu/pkgconfig/`\n\n *(Note: The exact paths like `x86_64-linux-gnu` might vary slightly based on your Linux distribution's multiarch setup.)*\n\n After installation, GStreamer should be able to automatically discover the plugin. Verify with:\n ```bash\n gst-inspect-1.0 vlmvision\n ```\n\nFor development without system install, you can set environment variables instead:\n```bash\nexport GST_PLUGIN_PATH=$(pwd)/build:$GST_PLUGIN_PATH\nexport GI_TYPELIB_PATH=$(pwd)/build:$GI_TYPELIB_PATH\n```\n\n---\n\n## ▶️ Running Examples\n\nSet at minimum an API key:\n\n```bash\n# OpenAI\nexport VLM_API_KEY=sk-...\n# or Gemini via OpenAI endpoint\nexport VLM_API_KEY=<google-api-key>\nexport VLM_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai\nexport VLM_MODEL=gemini-2.0-flash\n# or local vLLM (no key needed)\nexport VLM_BASE_URL=http://localhost:8000\nexport VLM_MODEL=llava-v1.6-34b\n```\n\n### C Example\nNavigate to the examples directory and compile/run:\n```bash\ncd examples && make all\n./bin/vlm_vision_example\n./bin/vlm_vision_example2 [video-uri]\n```\n\n### Python Example\n```bash\ncd examples\npython3 vlm_vision_example.py\n```\n\nYou should see descriptions from the VLM printed to the console! 🚀\n\n---\n\n## ⚙️ Property Reference\n\nFor a **full, detailed list** of all properties, their types, default values, ranges, and descriptions, please refer to the output of `gst-inspect-1.0`:\n\n➡️ **[View Full Plugin Details (gst-inspect-1.0 output)](docs/gst_inspect_vlmvision.txt)** ⬅️\n\nYou can also generate this information yourself by running:\n```bash\ngst-inspect-1.0 vlmvision\n```\n\nHere's a summary of the key properties:\n\n| Property | Type | Default | Description |\n|---|---|---|---|\n| `base-url` | string | `https://api.openai.com` | Base URL for the API endpoint |\n| `api-key` | string | NULL | API key; NULL = no auth header sent |\n| `model` | string | `\"\"` (must set) | Model name (e.g. `gpt-4o`, `gemini-2.0-flash`) |\n| `system-prompt` | string | NULL | System prompt |\n| `user-prompt` | string | `\"Describe what you see...\"` | User prompt about the image(s) |\n| `profile` | string | `openai` | API compatibility profile |\n| `analysis-interval` | double | 5.0 | Seconds between analyses (0.1--3600) |\n| `frames-per-request` | int | 1 | Images per request (1--10) |\n| `request-mode` | enum | `chat-completions` | API request type |\n| `stop-sequences` | GStrv | NULL | Stop generation strings |\n| `temperature` | double | 1.0 | Sampling temperature (0.0--2.0) |\n| `max-output-tokens` | int | 800 | Max tokens to generate |\n| `top-p` | double | 0.8 | Nucleus sampling (0.0--1.0) |\n| `output-mode` | flags | 3 | Bitmask: 1=meta, 2=signal, 4=bus, 8=json |\n| `timeout` | int | 30 | HTTP timeout in seconds (1--600) |\n| `max-inflight` | int | 1 | Max concurrent requests (1--16) |\n| `queue-policy` | enum | `drop` | Backpressure: drop or block |\n| `error-policy` | enum | `skip` | Error handling: skip, bus, or signal |\n| `template-body` | string | NULL | File path for body template override |\n| `template-headers` | string | NULL | File path for headers template override |\n| `template-response` | string | NULL | JSON path for response extraction |\n\nFull details with usage examples: [docs/properties.md](docs/properties.md)\n\n---\n\n## 📤 Output Modes\n\n`output-mode` is a bitmask combining:\n\n| Value | Flag | Effect |\n|---|---|---|\n| 1 | META | Attach `GstVlmVisionMeta` to downstream buffers |\n| 2 | SIGNAL | Emit `description-received` GObject signal |\n| 4 | BUS | Post `vlmvision-result` GstBus message |\n| 8 | JSON | Include raw JSON in bus message |\n\nDefault: 3 (meta + signal). See [docs/output-modes.md](docs/output-modes.md) for signal signatures, bus message structures, and code examples.\n\n---\n\n## ⚠️ Error Handling\n\n| Policy | Value | Behaviour |\n|---|---|---|\n| `skip` | default | Silently skip API errors |\n| `bus` | -- | Post `vlmvision-error` GstBus message with `{ message, http-status }` |\n| `signal` | -- | Emit `analysis-error(error_message, http_status)` GObject signal |\n\n---\n\n## 🧩 Template Mode\n\nFor APIs that don't follow the standard OpenAI format, use template files to override request serialization and response parsing.\n\n| Property | Purpose |\n|---|---|\n| `template-body` | File path -- JSON body with `{{PLACEHOLDER}}` substitutions |\n| `template-headers` | File path -- per-line header templates |\n| `template-response` | JSON path (e.g. `$.choices[0].message.content`) |\n\nSupported placeholders: `{{BASE_URL}}`, `{{MODEL}}`, `{{SYSTEM_PROMPT}}`, `{{USER_PROMPT}}`, `{{IMAGE_0_BASE64}}`, `{{IMAGE_0_DATA_URL}}`, `{{IMAGE_ARRAY_JSON}}`, `{{TEMPERATURE}}`, `{{MAX_TOKENS}}`, `{{TOP_P}}`, `{{SCHEMA_JSON}}`, `{{REQUEST_ID}}`, `{{STOP_SEQUENCES_JSON}}`.\n\nFull guide with examples: [docs/template-mode.md](docs/template-mode.md)\n\n---\n\n## 🐳 Docker: Your AI-Powered Media Lab in a Box!\n\nWant to dive straight into the action without wrestling with dependencies? Our Docker setup is your golden ticket! 🎟️ It's like having a pre-configured media lab, ready to build, test, and run GstVlmVision with just a few commands. No more \"it works on my machine\" – it'll work in *this* machine!\n\n**Step 1: Build the All-Powerful Docker Image**\n\nFirst, conjure up your Docker image. This image contains all the tools and magic needed. From your `GstVlmVision` project root:\n```bash\ndocker build -t gst-vlm-vision .\n```\n*(Psst! If you've already built it, you can skip this step unless you've made changes to the Dockerfile or the plugin build process itself.)*\n\n**Step 2: Unleash the Entrypoint Script!**\n\nThe Docker image comes with a super-handy `entrypoint.sh` script that acts as your mission control. You tell it what to do, and it handles the nitty-gritty. Here are your commands, Captain:\n\n- **`build` (Default Action): Compile the Mighty Plugin!**\n Just want to build the main `vlmvision` plugin? This is your command. It compiles the plugin but doesn't install it system-wide in the container. Perfect for a quick compilation check.\n ```bash\n # Run from your GstVlmVision project root\n docker run --rm \\\n --volume $(pwd)/gst-vlm-plugin:/builder \\\n gst-vlm-vision build\n ```\n\n- **`build-examples`: Build the Plugin & The Examples!**\n This action first ensures the main `vlmvision` plugin is built and installed *inside the container*. Then, it gallops over to your examples directory (`/examples` in the container) and builds them (either using the Makefile or compiling C files directly).\n ```bash\n # Run from your GstVlmVision project root\n docker run --rm \\\n --volume $(pwd)/gst-vlm-plugin:/builder \\\n --volume $(pwd)/examples:/examples \\\n gst-vlm-vision build-examples\n ```\n\n- **`test-examples <example_script_name>`: The Grand Showcase!**\n This is where the real fun begins! This action:\n 1. If you specify a C example (e.g., `vlm_vision_example.c`), it compiles it on the fly if not already built.\n 2. Runs your chosen example script (C or Python)!\n ✨ **Requires `VLM_API_KEY`!** ✨\n ```bash\n # Example for the C script (vlm_vision_example.c):\n # Run from your GstVlmVision project root\n docker run --rm \\\n -e VLM_API_KEY=\"YOUR_ACTUAL_API_KEY\" \\\n -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix \\\n --volume $(pwd)/gst-vlm-plugin:/builder \\\n --volume $(pwd)/examples:/examples \\\n gst-vlm-vision test-examples vlm_vision_example.c\n\n # Example for the Python script (vlm_vision_example.py):\n # Run from your GstVlmVision project root\n docker run --rm \\\n -e VLM_API_KEY=\"YOUR_ACTUAL_API_KEY\" \\\n -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix \\\n --volume $(pwd)/gst-vlm-plugin:/builder \\\n --volume $(pwd)/examples:/examples \\\n gst-vlm-vision test-examples vlm_vision_example.py\n ```\n Don't forget to replace `\"YOUR_ACTUAL_API_KEY\"`! The X11 forwarding lines are for examples that pop up a video window.\n\n- **`shell`: Your Personal Command Deck!**\n Want to poke around inside the container? Need to run some custom commands or debug something? The `shell` action drops you right into an interactive command line.\n ```bash\n # Run from your GstVlmVision project root\n docker run -it --rm \\\n -e VLM_API_KEY=\"YOUR_ACTUAL_API_KEY\" `# Optional, but good to have if you plan to test` \\\n -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix \\\n --volume $(pwd)/gst-vlm-plugin:/builder \\\n --volume $(pwd)/examples:/examples \\\n gst-vlm-vision shell\n ```\n Inside the shell, your plugin source will be at `/builder` and examples at `/examples`. The main plugin won't be installed by default with this action alone, but the `entrypoint.sh` script itself is available at `/entrypoint.sh` if you want to manually trigger parts of its logic, or use this shell after running `test-examples` to inspect a fully set-up environment.\n\n> [!NOTE]\n> Using `-e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix` allows GUI elements to display on your host machine. This requires running the command `xhost +` on your host Linux machine to allow the Docker container to access your display. If you're using a different display server or setup, you might need to adjust these flags accordingly.\n\n**Important Notes for Docker Adventures:**\n- **Volume Mounts are Key:** The `--volume $(pwd)/...:/...` flags map directories from your computer into the Docker container.\n - `/builder`: Points to your `gst-vlm-plugin` directory. This is where the main plugin source code lives.\n - `/examples`: Points to your `examples` directory.\n- **API Key:** For `test-examples`, the `VLM_API_KEY` environment variable (`-e`) is crucial. The plugin won't talk to the VLM without it!\n- **GUI Display:** If your examples use `autovideosink` or any other element that creates a window, you'll need the `-e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix` lines (and sometimes `xhost +local:docker` on your host Linux machine) to see the output.\n\nWith these commands, you're all set to explore the wonders of GstVlmVision without breaking a sweat over setup!\n\n---\n\n## 🔄 Migration from GstGeminiVision\n\n| Old | New |\n|---|---|\n| Element: `geminivision` | Element: `vlmvision` |\n| `model-name` | `model` |\n| `prompt` | `user-prompt` |\n| `output-metadata` (bool) | `output-mode` (bitmask) |\n| `top-k` | Removed (not in OpenAI API) |\n| URL hardcoded to Google | Configurable `base-url` |\n\n**Before:**\n```bash\ngeminivision api-key=\"YOUR_KEY\" prompt=\"Describe this\" model-name=\"gemini-2.0-flash\"\n```\n\n**After:**\n```bash\nvlmvision base-url=https://generativelanguage.googleapis.com/v1beta/openai \\\n api-key=\"YOUR_KEY\" user-prompt=\"Describe this\" model=\"gemini-2.0-flash\" \\\n profile=gemini-openai\n```\n\n---\n\n## 🏗️ Architecture\n\n```\nvideo frame -> jpeg-encoder -> VlmRequest -> serializer -> http-client (libcurl POST)\nVlmResult <- response-parser <- HTTP response\n |\n +-> GstVlmVisionMeta (buffer metadata)\n +-> description-received (GObject signal)\n +-> vlmvision-result (GstBus message)\n```\n\nKey source modules in `gst-vlm-plugin/src/`: `gstvlmvision.c` (element), `gstvlmvisionmeta.c` (metadata), `serializer-openai-chat.c`, `serializer-template.c`, `response-parser-openai-chat.c`, `response-parser-template.c`, `template-engine.c`, `http-client.c`, `jpeg-encoder.c`, `queue-policy.c`.\n\n---\n\n## 🙌 Contributing\n\nContributions welcome -- bug fixes, features, documentation. Open an issue or submit a PR.\n\n## 📜 License\n\nMIT License -- see the LICENSE file.\n\n---\n\nHappy Hacking and may your pipelines be ever insightful! 💡\n"}}}} |