1 line
63 KiB
JSON
1 line
63 KiB
JSON
{"repo": "ceezh/silvr", "files": {"README.md": {"56c0afe10cb12cbcc7ee7bc321a1a43d7bae737c": {"t": 1785746379.7871337, "n": 6314, "h": "28a5d5b0cfa5f530", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"https://arxiv.org/pdf/2505.24869\">SiLVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/), [Yan-Bo Lin](https://genjib.github.io/), [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n Accepted by <b>TMLR</b> (01/2026) \n<br>\n\n</p>\n<p align=\"center\">\n    <a href=\"https://arxiv.org/pdf/2505.24869\">Paper</a>    | \n    <a href=\"https://sites.google.com/cs.unc.edu/silvr\">Project Page</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [01/18/2026] SiLVR is accpted to **TMLR**.\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SiLVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSiLVR offers several benefits:\n- **Simplicity**: SiLVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SiLVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SiLVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SiLVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SiLVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SiLVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SiLVR Architecture\" width=\"800\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt\n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt\n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n## Citation\nIf our work is useful to your research, please consider citing it.\n\n```BibTeX\n@article{\nzhang2026silvr,\ntitle={Si{LVR}: A Simple Language-based Video Reasoning Framework},\nauthor={Ce Zhang and Yan-Bo Lin and Ziyang Wang and Mohit Bansal and Gedas Bertasius},\njournal={Transactions on Machine Learning Research},\nissn={2835-8856},\nyear={2026},\nurl={https://openreview.net/forum?id=mQZbh9Zlbw},\nnote={}\n}\n```\n"}, "98f7e33758f30afcc2f4bfa1d999ec6211eca499": {"t": 1785763128.3455346, "n": 6123, "h": "3c43e71e6d66cc45", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"https://arxiv.org/pdf/2505.24869\">SiLVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"https://arxiv.org/pdf/2505.24869\">Paper</a>    | \n    <a href=\"https://sites.google.com/cs.unc.edu/silvr\">Project Page</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SiLVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSiLVR offers several benefits:\n- **Simplicity**: SiLVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SiLVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SiLVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SiLVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SiLVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SiLVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SiLVR Architecture\" width=\"800\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt\n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt\n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n## Citation\nIf our work is useful to your research, please consider citing it.\n\n```BibTeX\n@article{zhang2025silvr,\n title={SiLVR: A Simple Language-based Video Reasoning Framework},\n author={Zhang, Ce and Lin, Yan-Bo and Wang, Ziyang and Bansal, Mohit and Bertasius, Gedas},\n year={2025},\n journal={arXiv preprint arXiv:2505.24869},\n}\n```"}, "a70a26f6ba1277eaba88bb2c36df5bb3f90ae846": {"t": 1785763129.6008575, "n": 6238, "h": "4352d3bdfdd82211", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"https://arxiv.org/pdf/2505.24869\">SiLVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"https://arxiv.org/pdf/2505.24869\">Paper</a>    | \n    <a href=\"https://sites.google.com/cs.unc.edu/silvr\">Project Page</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SiLVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSiLVR offers several benefits:\n- **Simplicity**: SiLVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SiLVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SiLVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SiLVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SiLVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SiLVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SiLVR Architecture\" width=\"800\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt\n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt\n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n## Citation\nIf our work is useful to your research, please consider citing it.\n\n```BibTeX\n@misc{zhang2025silvrsimplelanguagebasedvideo,\n title={SiLVR: A Simple Language-based Video Reasoning Framework}, \n author={Ce Zhang and Yan-Bo Lin and Ziyang Wang and Mohit Bansal and Gedas Bertasius},\n year={2025},\n eprint={2505.24869},\n archivePrefix={arXiv},\n primaryClass={cs.CV},\n url={https://arxiv.org/abs/2505.24869}, \n}\n```"}, "88fe14bf5727256b4e20781397495fcd923a9ebc": {"t": 1785763131.8637774, "n": 5736, "h": "7f17b904703d7633", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SiLVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"https://sites.google.com/cs.unc.edu/silvr\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SiLVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSiLVR offers several benefits:\n- **Simplicity**: SiLVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SiLVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SiLVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SiLVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SiLVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SiLVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SiLVR Architecture\" width=\"800\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt\n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt\n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt\n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n## Citation\nTODO"}, "8d42aed8aa9de1a1dfb876cea38fbcdb7a7adae2": {"t": 1785763133.157163, "n": 5748, "h": "dc4ba5d115a814d3", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SiLVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"https://sites.google.com/cs.unc.edu/silvr\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SiLVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSiLVR offers several benefits:\n- **Simplicity**: SiLVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SiLVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SiLVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SiLVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SiLVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SiLVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SiLVR Architecture\" width=\"800\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt / \n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt / \n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n## Citation\nTODO"}, "cabeafd4d110645027fb9af5d0630cdd3bdcf57a": {"t": 1785763134.5008447, "n": 5748, "h": "2178735a4d212d0e", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SILVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"https://sites.google.com/cs.unc.edu/silvr\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SILVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSILVR offers several benefits:\n- **Simplicity**: SILVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SILVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SILVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SILVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SILVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SILVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SILVR Architecture\" width=\"800\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt / \n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt / \n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"800\">\n</div>\n\n## Citation\nTODO"}, "95f72a6fbb6314ac5ab6cc727038fa3c756fee12": {"t": 1785763135.7214491, "n": 5711, "h": "6144a6ab93a11ff3", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SILVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"TODO\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SILVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSILVR offers several benefits:\n- **Simplicity**: SILVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SILVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SILVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SILVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SILVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SILVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SILVR Architecture\" width=\"600\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt / \n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt / \n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n## Citation\nTODO"}, "24d9828df8641388401e7d797eb3c3c7470d2587": {"t": 1785763137.5709126, "n": 5933, "h": "522c294aecbeca4f", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SILVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"TODO\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SILVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSILVR offers several benefits:\n- **Simplicity**: SILVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SILVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SILVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SILVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SILVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SILVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SILVR Architecture\" width=\"600\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt / \n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt / \n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in [`./scripts`](https://github.com/CeeZh/SILVR/tree/main/scripts). The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Performance\n<div align=\"center\">\n<img src=\"assets/performance.png\" alt=\"Main Performance\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/performance_2.png\" alt=\"Additional Performance\" width=\"300\">\n</div>\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n## Citation\nTODO"}, "4164f4ff8980b504c392f6c4e6d8854a40100644": {"t": 1785763139.8028514, "n": 5881, "h": "271cd2fe159152c5", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SILVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"TODO\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] Code Release. \n\n\n## Introduction\nWe present **SILVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSILVR offers several benefits:\n- **Simplicity**: SILVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SILVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SILVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SILVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SILVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SILVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SILVR Architecture\" width=\"600\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt / \n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt / \n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in `./scripts`. The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n## Performance\n<div align=\"center\">\n<img src=\"assets/performance.png\" alt=\"Main Performance\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/performance_2.png\" alt=\"Additional Performance\" width=\"300\">\n</div>\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n## Citation\nTODO"}, "8e041867baea892fa31d1e9aadbfd0595143860b": {"t": 1785763142.0152023, "n": 5887, "h": "b568d73815fe374a", "c": "\n<div align=\"center\">\n\n\n<h2><a href=\"TODO\">SILVR : A Simple Language-based Video Reasoning Framework</a></h2>\n\n[Ce Zhang](https://ceezh.github.io/)\\*, [Yan-Bo Lin](https://genjib.github.io/)\\*, [Ziyang Wang](https://ziyangw2000.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/), [Gedas Bertasius](https://www.gedasbertasius.com/)\n<p align=\"center\">\n    <a href=\"TODO\">Paper</a>    | \n    <a href=\"TODO\">Website</a>   \n<br>\n\n</p>\n\n\n</div>\n\n\n\n## Updates\n- [05/28/2025] SILVR Code release. \n\n\n## Introduction\nWe present **SILVR**, a **Si**mple **L**anguage-based **V**ideo **R**easoning framework that decomposes complex video understanding into two stages.\n- In the first stage, we convert raw videos into rich language-based descriptions. Specifically, we densely sample short clips from the input videos and use a pre-trained visual captioner (e.g., NVILA) to extract captions for each clip. Additionally, we use automatic speech recognition (ASR) tools to convert speech into language descriptions. \n- In the second stage, we feed the rich language descriptions into a strong reasoning LLM (e.g. DeepSeek-R1) to solve complex video-language understanding tasks.\n\nSILVR offers several benefits:\n- **Simplicity**: SILVR does not require complex RL-based optimization or specialized modules for different tasks.\n- **Generalizability**: SILVR can be applied to a wide range of complex video-language tasks without task-specific fine-tuning.\n- **Modularity**: SILVR's modular design enables seamless use of powerful visual captioning models and strong reasoning LLMs.\n- **Flexibility**: SILVR supports plug-and-play integration of different captioning models, speech recognition models, and LLMs. An overview of our method is illustrated in Figure 2.\n- **Strong Performance**: SILVR achieves state-of-the-art results on multiple VideoQA benchmarks, including Video-MME (long), Video-MMMU (comprehension), Video-MMLU (quiz), CGBench, and EgoLife.\n\n\nWe believe the simple yet effective design of SILVR will enable the research community to build on our work and use our simple framework as a baseline to develop even more powerful video-language reasoning models.\n\n<div align=\"center\">\n<img src=\"assets/arch.png\" alt=\"SILVR Architecture\" width=\"600\">\n</div>\n\n## Performance\n<div align=\"center\">\n<img src=\"assets/performance.png\" alt=\"Main Performance\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/performance_2.png\" alt=\"Additional Performance\" width=\"300\">\n</div>\n\n\n## Installation\n```sh\nconda create --name=silvr python=3.9\nconda activate silvr\n\ngit clone .\ncd .\npip install -r requirements.txt\n```\n\n## Data Preparation\nDownload the caption and subtitle files for each dataset from here: https://drive.google.com/file/d/13L1Y1hr6aMoxarGxhN1QXY962y7ClORd/view?usp=drive_link. Unzip and move the caption and subtitle files to `./data`:\n```text\nSILVR/\n├── data/ # Directory for input data\n│ ├── videomme/\n│ | ├── subtitles/\n│ | ├── _8lBR0E_Tx8.srt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.srt / \n│ | ├── captions_1s/\n│ | ├── _8lBR0E_Tx8.txt / \n│ | ├── ... \n│ | └── ZXoaMa6jlO4.txt / \n│ | ├── captions_8s/\n│ | ├── captions_64s/\n│ | ├── captions_64s_qwen7b/\n│ | └── captions_64s_qwen72b/\n│ ├── videommmu/\n│ ├── videommlu/\n│ ├── cgbench/\n│ ├── cinepile/\n│ ├── mmvu/\n│ ├── mmworld/\n│ ├── egolife/\n│ └── hourvideo/\n├── output/ # (Empty) Directory for output files\n├── eval/ # Evaluation scripts for different benchmarks\n├── main.py\n├── dataset.py # Dataset loading and processing utilities\n├── model.py\n├── prompts.py # Prompt templates\n└── utils.py\n```\nDownload Video-MMLU annotations (https://huggingface.co/datasets/Enxin/Video-MMLU) to `data/videommlu/Video-MMLU`.\n\nDownload Video-MMLU category file (https://huggingface.co/datasets/Enxin/Video-MMLU/blob/main/video_sources.jsonl) to `data/videommlu/Video-MMLU/video_sources.jsonl`.\n\nDownload HourVideo dev set annotations (https://huggingface.co/datasets/HourVideo/HourVideo/viewer/default/dev) to `data/hourvideo/HourVideo`.\n\nDownload EgoLife annotations (https://huggingface.co/datasets/lmms-lab/EgoLife/tree/main) to `data/egolife/EgoLife`.\n\n\n## Experiments\nWe provide the output files here: https://drive.google.com/file/d/13PmRQsu71XUJ7UyMqjJBfJUBtlfMl0V5/view?usp=drive_link. \n\nFeel free to download and check the raw outputs of our methods.\n\n### Setup\nPlease prepare the following API keys:\n\n- `$API_KEY`: the target API key that you want to use. This can be DeepSeek, OpenAI, Lambda, etc.\n- `$API_URL`: the API URL which the code makes request to. By default, it is https://api.deepseek.com/v1/chat/completions pointing to DeepSeek.\n- `$HF_TOKEN`: your Hugging Face login token. This is used for retriving dataset annotations from Huggface Face.\n- `$OPENAI_API_KEY`: OpenAI API key. This is only needed for MMVU which needs GPT-4 for open-ended question evaluation.\n\n### Inference and Evaluation\nWe organized inference and evaluation scripts for all datasets in `./scripts`. The datasets include CGBench, CinePile, EgoLife, HourVideo, MMVU, MMWorld, Video-MME, Video-MMLU, Video-MMMU.\n\n### Use Local LLMs for Inference\nBy default, we use API services for LLM inference. If you want to use local LLMs, make sure to set `--single_process` to disable multi-processing.\n\n### Debug\n- `--num_examples_to_run`: how many examples to run.\n- `--single_process`: disable multi-processing.\n\n\n\n## Visualization\n<div align=\"center\">\n<img src=\"assets/q1.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q2.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q3.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n<div align=\"center\">\n<img src=\"assets/q4.png\" alt=\"visulization\" width=\"600\">\n</div>\n\n## Citation\nTODO"}}}} |