1 line
10 KiB
JSON
1 line
10 KiB
JSON
{"repo": "baicai-1145/vllm-gpt-sovits", "files": {"docs/user_guide/examples/offline_inference/mimo_audio.md": {"387b0ddcee053af5accef988c60c350d4edfc7d5": {"t": 1785595282.690667, "n": 9666, "h": "4e501f2bdf20a96b", "c": "# MiMo-Audio Offline Inference\n\nSource <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/mimo_audio>.\n\n\nThis directory contains an offline demo for running MiMo-Audio models with vLLM Omni. It builds task-specific inputs and generates WAV files or text outputs locally.\n\n## Model Overview\n\nMiMo-Audio provides multiple task variants for audio understanding and generation:\n\n- **tts_sft**: Basic text-to-speech generation from text input.\n- **tts_sft_with_instruct**: TTS generation with explicit voice style instructions.\n- **tts_sft_with_audio**: TTS generation with audio reference for voice cloning.\n- **tts_sft_with_natural_instruction**: TTS generation from natural language descriptions embedded in text.\n- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text). (note: the upstream task name uses the spelling 'trancribing', don't fix it)\n- **audio_understanding_sft**: Understand and analyze audio content with text queries.\n- **audio_understanding_sft_with_thinking**: Audio understanding with reasoning chain.\n- **spoken_dialogue_sft_multiturn**: Multi-turn spoken dialogue with audio input/output.\n- **speech2text_dialogue_sft_multiturn**: Multi-turn dialogue converting speech to text.\n- **text_dialogue_sft_multiturn**: Multi-turn text-only dialogue.\n\n## Setup\n\nPlease refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.\n\n### Environment Variables\n\nThe `MIMO_AUDIO_TOKENIZER_PATH` environment variable is mandatory due to the specialized architecture:\n\n```bash\nexport MIMO_AUDIO_TOKENIZER_PATH=\"XiaomiMiMo/MiMo-Audio-Tokenizer\"\n```\n\n## Quick Start\n\nRun a single sample for basic TTS:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type tts_sft\n```\n\nRun batch samples for basic TTS:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type tts_sft \\\n --num-prompts {batch_size}\n```\n\nWhen enabling multi-batch processing, if the total number of tokens passed to the next stage exceeds the `max_model_len` value in the `mimo_audio.yaml` configuration file, you must also synchronously update the `max_position_embeddings` value in `MiMo-Audio-7B-Instruct/config.json` to match the modified value.\n\nGenerated audio files are saved to `output_audio/` by default. `--num-prompts` also can be used to all tasks below.\n\n## Task Usage\n\n### tts_sft (Basic Text-to-Speech)\n\nGenerate speech from text input:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type tts_sft \\\n --text \"The weather is so nice today.\"\n```\n\n### tts_sft_with_instruct (TTS with Voice Instructions)\n\nGenerate speech with explicit voice style instructions:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type tts_sft_with_instruct \\\n --text \"The weather is so nice today.\" \\\n --instruct \"Speak happily in a child's voice\"\n```\n\n### tts_sft_with_audio (TTS with Audio Reference)\n\nGenerate speech using an audio reference for voice cloning:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type tts_sft_with_audio \\\n --text \"The weather is so nice today.\" \\\n --audio-path \"./spoken_dialogue_assistant_turn_1.wav\"\n```\n\n### tts_sft_with_natural_instruction (Natural Language TTS)\n\nGenerate speech from text containing natural voice descriptions:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type tts_sft_with_natural_instruction \\\n --text \"In a panting young male voice, he said: I can't run anymore, wait for me!\"\n```\n\n### audio_trancribing_sft (Speech-to-Text)\n\nTranscribe audio to text:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type audio_trancribing_sft \\\n --audio-path \"./spoken_dialogue_assistant_turn_1.wav\"\n```\n\n### audio_understanding_sft (Audio Understanding)\n\nUnderstand and analyze audio content with text queries:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type audio_understanding_sft \\\n --text \"Summarize the audio.\" \\\n --audio-path \"./spoken_dialogue_assistant_turn_1.wav\"\n```\n\n### audio_understanding_sft_with_thinking (Audio Understanding with Reasoning)\n\nAudio understanding with reasoning chain:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type audio_understanding_sft_with_thinking \\\n --text \"Summarize the audio.\" \\\n --audio-path \"./spoken_dialogue_assistant_turn_1.wav\"\n```\n\n### spoken_dialogue_sft_multiturn (Multi-turn Spoken Dialogue)\n\nMulti-turn dialogue with audio input and output:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type spoken_dialogue_sft_multiturn \\\n --audio-path \"./prompt_speech_zh_m.wav\"\n```\n\nNote: This task uses hardcoded audio files in the script. The audio files used in examples are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples\n\n### speech2text_dialogue_sft_multiturn (Speech-to-Text Dialogue)\n\nMulti-turn dialogue converting speech to text:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type speech2text_dialogue_sft_multiturn\n```\n\nNote: This task uses hardcoded audio files and message lists in the script.\n\n### text_dialogue_sft_multiturn (Text Dialogue)\n\nMulti-turn text-only dialogue:\n\n```bash\npython3 -u end2end.py \\\n --stage-configs-path vllm_omni/model_executor/stage_configs/mimo_audio.yaml \\\n --model-name XiaomiMiMo/MiMo-Audio-7B-Instruct \\\n --query-type text_dialogue_sft_multiturn\n```\n\nNote: This task uses hardcoded message lists in the script.\n\n## Troubleshooting\n\n### Audio dependencies (soundfile, librosa)\n\nThis example depends on **soundfile** (read/write WAV) and **librosa** (load audio including MP3). Install the project requirements first:\n\n```bash\npip install -r requirements/common.txt\n# or at least: pip install soundfile>=0.13.1 librosa>=0.11.0\n```\n\n- **`soundfile` / libsndfile not found** \n `soundfile` uses the C library **libsndfile**. On Linux, install the system package before pip:\n - Debian/Ubuntu: `sudo apt-get install libsndfile1`\n - For development builds: `sudo apt-get install libsndfile1-dev`\n - Then: `pip install soundfile`\n\n- **`librosa` fails to load MP3 or reports \"No backend available\"** \n Loading MP3 (e.g. in `spoken_dialogue_sft_multiturn` with `.mp3` files) uses **ffmpeg** as the backend. Install ffmpeg:\n - Debian/Ubuntu: `sudo apt-get install ffmpeg`\n - macOS: `brew install ffmpeg`\n\n- **`ImportError: No module named 'soundfile'` or `ModuleNotFoundError: ... librosa`** \n Ensure you are in the same Python environment where vLLM Omni and the example dependencies are installed, and that `requirements/common.txt` (or the packages above) are installed.\n\n### Tokenizer path\n\n- **`MIMO_AUDIO_TOKENIZER_PATH` not set or model fails to find tokenizer** \n Export the tokenizer path before running:\n ```bash\n export MIMO_AUDIO_TOKENIZER_PATH=\"XiaomiMiMo/MiMo-Audio-Tokenizer\"\n ```\n See [Environment Variables](#environment-variables) in Setup.\n\n### Other\n\n- If the model or stage config fails to load, check [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for memory and GPU settings.\n- For errors when reading/writing WAV (e.g. unsupported format), ensure input files are standard WAV/MP3 and that `soundfile` is linked to a working libsndfile (see above).\n\n## Notes\n\n- The script uses default model paths and audio files embedded in `end2end.py`. Update them if your local cache path differs.\n- Use `--output-dir` to change the output folder (default: `./output_audio`).\n- Use `--num-prompts` to generate multiple prompts in one run (default: 1).\n- Audio files used in multi-turn dialogue examples are available at: https://github.com/XiaomiMiMo/MiMo-Audio/tree/main/examples\n- The script supports various configuration options for initialization timeouts, batch timeouts, and shared memory thresholds. See `--help` for details.\n\n## Example materials\n\n??? abstract \"end2end.py\"\n ``````py\n --8<-- \"examples/offline_inference/mimo_audio/end2end.py\"\n ``````\n??? abstract \"message_base64_wav.json\"\n ``````json\n --8<-- \"examples/offline_inference/mimo_audio/message_base64_wav.json\"\n ``````\n??? abstract \"message_convert.py\"\n ``````py\n --8<-- \"examples/offline_inference/mimo_audio/message_convert.py\"\n ``````\n??? abstract \"process_speechdata.py\"\n ``````py\n --8<-- \"examples/offline_inference/mimo_audio/process_speechdata.py\"\n ``````\n"}}}} |