1 line
13 KiB
JSON
1 line
13 KiB
JSON
{"repo": "alievk/voi-server", "files": {"README.md": {"f7b6ad87fc3f10424fe7f2608844a88e4e4cd165": {"t": 1785674602.4130378, "n": 13264, "h": "cb442fecd5f2e5c4", "c": "<p align=\"center\">\n <img src=\"https://github.com/user-attachments/assets/f4f571a1-b06f-4c38-8cb6-88d6ca54d19f\" width=\"256\" height=\"256\" />\n</p>\n\nVoi is a free and open source backend for realtime voice agents. Check the JS [client](https://github.com/alievk/voi-js-client).\n\n# Table of Contents\n- [News](#news)\n- [Requirements](#requirements)\n - [Hardware](#hardware)\n - [Software](#software)\n- [Setup](#setup)\n - [Development environment](#development-environment)\n - [Caddy](#caddy)\n - [LiteLLM](#litellm)\n - [Voi server](#voi-server)\n - [Environment variables](#environment-variables)\n - [Speech-to-text model](#speech-to-text-model)\n - [Text-to-speech models](#text-to-speech-models)\n - [Custom text-to-speech models](#custom-text-to-speech-models)\n - [Agents](#agents)\n - [Run the server](#run-the-server)\n - [Access tokens](#access-tokens)\n - [Roadmap](#roadmap)\n - [Philosophy](#philosophy)\n - [Contribution](#contribution)\n - [License](#license)\n\n## News\n\n- 02/10/2025 - Voi is open source 🎉\n- 02/22/2025 - Added user images support\n- 03/07/2025 - Added call mode 📞\n\n## Requirements\n\n### Hardware\n- 9Gb+ of GPU memory. I recommend GeForce RTX 3090 or better for a single worker.\n- 8-core CPU, 32Gb RAM is enough.\n- 10Gb of disk space.\n\n### Software\n- Ubuntu 22.04 or higher.\n- Fresh Nvidia drivers (tested on 545+ driver versions).\n- Docker with Nvidia runtime support.\n- Caddy server.\n- LiteLLM server.\n\n## Setup\nVoi uses Docker Compose to run a server. It uses Docker mostly for runtime, while keeping source code, Python packages and model weights on the host file system. This was made intentionally to allow fast development. \n\nThere are two Docker environments to run Voi server, production and development. Basically they are the same, except the production config starts the Voi server automatically and uses a different port.\n\n### Development environment\n\nGet the sources.\n```bash\ngit clone https://github.com/alievk/voi-core.git\ncd voi-core\n```\n\nCopy your `id_rsa.pub` into `docker` folder to be able to ssh directly into the container.\n```bash\ncp ~/.ssh/id_rsa.pub docker/\n```\n\nMake a copy of `docker/docker-compose-dev.example.yml`.\n```bash\ncp docker/docker-compose-dev.example.yml docker/docker-compose-dev.yml\n```\nIn the `docker-compose-dev.yml`, edit the `environment`, `ports` and `volumes` sections as you need. If you need a Jupyter server, set your token in the `JUPYTER_TOKEN` variable, otherwise it won't run for safety reasons.\n\nBuild and run the development container.\n```bash\ncd docker\n./up-dev.sh\n```\n\nWhen the container is created, you will see `voice-agent-core-container-dev` in `docker ps` output, otherwise check `docker-compose logs` for errors. If there were no errors, ssh daemon and Jupyter server will be listening on the ports defined in `docker-compose-dev.yml`.\n\nConnect via ssh into the container from e.g. your laptop: \n```bash\nssh user@<host> -p <port>\n``` \nwhere `<host>` is the address of your host machine and `port` is the port specified in `docker-compose-dev.yml`. You will see a bash prompt like `user@8846788f5e9c:~$`.\n\nMy personal recommendation is to add a config to your `~/.ssh/config` file to easily connect to the container:\n```\nHost voi_docker_dev\n Hostname your_host_address\n AddKeysToAgent yes\n UseKeychain yes\n User user\n Port port_from_the_above\n```\n\nThen you do just this and get into the container:\n```bash\nssh voi_docker_dev\n```\n\nIn the container, install the Python dependencies: \n```bash\ncd voi-core\n./install.sh\n```\nThis step is intentionally not incorporated in the Dockerfile because at the active development stage you often change the requirements and don't want to rebuild the container each time. You won't need to do this each time when the container was restarted if you have mapped `.local` directory properly in the `docker-compose-dev.yml`.\n\n### Caddy\nThe Voi server uses secure web socket connection and relies on Caddy, which nicely manages SSL certificates for us. Follow [the docs](https://caddyserver.com/docs/install) to get it. \n\nOn your host machine, make sure you have proper config in the Caddyfile (usually `/etc/caddy/Caddyfile`): \n```\nyour_domain.com:8774 {\n reverse_proxy localhost:8775\n}\n```\nThis will proxy secure web socket connection from `8775` to `8774` port. \n\n### LiteLLM\nLiteLLM allows calling all LLM APIs using OpenAI format, which is neat. \n\n#### Restricted regions\nIf you run a Voi server in a country restricted by OpenAI (like Russia or China), you will need to run a remote LiteLLM server in a closest unrestricted country. You can do this for just $10/mo using AWS Lightsail. These are minimal specs you need:\n- 2 GB RAM, 2 vCPUs, 60 GB SSD\n- Ubuntu\n\nIf you use AWS Lightsail, do not forget to add a custom TCP rule for the port 4000.\n\nIf you are not in the restricted region, you can run LiteLLM server locally on your host machine.\n\n#### Setup\nFor the details of setting up LiteLLM, visit [the repo](https://github.com/BerriAI/litellm), but basically you need to follow these steps.\n\nGet the code.\n```bash\ngit clone https://github.com/BerriAI/litellm\ncd litellm\n``` \n\nAdd the master key - you can change this after setup.\n```bash\necho 'LITELLM_MASTER_KEY=\"sk-1234\"' > .env\nsource .env\n``` \n\nCreate models configuration file.\n```bash\nvim litellm_config.yaml\n``` \n\nExample configuration:\n```yaml\nmodel_list:\n - model_name: gemini-1.5-flash\n litellm_params:\n model: openai/gemini-1.5-flash\n api_key: your_googleapi_key\n api_base: https://generativelanguage.googleapis.com/v1beta/openai\n - model_name: meta-llama-3.1-70b-instruct-turbo\n litellm_params:\n model: openai/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo\n api_key: your_deepinfra_key\n api_base: https://api.deepinfra.com/v1/openai\n```\n\nThe `model` format is `{API format}/{model name}`, where `{API format}` is `openai`/`anthropic` and `{model name}` is the model name in the provider's format (`gpt-4o-mini` for OpenAI or `meta-llama/Meta-Llama-3.1-8B-Instruct` for DeepInfra). Look at [litellm_config.example.yaml](litellm_config.example.yaml) for more examples. \n\n- Start LiteLLM server.\n```bash\ndocker-compose up\n```\n\n### Voi server\nBefore running the server, we need to set the environment variables and create agents config. \n\nMy typical workflow is to run the [development environment]() and ssh to the container using Cursor (`Connect to Host` -> `voi_docker_dev`). In this way, I can edit source code and run the scripts in one place.\n\n#### Environment variables\nMake a copy of `.env.example`.\n```bash\n# Assuming you are in the Voi root\ncp .env.example .env\n```\n\n- `LITELLM_API_BASE` is the address of your LiteLLM server, like `http://111.1.1.1:4000` of `http://localhost:4000`.\n- `LITELLM_API_KEY` is `LITELLM_MASTER_KEY` from the LiteLLM's `.env` file.\n- `TOKEN_SECRET_KEY` is a secret key for generating [access tokens](#access-tokens) to the websocket endpoint. You should not reveal this key to a client.\n- `API_KEY` is the HTTPS API access key. You need to share it with a client.\n\n#### Speech-to-text model\nVoi relies on [Whisper](https://github.com/openai/whisper) for speech transcribition and adds realtime (transcribe-as-you-speak) processing on top of that. The model weights are downloaded automatically on the first launch.\n\n#### Text-to-speech models\nVoi uses xTTS-v2 model to generate speech. It gives the best tradeoff between the quality and speed. \n\nTo test your agents, you can download the pre-trained multi-speaker model from [HuggingFace](https://huggingface.co/coqui/XTTS-v2/tree/main). Download these files and put them in a directory of your choice (f.e., `models/xtts_v2`):\n- `model.pth`\n- `config.json`\n- `vocab.json`\n- `speakers_xtts.pth`\n\nThen make a copy of `tts_models.example.json` and fix the paths in `multispeaker_original` so that they point to the model files above.\n```bash\ncp tts_models.example.json tts_models.json\n```\n\n#### Custom text-to-speech models\nVoi allows changing voice tone of the agent dynamically during the conversation (like neutral or excited), but the pre-trained model coming along with xTTS doesn't allow this. I have a custom pipeline for fine-tuning text-to-speech models on audio datasets and enabling dynamic tone changing, which I'm not open sourcing today. If you need a custom model, please DM me on [X](https://x.com/alievk0).\n\n#### Agents\nAgents are defined in JSON files in the `agents` directory. The control agents are defined in `agents/control_agents.json`. To add a new agent, simply create a JSON file with agent configurations in the `agents` directory and it will be loaded when the server starts. A client can also send an agent config when opening a new connection using the `agent_config` field.\n\nAn example of agent configurations can be found in the [voi-js-client repository](https://github.com/alievk/voi-js-client/blob/main/config/agents.example.json).\n\nEach agent configuration has the following structure:\n- `llm_model`: The language model to use (must match models in `litellm_config.yaml`)\n- `control_agent` (optional): Name of an agent that filters/controls the main agent's responses\n- `voices`: Configuration for speech synthesis\n - `character`: Main voice settings\n - `model`: TTS model name from `tts_models.json`\n - `voice`: Voice identifier for the model\n - `speed` (optional): Speech speed multiplier\n - `narrator` (optional): Voice for narrative comments\n - Same settings as `character` plus:\n - `leading_silence`: Silence before narration\n - `trailing_silence`: Silence after narration\n- `system_prompt`: Array of strings defining the agent's personality and behavior. Can include special templates:\n - `{character_agent_message_format_voice_tone}`: Adds instructions for voice tone control (neutral/warm/excited/sad)\n - `{character_agent_message_format_narrator_comments}`: Adds instructions for narrator comments format (*actions in third person*)\n- `examples` (optional): List of conversation examples for few-shot learning\n- `greetings`: Initial messages configuration\n - `choices`: List of greeting messages (can include pre-cached voice files)\n - `voice_tone`: Emotional tone for greeting (must match tones in `tts_models.json`)\n\nSpecial agents like `control_agent` can have additional fields:\n- `model`: Processing type (e.g. \"pattern_matching\")\n- `denial_phrases`: Phrases to filter out\n- `giveup_after`: Number of retries before giving up\n- `giveup_response`: Fallback responses\n\n#### Run the server\nSsh into the container and run:\n```bash\npython3 ws_server.py\n```\n\nNote that the first time the client connects to an agent it may take some time to load the text-to-speech models.\n\n### Access tokens\nClients and agents communicate via the websocket. A client must receive it's personal token to access the websocket endpoint. This can be made in two ways:\n1. Through the API:\n```bash\ncurl -I -X POST \"https://your_host_address:port/integrations/your_app\" \\\n-H \"API-Key: your_api_key\"\n```\nwhere \n- `your_host_address:port` is the address of the host running Voi server and `port` is the port where the server is listening.\n- `your_app` is the name of your app, like `relationships_coach`.\n- `your_api_key` is `API_KEY` from `.env`.\n\nNote that this will generate a token which will expire after 1 day.\n\n2. Manually:\n```bash\npython3 token_generator.py your_app --expire n_days\n```\nHere you can set any number of `n_days` when your token will expire.\n\n## Roadmap\n\n- [x] Make it open source\n- [x] Support for user images\n- [ ] Incoming calls\n - [ ] Context gathering: understand user's problem\n - [ ] Function calling: add external actuators like DB inquiry\n - [x] Turn detection: detect a moment when agent can start to speak\n - [ ] Add a call-center-like voice\n- [ ] WebRTC support\n- [ ] VoIP support\n- [ ] Outcoming calls\n\n## Philosophy\nRealtime conversation with a human is a really complex task, as it requires from the agent an empathy, competence and speed. If you lack a single piece of these, your agent is useless. That's why making a good voice agent is not just stacking a bunch of APIs together. You have to develop it very carefully, making a small step, then testing, making a small step, then testing... \n\nThere are two main factors which enabled me to run this project. First, the emergence of smart, fast and cheap LLMs necessary for agents intelligence. Second, the advancement of code copilots. Though I have a deep learning background, there are lots of topics beyond my competence required to build a good voice agent.\n\nWhile open sourcing Voi, I realized many people could use it for learning software engineering. Yes, this is still actual, because this project is basically many pieces of AI-generated code carefully stitched together by a human engineer.\n\n## Contribution\nYou are welcome to open PRs with bug fixes, new features and documentation improvements.\n\nOr you can just buy me a coffee and I will convert it to code! \n[](https://buymeacoffee.com/alievk)\n\n## License\nVoi uses the MIT license, which basically means you can do anything with it, free of charge. However, the dependencies may have different licenses. Check them if you care.\n"}}}} |