1 line
127 KiB
JSON
1 line
127 KiB
JSON
{"repo": "allwefantasy/byzer-llm", "files": {"README.md": {"8d28eee121fe7d2048dfb70aaf0ff32d31d396f5": {"t": 1785674102.5487309, "n": 76348, "h": "b259e92425a1b09e", "c": "<p align=\"center\">\n <picture> \n <img alt=\"Byzer-LLM\" src=\"https://github.com/allwefantasy/byzer-llm/blob/master/docs/source/assets/logos/logo.jpg\" width=55%>\n </picture>\n</p>\n\n<h3 align=\"center\">\nEasy, fast, and cheap pretrain,finetune, serving, python api for everyone\n</h3>\n\n<p align=\"center\">\n| <a href=\"./docs/zh/\"><b>English</b></a> | <a href=\"./docs/en/\"><b>中文</b></a> |\n\n</p>\n\n---\n\n*Latest News* 🔥\n\n- [2024/04] Release Byzer-LLM 0.1.87\n...\n- [2023/12] Release Byzer-LLM 0.1.30\n\n---\n\n\nByzer-LLM is Ray based , a full lifecycle solution for LLM that includes pretrain, fintune, deployment and serving.\n\nThe unique features of Byzer-LLM are:\n\n1. Full lifecyle: pretrain and finetune,deploy and serving support\n2. Python/SQL API support\n3. Ray based, easy to scale\n\n---\n\n* [Versions](#Versions)\n* [Installation](#Installation)\n* [Quick Start](#Quick-Start)\n* [How to connect Models from outside of Ray Cluster](#how-to-connect-models-from-outside-of-ray-cluster)\n* Embedding/Rerank\n * [Embedding Model](#Embedding-Model)\n * [Embedding Rerank Model](#Embedding-Rerank-Model)\n* [Quatization](#Quatization)\n* [Supported Models](#Supported-Models)\n* Serving\n * Backend\n * [vLLM Support](#vLLM-Support)\n * [vLLM troubleshooting](#vLLM-troubleshooting)\n * [DeepSpeed Support](#DeepSpeed-Support)\n * [llama_cpp Support](#llama_cpp-Support)\n * [Byzer-LLM OpenAI-Compatible RESTful API server](#byzer-llm-openai-compatible-restful-api-server)\n* LLM && Python \n * [Function Calling](#Function-Calling)\n * [Respond with pydantic class](#Respond-with-pydantic-class)\n * [Function Implementation](#Function-Implementation)\n * [LLM-Friendly Function/DataClass](#llm-friendly-functiondataclass)\n * [Prompt Function && Prompt Class](#Prompt-Function--Prompt-Class)\n* Model Configurations \n * [Model Meta](#Model-Meta)\n * [Chat Template](#Chat-Template)\n * [LLM Default Generation Parameters](#LLM-Default-Generation-Parameters)\n * [Pin Model Worker Mapping](#Pin-Model-Worker-Mapping)\n * [Model Worker Load Balance](#Model-Worker-Load-Balance)\n* [SaaS Models](#SaaS-Models)\n * [qianwen/通义千问](#qianwen_通义千问)\n * [qianwen_emb/通义千问](#qianwen_emb_通义千问)\n * [qianwen_vl/通义千问多模态](#qianwen_vl通义千问多模态)\n * [yi_vl_plus/01万物多模态](#yi_vl_plus01万物多模态)\n * [baichuan/百川](#baichuan/百川)\n * [azure openai](#azure-openai)\n * [openai](#openai)\n * [zhipu/智谱](#zhipu/智谱)\n * [sparkdesk/星火](#sparkdesk_星火) \n * [AmazonBedrock](#AmazonBedrock)\n * [Claude](#claude)\n * [Gemini](#gemini)\n * [Kimi](#moonshot_kimi)\n * [DeepSeek](#deepseek)\n * [Volcano TTS](#volcano_tts)\n * [OpenAI TTS](#openai_tts)\n * [Azure TTS](#azure_tts)\n * [OpenAI Image Generation](#openai_image_generation)\n* [Multi Modal](#Multi-Modal)\n* [StableDiffusion](#StableDiffusion)\n* [SQL Support](#SQL-Support)\n* [Pretrain](#Pretrain)\n* [Finetune](#Finetune)\n* [Stream Chat](#Stream-Chat)\n* [Articles](#Articles)\n* [Contributing](#Contributing)\n\n---\n\n## Versions\n- 0.1.71: stream_chat_oai/async_chat_oai add delta_mode parameter\n...\n- 0.1.5: Support python wrapper for [byzer-retrieval](https://github.com/allwefantasy/byzer-retrieval)\n\n---\n\n## Installation\n\nRecommend Env:\n\n1. Conda: python==3.10.11 \n2. OS: ubuntu 22.04\n3. Cuda: 12.1.0 (Optional if you only use SaaS model)\n\n```bash\n## Make sure you python version is 3.10.11\npip install -r requirements.txt\n## Skip this step if you have no Nvidia GPU\npip install vllm==0.3.3\npip install -U byzerllm\nray start --head\n```\n\nIf your cuda version is 11.8, please check the following link to install vLLM:\n\nhttps://docs.vllm.ai/en/latest/getting_started/installation.html\n\nThe key steps are:\n\n```shell\nAs of now, vLLM’s binaries are compiled on CUDA 12.1 by default. However, you can install vLLM with CUDA 11.8 by running:\n\n# Install vLLM with CUDA 11.8.\nexport VLLM_VERSION=0.2.6\nexport PYTHON_VERSION=310\npip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl\n\n# Re-install PyTorch with CUDA 11.8.\npip uninstall torch -y\npip install torch --upgrade --index-url https://download.pytorch.org/whl/cu118\n\n# Re-install xFormers with CUDA 11.8.\npip uninstall xformers -y\npip install --upgrade xformers --index-url https://download.pytorch.org/whl/cu118\n```\n\n### Raw Machine \n\n> Only tested on Ubuntu 20.04/22.04 CentOS 8.0\n\nIf your machine is a raw machine whithout GPU Driver and Cuda installed, you can use the following script to setup the machine:\n\n```shell\ngit clone https://gitee.com/allwefantasy/byzer-llm\ncd byzer-llm/setup-machine\n```\n\nThen switch to the ROOT user, and run the following script:\n\n```shell\nROLE=master ./setup-machine.sh\n```\nThis step will create a user called `byzerllm`.\n\nThen switch to the `byzerllm` user, and run the following script:\n\n```shell\nROLE=master ./setup-machine.sh\n```\n\nThis time the script will automatically install the following software:\n\n1. \n2. Conda\n3. Nvidia Driver 535\n4. Cuda 12.1.0\n5. Ray\n6. all the python packages in requirements.txt\n7. Byzer-SQL/Byzer-Notebook\n\nIf you need to add more worker nodes to the Ray cluster, repeat the above steps on the worker nodes.\nNotice that the `ROLE` should be `worker` on the worker nodes.\n\n```shell\nROLE=worker ./setup-machine.sh\n```\n\n---\n\n## Quick Start\n\n### Command Line\n\nDeploy a model:\n\n```shell\nbyzerllm deploy --model_path /home/byzerllm/models/openbuddy-llama2-13b64k-v15 \\\n--pretrained_model_type custom/auto \\\n--gpu_gpus_per_worker 4 \\\n--num_workers 1 \\\n--model llama2_chat \n```\n\n\nThen you can chat with the model:\n\n```shell\nbyzerllm query --model llama2_chat --query \"你好\"\n```\n\nYou can undeploy the model like this:\n\n```shell\nbyzerllm undeploy --model llama2_chat\n```\n\nIf you need to specify the Ray address, use `--ray_address`.\n\nHere is another case to show you how to deploy SaaS model:\n\n```shell\nbyzerllm deploy --pretrained_model_type saas/qianwen \\\n--cpus_per_worker 0.01 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=xxxxx saas.model=qwen-max \\\n--model qianwen_short_chat \n```\n\n\n\n### Python\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\n\nllm = ByzerLLM()\n\nllm.setup_gpus_per_worker(4).setup_num_workers(1)\nllm.setup_infer_backend(InferBackend.transformers)\n\nllm.deploy(model_path=\"/home/byzerllm/models/openbuddy-llama2-13b64k-v15\",\n pretrained_model_type=\"custom/llama2\",\n udf_name=\"llama2_chat\",infer_params={})\n\n\n\nllm_client = ByzerLLM()\nllm_client.setup_template(\"llama2_chat\",\"auto\")\n\nv = llm.chat_oai(model=\"llama2_chat\",conversations=[{\n \"role\":\"user\",\n \"content\":\"hello\",\n}])\n\nprint(v[0].output)\n```\n\nThe above code will deploy a llama2 model and then use the model to infer the input text. If you use transformers as the inference backend, you should specify the `pretrained_model_type` manually since the transformers backend can not auto detect the model type.\n\nByzer-LLM also support `deploy` SaaS model with the same way. This feature provide a unified interface for both open-source model and SaaS model. The following code will deploy a Azure OpenAI model and then use the model to infer the input text.\n\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\n\nllm = ByzerLLM()\n\nllm.setup_gpus_per_worker(0).setup_num_workers(10)\nllm.setup_infer_backend(InferBackend.transformers)\n\nllm.deploy(pretrained_model_type=\"saas/azure_openai\",\n udf_name=\"azure_openai\",\n infer_params={\n \"saas.api_type\":\"azure\",\n \"saas.api_key\"=\"xxx\"\n \"saas.api_base\"=\"xxx\"\n \"saas.api_version\"=\"2023-07-01-preview\"\n \"saas.deployment_id\"=\"xxxxxx\"\n })\n\n\nllm_client = ByzerLLM()\nllm_client.setup_template(\"azure_openai\",\"auto\")\n\nv = llm.chat_oai(model=\"azure_openai\",conversations=[{\n \"role\":\"user\",\n \"content\":\"hello\",\n}])\n\nprint(v[0].output)\n```\n\nNotice that the SaaS model does not need GPU, so we set the `setup_gpus_per_worker` to 0, and you can use `setup_num_workers`\nto control max concurrency,how ever, the SaaS model has its own max concurrency limit, the `setup_num_workers` only control the max\nconcurrency accepted by the Byzer-LLM.\n\n## How to connect Models from outside of Ray Cluster\n\nThe recommended way is to start a empty Ray worker in your target machine(e.g. Your web server machine):\n\n```shell\nray start --address=\"xxxxx:6379\" --num-gpus=0 --num-cpus=0 \n```\n\nThen you can connect the Models from outside of the Ray cluster:\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\n\n## connect the ray cluster by the empty worker we started before\n## this code should be run once in your prorgram\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\n\n## new a ByzerLLM instance\n\nllm_client = ByzerLLM()\nllm_client.setup_template(\"llama2_chat\",\"auto\")\n\nv = llm.chat_oai(model=\"llama2_chat\",conversations=[{\n \"role\":\"user\",\n \"content\":\"hello\",\n}])\n\nprint(v[0].output)\n```\n\nOr \n\n\n```python\nfrom dataclasses import asdict\nimport json\nimport base64\n\nimport byzerllm\nfrom byzerllm.utils.client.types import Templates\n\nbyzerllm.connect_cluster(address=\"ray://CLUSTER_IP:10001\")\n\nmodel_name = \"zephyr_7b_chat\"\nllm = byzerllm.ByzerLLM()\nllm.setup_template(model=model_name,template=Templates.default())\nllm.setup_default_model_name(model_name)\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\":\"你好\"}])\nprint(t[0].output)\n```\n\n\n## Embedding Model\n\nThe following code is a example of deploying BGE embedding model\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\nllm = ByzerLLM()\n\nllm.setup_gpus_per_worker(0.4).setup_num_workers(2).setup_infer_backend(InferBackend.Transformers)\nllm.deploy(\n model_path=\"/home/byzerllm/models/bge-large-zh\",\n pretrained_model_type=\"custom/bge\",\n udf_name=\"emb\",\n infer_params={}\n) \n```\n\nThen you can convert any text to vector :\n\n```python\nt = llm.emb(\"emb\",LLMRequest(instruction=\"wow\"))\nt[0].output\n#output: [-0.005588463973253965,\n -0.01747054047882557,\n -0.040633779019117355,\n...\n -0.010880181565880775,\n -0.01713103987276554,\n 0.017675869166851044,\n -0.010260719805955887,\n ...]\n```\n\nThe Byzer-LLM also support SaaS embedding model. The following code will deploy a Baichuan embedding model and then use the model to infer the input text.\n\n```python\nimport os\nos.environ[\"RAY_DEDUP_LOGS\"] = \"0\" \n\nimport ray\nfrom byzerllm.utils.retrieval import ByzerRetrieval\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,LLMResponse,LLMHistoryItem,InferBackend\nfrom byzerllm.utils.client import Templates\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nllm = ByzerLLM(verbose=True)\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nchat_name = \"baichuan_emb\"\nif llm.is_model_exist(chat_name):\n llm.undeploy(udf_name=chat_name)\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/baichuan\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"\", \n \"saas.model\":\"Baichuan-Text-Embedding\"\n })\nllm.setup_default_emb_model_name(chat_name)\n\nv = llm.emb(None,LLMRequest(instruction=\"你好\"))\nprint(v.output)\n```\n\n## Embedding Rerank Model\n\nIf you need to use embedding rerank model, you can refer to the following usage.\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\nllm = ByzerLLM()\n\nllm.setup_gpus_per_worker(0.4).setup_num_workers(2).setup_infer_backend(InferBackend.Transformers)\nllm.deploy(\n model_path=\"/Users/wanghan/data/bge-reranker-base\",\n pretrained_model_type=\"custom/bge_rerank\",\n udf_name=\"emb_rerank\",\n infer_params={}\n) \nllm.setup_default_emb_model_name(\"emb_rerank\")\n```\nThen you can get a relevance score by inputting query and passage to the reranker\n```python\nsentence_pairs_01 = ['query', 'passage']\nt1 = llm.emb_rerank(sentence_pairs=sentence_pairs_01)\nprint(t1[0].output)\n#output [['query', 'passage'], 0.4474925994873047]\n\nsentence_pairs_02 = [['what is panda?', 'hi'], ['what is panda?','The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]\nt2 = llm.emb_rerank(sentence_pairs=sentence_pairs_02)\nprint(t2[0].output)\n#output [[['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.'], 6.1821160316467285], [['what is panda?', 'hi'], -8.154398918151855]]\n```\n\n## Quatization\n\nIf the backend is `InferBackend.transformers`, here is the baichuan2 example:\n\n```python\nllm.setup_gpus_per_worker(2).setup_num_workers(1).setup_infer_backend(InferBackend.Transformers)\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/baichuan2\",\n udf_name=\"baichuan2_13_chat\",\n infer_params={\"quatization\":\"4\"}\n)\n```\nThe available `quatization` values:\n\n1. 4\n2. 8\n3. true/false\n\nWhen it's set true, the int4 will be choosed.\n\nIf the bakcend is `InferBackend.VLLM`, here is the Yi example:\n\nIf you need to deploy model with Quantization, you can set the `infer_params` as the following code:\n\n```python\nllm.setup_gpus_per_worker(1).setup_num_workers(1).setup_infer_backend(InferBackend.VLLM)\nllm.deploy(\n model_path=\"/home/winubuntu/models/Yi-6B-Chat-4bits\",\n pretrained_model_type=\"custom/auto\",\n udf_name=\"chat\",\n infer_params={\"backend.quantization\":\"AWQ\"}\n)\n```\n\nThe parameter `backend.quantization` can be GPTQ/AWQ.\n\n\n## Supported Models\n\nThe supported open-source `pretrained_model_type` are:\n\n1. custom/llama2\n2. bark\t\n3. whisper\t\n3. chatglm6b\n4. custom/chatglm2\n5. moss\n6. custom/alpha_moss\n7. dolly\n8. falcon\n9. llama\n10. custom/starcode\n11. custom/visualglm\n12. custom/m3e\n13. custom/baichuan\n14. custom/bge\n15. custom/qwen_vl_chat\n16. custom/stable_diffusion\n17. custom/zephyr\n\nThe supported SaaS `pretrained_model_type` are:\n\n1. saas/chatglm\tChatglm130B\n2. saas/sparkdesk\t星火大模型\n3. saas/baichuan\t百川大模型\n4. saas/zhipu\t智谱大模型\n5. saas/minimax\tMiniMax 大模型\n6. saas/qianfan\t文心一言\n7. saas/azure_openai\t\n8. saas/openai\n\nNotice that the derived models from llama/llama2/startcode are also supported. For example, you can use `llama` to load vicuna model.\n\n## vLLM Support\n\nThe Byzer-llm also support vLLM as the inference backend. The following code will deploy a vLLM model and then use the model to infer the input text.\n\n```python\nimport ray\nfrom byzerllm.utils.retrieval import ByzerRetrieval\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\nllm = ByzerLLM()\n\nllm.setup_gpus_per_worker(2)\nllm.setup_num_workers(1)\nllm.setup_infer_backend(InferBackend.VLLM)\n\nllm.deploy(\n model_path=\"/home/byzerllm/models/openbuddy-zephyr-7b-v14.1\",\n pretrained_model_type=\"custom/auto\",\n udf_name=\"zephyr_chat\"\",\n infer_params={}\n)\n\nv = llm.chat_oai(model=\"zephyr_chat\",conversations=[{\n \"role\":\"user\",\n \"content\":\"hello\",\n}])\nprint(v[0].output)\n```\n\nThere are some tiny differences between the vLLM and the transformers backend. \n\n1. The `pretrained_model_type` is fixed to `custom/auto` for vLLM, since the vLLM will auto detect the model type.\n2. Use `setup_infer_backend` to specify `InferBackend.VLLM` as the inference backend.\n\n### vLLM troubleshooting\nIf you use the version of vLLM > 0.2.7 and meets the following error:\n\n```shell\n Error Type: TASK_EXECUTION_EXCEPTION\n\n self._call_impl(*args, **kwargs)\n File \"/home/byzerllm/miniconda3/envs/byzerllm-dev2/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1527, in _call_impl\n return forward_call(*args, **kwargs)\n File \"/home/byzerllm/miniconda3/envs/byzerllm-dev2/lib/python3.10/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py\", line 96, in forward\n input_mask = ((input_ < self.vocab_start_index) |\nTypeError: '<' not supported between instances of 'TensorMetadata' and 'int'\n```\n\nYou can open the Ray dashboard and find the RayVLLMWorker Actor, go into and check the tasks tab and click the error task to get the error message above.\n\nThe solution is to modify the `$CONDA_ENV/site-packages/vllm/model_executor/parallel_utils/communication_op.py` in the vLLM pip package:\n\nReplace the following code:\n\n```python\nTensorMetadata = namedtuple(\"TensorMetadata\", [\"dtype\", \"size\"])\n```\n\nto \n\n```python\nclass TensorMetadata:\n def __init__(self, dtype, size):\n self.dtype = dtype\n self.size = size\n```\n\n\n\n \n\n### Stream Chat\n\nIf the model deployed with the backend vLLM, then it also support `stream chat`：\nthe `stream_chat_oai` will return a generator, you can use the generator to get the output text.\n\n```python\n\nllm.setup_default_model_name(chat_model_name) \n\nt = llm.stream_chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\":\"Hello, how are you?\"\n}])\n\nfor line in t:\n print(line+\"\\n\")\n```\n\n## DeepSpeed Support\n\nThe Byzer-llm also support DeepSpeed as the inference backend. The following code will deploy a DeepSpeed model and then use the model to infer the input text.\n\n```python\nimport ray\nfrom byzerllm.utils.retrieval import ByzerRetrieval\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,InferBackend\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\nllm = ByzerLLM()\n\nllm.setup_gpus_per_worker(4)\nllm.setup_num_workers(1)\nllm.setup_infer_backend(InferBackend.DeepSpeed)\n\nllm.deploy(\n model_path=\"/home/byzerllm/models/openbuddy-llama-13b-v5-fp16\",\n pretrained_model_type=\"custom/auto\",\n udf_name=\"llama_chat\"\",\n infer_params={}\n)\n\nllm.chat(\"llama_chat\",LLMRequest(instruction=\"hello world\"))[0].output\n```\n\nThe code above is totally the same as the code for vLLM, except that the `InferBackend` is `InferBackend.DeepSpeed`.\n\n## llama_cpp Support\n\n```bash\nbyzerllm deploy --pretrained_model_type custom/auto \\\n--infer_backend llama_cpp \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--worker_concurrency 10 \\\n--infer_params verbose=true \\\n--model_path /Users/allwefantasy/Downloads/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf \\\n--model llama_3_chat\n```\n\nYou can set \n\n1. `num_gpu_layers=-1` in `--infer_params` \n2. gpu_gpus_per_worker=x\n\nto put all model to the GPU or `num_gpu_layers=0` and gpu_gpus_per_worker=0 to disable it.\n\nThe num_gpu_layers also accept a positive number to specify the number of layers to use GPU.\n\n## Byzer-LLM OpenAI-Compatible RESTful API server\n\nYou can use the following code to start a ByzerLLm OpenAI-Compatible RESTful API server:\n\n```shell\nbyzerllm serve --ray_address auto --port 8000\n```\n\nBy default, the server will listen on port 8000, you can use the following code to test the API:\n\n```python\nfrom openai import OpenAI\nclient = OpenAI(\n base_url=\"http://127.0.0.1:8000/v1\",\n api_key=\"xxxx\"\n)\n\nchat_completion = client.chat.completions.create( \n model=\"wenxin_chat\",\n messages=[{\"role\": \"user\", \"content\": \"写一个排序算法\"}],\n stream=False\n)\n\nprint(chat_completion.choices[0].message.content)\n```\n\nStream chat:\n\n```python\n\nfrom openai import OpenAI\nclient = OpenAI(\n base_url=\"http://127.0.0.1:8000/v1\",\n api_key=\"simple\"\n)\n\nchat_completion = client.chat.completions.create( \n model=\"wenxin_chat\",\n messages=[{\"role\": \"user\", \"content\": \"写一个排序算法\"}],\n stream=True\n)\n\nfor chunk in chat_completion: \n print(chunk.choices[0].delta.content or \"\", end=\"\")\n```\n\n## Function Calling\n\nHere is a simple example for function calling based on QWen 72B\n\nDeploy Model:\n\n\n```python\nimport ray\nray.init(address=\"auto\",namespace=\"default\") \nllm = ByzerLLM()\n\nmodel_location=\"/home/byzerllm/models/Qwen-72B-Chat\"\n\nllm.setup_gpus_per_worker(8).setup_num_workers(1).setup_infer_backend(InferBackend.VLLM)\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/auto\",\n udf_name=chat_model_name,\n infer_params={}\n)\n\nllm.setup_default_model_name(\"chat\")\n# from 0.1.24 \n# llm.setup_auto(\"chat\")\nmeta = llm.get_meta()\nllm.setup_max_model_length(\"chat\",meta.get(\"max_model_len\",32000))\nlm.setup_template(\"chat\",Templates.qwen()) \n```\n\nTry to create some Python functions:\n\n```python\n\nfrom typing import List,Dict,Any,Annotated\nimport pydantic \nimport datetime\nfrom dateutil.relativedelta import relativedelta\n\ndef compute_date_range(count:Annotated[int,\"时间跨度，数值类型\"],\n unit:Annotated[str,\"时间单位，字符串类型\",{\"enum\":[\"day\",\"week\",\"month\",\"year\"]}])->List[str]:\n '''\n 计算日期范围\n\n Args:\n count: 时间跨度，数值类型\n unit: 时间单位，字符串类型，可选值为 day,week,month,year\n ''' \n now = datetime.datetime.now()\n now_str = now.strftime(\"%Y-%m-%d %H:%M:%S\")\n if unit == \"day\":\n return [(now - relativedelta(days=count)).strftime(\"%Y-%m-%d %H:%M:%S\"),now_str]\n elif unit == \"week\":\n return [(now - relativedelta(weeks=count)).strftime(\"%Y-%m-%d %H:%M:%S\"),now_str]\n elif unit == \"month\":\n return [(now - relativedelta(months=count)).strftime(\"%Y-%m-%d %H:%M:%S\"),now_str]\n elif unit == \"year\":\n return [(now - relativedelta(years=count)).strftime(\"%Y-%m-%d %H:%M:%S\"),now_str]\n return [\"\",\"\"]\n\ndef compute_now()->str:\n '''\n 计算当前时间\n '''\n return datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\")\n```\n\nHere we provide two functions:\n\n1. compute_date_range: compute the date range based on the count and unit\n2. compute_now: get the current date\n\nWe will use the model to call these tools according to the user's question.\n\n```python\nt = llm.chat_oai([{\n \"content\":'''计算当前时间''',\n \"role\":\"user\" \n}],tools=[compute_date_range,compute_now],execute_tool=True)\n\nt[0].values\n\n## output: ['2023-12-18 17:30:49']\n```\n\n```python\nt = llm.chat_oai([{\n \"content\":'''最近三个月趋势''',\n \"role\":\"user\" \n}],tools=[compute_date_range,compute_now],execute_tool=True)\n\nt[0].values\n\n## output: [['2023-09-18 17:31:21', '2023-12-18 17:31:21']]\n```\n\n```python\nt = llm.chat_oai([{\n \"content\":'''最近三天''',\n \"role\":\"user\" \n}],tools=[compute_date_range,compute_now],execute_tool=True)\n\nt[0].values\n\n## output: [['2023-12-15 17:23:38', '2023-12-18 17:23:38']]\n```\n\n```python\nt = llm.chat_oai([{\n \"content\":'''你吃饭了么？''',\n \"role\":\"user\" \n}],tools=[compute_date_range,compute_now],execute_tool=True)\n\nif t[0].values:\n print(t[0].values[0])\nelse:\n print(t[0].response.output) \n\n## output: '您好，我是一个人工智能语言模型，暂时无法吃饭。'\n```\n\nYou can check the default prompt template function in `from byzerllm.utils import function_calling_format`.\nIf the model is not work well with the default function, you can setup your custom function:\n\n```python\ndef custom_function_calling_format(prompt:str,tools:List[Callable],tool_choice:Callable)->str:\n.....\n\n\nllm.setup_function_calling_format_func(\"chat\",custom_function_calling_format)\n```\n\n## Respond with pydantic class\n\nWhen you chat with LLM, you can specify the reponse class, \n\n```python\nimport pydantic \n\nclass Story(pydantic.BaseModel):\n '''\n 故事\n '''\n\n title: str = pydantic.Field(description=\"故事的标题\")\n body: str = pydantic.Field(description=\"故事主体\")\n\n\n\nt = llm.chat_oai([\n{\n \"content\":f'''请给我讲个故事，分成两个部分，一个标题，一个故事主体''',\n \"role\":\"user\"\n},\n],response_class=Story)\n\nt[0].value\n\n## output: Story(title='勇敢的小兔子', body='在一个美丽的森林里，住着一只可爱的小兔子。小兔子非常勇敢，有一天，森林里的动物们都被大灰狼吓坏了。只有小兔子站出来，用智慧和勇气打败了大灰狼，保护了所有的动物。从此，小兔子成为了森林里的英雄。')\n```\n\nThe above code will ask the LLM to generate the Story class directly. However, sometimes we hope the LLM \ngenerate text first, then extract the structure from the text, you can set `response_after_chat=True` to \nenable this behavior. However, this will bring some performance penalty(additional inference).\n\n```python\nt = llm.chat_oai([\n{\n \"content\":f'''请给我讲个故事，分成两个部分，一个标题，一个故事主体''',\n \"role\":\"user\"\n},\n],response_class=Story,response_after_chat=True)\n\nt[0].value\n## output: Story(title='月光下的守护者', body='在一个遥远的古老村庄里，住着一位名叫阿明的年轻人。阿明是个孤儿，从小在村里长大，以种田为生。他善良、勤劳，深受村民们喜爱。\\n\\n村子里有个传说，每当满月时分，月亮女神会在村子后山的古树下出现，赐福给那些善良的人们。然而，只有最纯洁的心才能看到她。因此，每年的这个时候，阿明都会独自一人前往后山，希望能得到女神的祝福。\\n\\n这一年，村子遭受了严重的旱灾，庄稼枯黄，人们生活困苦。阿明决定向月亮女神祈求降雨，拯救村子。他在月光下虔诚地祈祷，希望女神能听到他的呼唤。\\n\\n就在这个时刻，月亮女神出现了。她被阿明的善良和执着所感动，答应了他的请求。第二天早晨，天空乌云密布，大雨倾盆而下，久旱的土地得到了滋润，庄稼重新焕发生机。\\n\\n从此以后，每年的满月之夜，阿明都会去后山等待月亮女神的出现，他成为了村民心中的守护者，用他的善良和执着，守护着整个村庄。而他也终于明白，真正的守护者，并非需要超凡的力量，只需要一颗充满爱与善良的心。')\n```\n\nYou can check the default prompt template function in `from byzerllm.utils import response_class_format,response_class_format_after_chat`.\nIf the model is not work well with the default function, you can setup your custom function:\n\n```python\ndef custom_response_class_format(prompt:str,cls:pydantic.BaseModel)->str:\n.....\n\n\nllm.setup_response_class_format_func(\"chat\",custom_response_class_format)\n```\n\n## Function Implementation\n\nThe Byzer-llm also support function implementation. You can define a empty function, and combine the doc in the function/the user's quesion to guide the LLM to implement the function. \n\nHere is a simple example:\n\n```python\nfrom byzerllm.utils.client import code_utils,message_utils\nfrom typing import List,Union,Optional\nimport pydantic\n\nclass Time(pydantic.BaseModel):\n time: str = pydantic.Field(...,description=\"时间，时间格式为 yyyy-MM-dd\")\n\n\n@llm.impl()\ndef calculate_current_time()->Time:\n '''\n 计算当前时间\n '''\n pass \n\n\ncalculate_current_time()\n#output: Time(time='2024-01-28')\n```\n\nBy default, the function implementation will be cached, \n\n```python\nstart = time.monotonic()\ncalculate_current_time()\nprint(f\"first time cost: {time.monotonic()-start}\")\n\nstart = time.monotonic()\ncalculate_current_time()\nprint(f\"second time cost: {time.monotonic()-start}\")\n\n# output:\n# first time cost: 6.067266260739416\n# second time cost: 4.347506910562515e-05\n```\n\nyou can use `llm.clear_impl_cache()` to clear the cache. \n\nHere is the example for function implementation with parameters:\n\n```python\nfrom byzerllm.utils.client import code_utils,message_utils\nfrom typing import List,Union,Optional,Annotated\nimport pydantic\nfrom datetime import datetime\n\nclass Time(pydantic.BaseModel):\n time: str = pydantic.Field(...,description=\"时间，时间格式为 yyyy-MM-dd\")\n\n\n@llm.impl()\ndef add_one_day(current_day:Annotated[datetime,\"当前日期，类型是datatime.datetime\"])->Time:\n '''\n 给传入的日期加一天，得到明天的时间\n '''\n pass \n\n\nadd_one_day(datetime.now())\n# output:Time(time='2024-01-29')\n```\n\nWith instruction:\n\n```python\nfrom byzerllm.utils.client import code_utils,message_utils\nfrom typing import List,Union,Optional\nimport pydantic\n\nclass TimeRange(pydantic.BaseModel):\n '''\n 时间区间\n 格式需要如下： yyyy-MM-dd\n ''' \n \n start: str = pydantic.Field(...,description=\"开始时间.时间格式为 yyyy-MM-dd\")\n end: str = pydantic.Field(...,description=\"截止时间.时间格式为 yyyy-MM-dd\")\n\n@llm.impl(instruction=\"去年三月到七月\")\ndef calculate_time_range()->TimeRange:\n '''\n 计算时间区间，时间格式为 yyyy-MM-dd. \n '''\n pass \n\ncalculate_time_range()\n# output: TimeRange(start='2023-03-01', end='2023-07-31')\n```\n\nIf you want to replace instruction with the user's question, you can use the following code:\n\n```python\nfrom byzerllm.utils.client import code_utils,message_utils\nfrom typing import List,Union,Optional\nimport pydantic\n\nclass TimeRange(pydantic.BaseModel):\n '''\n 时间区间\n 格式需要如下： yyyy-MM-dd\n ''' \n \n start: str = pydantic.Field(...,description=\"开始时间.时间格式为 yyyy-MM-dd\")\n end: str = pydantic.Field(...,description=\"截止时间.时间格式为 yyyy-MM-dd\")\n\ndef calculate_time_range()->TimeRange:\n '''\n 计算时间区间，时间格式为 yyyy-MM-dd. \n '''\n pass \n\n\nllm.impl(instruction=\"去年三月到七月\")(calculate_time_range)()\n```\n\nYou can use `verbose=True` to get more information about the function implementation:\n\n```python\n@llm.impl()\ndef add_one_day(current_day:Annotated[datetime,\"当前日期，类型是datatime.datetime\"])->Time:\n '''\n 给传入的日期加一天，得到明天的时间\n '''\n pass \n```\n\nYou can also use the basic chat_oai function to implement the function:\n\n\n```python\nclass TimeRange(pydantic.BaseModel):\n '''\n 时间区间\n 格式需要如下： yyyy-MM-dd\n ''' \n \n start: str = pydantic.Field(...,description=\"开始时间.时间格式为 yyyy-MM-dd\")\n end: str = pydantic.Field(...,description=\"截止时间.时间格式为 yyyy-MM-dd\")\n\n\ndef calculate_time_range():\n '''\n 计算时间区间，时间格式为 yyyy-MM-dd. \n '''\n pass \n \nt = llm.chat_oai([{\n \"content\":\"去年三月到七月\",\n \"role\":\"user\" \n}],impl_func=calculate_time_range,response_class=TimeRange,execute_impl_func=True)\n```\n\nThe above code , we define a function called `calculate_time_range`, and the function is empty, then we discribe the function in the doc string, and define the response class `TimeRange`, to make sure the return value is a `TimeRange` instance. Since the function should be used to resolve the user's question, so the implementation of the function should be related to the user's question. Instead try to implement a common use function, we can just implement a function which can only resolve the user's current question.\n\nAfter the execution, you can get the output like this:\n\n```python\nt[0].value\n# start='2023-03-01' end='2023-07-31'\n```\n\nIf the value is None or not correct, you can get the error message:\n\n```python\nt[0].metadata.get(\"resason\",\"\")\n```\n\nIf your function has parameters, you can pass the parameters to the function by `impl_func_params`:\n\n```python\nt = llm.chat_oai([{\n \"content\":\"xxxxx\",\n \"role\":\"user\" \n}],\nimpl_func=calculate_time_range,\nimpl_func_params={},\nresponse_class=TimeRange,execute_impl_func=True)\n```\n\nIf you want to replace the default prompt template function, here is a example:\n\n```python\nimport pydantic\nfrom typing import List,Optional,Union,Callable\nfrom byzerllm.utils import serialize_function_to_json\n\ndef function_impl_format2(prompt:str,func:Optional[Union[Callable,str]],\n cls:Union[pydantic.BaseModel,str])->str:\n \n tool_choice_ser = serialize_function_to_json(func) \n _cls = \"\"\n if isinstance(cls, str):\n _cls = cls\n else:\n _cls = cls.schema_json(ensure_ascii=False)\n \n msg = f''''生成一个python函数，给出详细的思考逻辑，对最后生成的函数不要进行示例说明。\n\n生成的函数的名字以及参数需要满足如下约束：\n\n\\```json\n{tool_choice_ser}\n\\```\n\n生成的函数的返回值必须是 Json 格式。\n\n下面是使用 OpenAPI 3.1. 规范描述了你需如何进行json格式的生成。\n\n\\```json\n{_cls}\n\\```\n\n根据用的户问题,{func.__doc__}。用户的问题是：{prompt}\n\n请你实现这个函数。\n''' \n \n return msg\n\nllm.setup_impl_func_format_func(chat_model_name,function_impl_format2)\n```\n\nThe default prompt template function is `function_impl_format`, you can check the source code in `from byzerllm.utils import function_impl_format`.\n\n\n## LLM-Friendly Function/DataClass\n\nIf you want to improve the performance of Function Calling or Response Class, you should make your Function(Tool) and Data Class is LLM-Friendly. \n\nLet's take a look at the following python code:\n\n```python\ndef compute_date_range(count:int, unit:str)->List[str]: \n now = datetime.datetime.now()\n ....\n```\n\nThis code is not LLM-Friendly Function since it's difficult to know the usage of this funciton and \nwhat's the meaning of the input parameters.\n\nThe LLM just like human, it's hard to let the LLM know when or how to invoke this function. Especially the parameter `unit`\nactually is enum value but the LLM no way to get this message.\n\nSo, in order to make the LLM knows more about this function in Byzer-LLM, you should \nfollow some requirments:\n\n1. Adding pythonic function comment \n2. Use annotated to provide type and comment for every parameter, if the parameter is a enum, then provide enum values.\n\nHere is the LLM-Friendly fuction definision.\n\n```python\ndef compute_date_range(count:Annotated[int,\"时间跨度，数值类型\"],\n unit:Annotated[str,\"时间单位，字符串类型\",{\"enum\":[\"day\",\"week\",\"month\",\"year\"]}])->List[str]:\n '''\n 计算日期范围\n\n Args:\n count: 时间跨度，数值类型\n unit: 时间单位，字符串类型，可选值为 day,week,month,year\n ''' \n now = datetime.datetime.now()\n ....\n```\n\nIf the LLM make something wrong to your function (e.g. provide the bad parameters), try to optimize the function comment \nand the parameter Annotated comment.\n\n## Model Meta\n\nThe Byzer-llm also support get the model meta information. The following code will get the meta information of model instance called `chat`:\n\n```python\n\n```python\nllm.get_meta(model=\"chat\")\n\n#output:\n# {'model_deploy_type': 'proprietary',\n# 'backend': 'ray/vllm',\n# 'max_model_len': 32768,\n# 'architectures': ['QWenLMHeadModel']}\n```\n\n## Chat Template\n\nThe different models have different chat templates, the Byzer-LLM have provide some chat templates for the models. You can use the following code to setup the chat template:\n\n```python\nfrom byzerllm.utils.client import Templates\nllm.setup_template(\"chat\",Templates.qwen()) \n```\n\nHowever, we also support the `tokeninzer.apply_chat_template`, you can use the following code to apply the chat template:\n\n```python\nllm.setup_template(\"chat\",\"auto\") \n```\n\nIf the model is not work well with the `tokeninzer.apply_chat_template`, this function will raise an exception. In this case, you can use the `llm.setup_template` to setup the chat template manually.\n\nYou can also use the `llm.get_meta` to check if the model support the `apply_chat_template`:\n\n```python\nllm.get_meta(model=\"chat\")\n```\n\nThe output:\n\n```json\n{'model_deploy_type': 'proprietary',\n 'backend': 'ray/vllm',\n 'support_stream': True,\n 'support_chat_template': True,\n 'max_model_len': 4096,\n 'architectures': ['LlamaForCausalLM']}\n```\n\nNotice that this feature will cause additional RPC call, so it will bring some performance penalty.\n\n## Prompt Function && Prompt Class\n\nPrompt Function is a function implemented by text instead of code. The Prompt function will auto bind the function parameters to the \nfunction's doc string.\n\nPrompt Class is a class which contains the prompt function.\n\nHere is a simple example for prompt function:\n\n```python\n@llm.prompt(render=\"simple\")\ndef generate_answer(context:str,question:str)->str:\n '''\n Answer the question based on only the following context:\n {context}\n Question: {question}\n Answer:\n '''\n pass\n\ncontext='''\nByzer产品栈从底层存储到大模型管理和serving再到应用开发框架：\n1. Byzer-Retrieval, 一个支持向量+搜索的混合检索数据库。\n2. Byzer-LLM, 可以衔接SaaS/开源模型，能部署，可以统一入口。\n3. Byzer-Agent ,一套分布式 Agent 框架，做为应用开发框架\n4. ByzerPerf, 一套性能吞吐评测框架\n5. ByzerEvaluation, 一套大模型效果评测框架 （未开源）\n6. Byzer-SQL， 一套全SQL方言，支持ETL，数据分析等工作，并且支持大模型的管理和调用（model as UDF）,方便数据预处理。\n'''\nprint(generate_answer(context=context,question=\"Byzer SQL是什么？\"))\n## output: Byzer SQL 是一个全SQL方言，支持ETL、数据分析等工作，并且支持大模型的管理和调用（model as UDF），方便数据预处理。\n```\n\nNow we try to convert the prompt function to prompt class:\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM\nimport byzerllm\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nclass RAG():\n def __init__(self): \n self.llm = ByzerLLM()\n self.llm.setup_template(model=\"sparkdesk_chat\",template=\"auto\")\n self.llm.setup_default_model_name(\"sparkdesk_chat\") \n \n @byzerllm.prompt(lambda self: self.llm,render=\"simple\")\n def generate_answer(self,context:str,question:str)->str:\n '''\n Answer the question based on only the following context:\n {context}\n Question: {question}\n Answer:\n '''\n pass\n\nt = RAG()\nprint(t.generate_answer(context=context,question=\"Byzer SQL是什么？\")) \n```\n\nThe first parameter of the `byzerllm.prompt` is the `llm` instance, the following type can be accepted:\n\n1. lambda function: the lambda function will accept the `self` as the parameter, and then you can access the llm instance by self\n2. string: the string is the model name, we will create the llm instance by the model name automatically.\n3. llm: the llm instance.\n\nNotice that the llm instance we mentioned above is the `ByzerLLM` instance.\n\nIf you only want to get the prompt which is bind the parameters instead of execute the prompt, you can use the following code:\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM\nimport byzerllm\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nclass RAG():\n def __init__(self): \n self.llm = ByzerLLM()\n self.llm.setup_template(model=\"sparkdesk_chat\",template=\"auto\")\n self.llm.setup_default_model_name(\"sparkdesk_chat\") \n \n @byzerllm.prompt(render=\"simple\")\n def generate_answer(self,context:str,question:str)->str:\n '''\n Answer the question based on only the following context:\n {context}\n Question: {question}\n Answer:\n '''\n pass\n\nt = RAG()\nprint(t.generate_answer(context=context,question=\"Byzer SQL是什么？\"))\n```\n\nThe difference between the two code is that the `byzerllm.prompt whether have delived the `llm` instance to the prompt function.\n\nHere is the output:\n\n```text\n\nAnswer the question based on only the following context:\n\nByzer产品栈从底层存储到大模型管理和serving再到应用开发框架：\n1. Byzer-Retrieval, 一个支持向量+搜索的混合检索数据库。\n2. Byzer-LLM, 可以衔接SaaS/开源模型，能部署，可以统一入口。\n3. Byzer-Agent ,一套分布式 Agent 框架，做为应用开发框架\n4. ByzerPerf, 一套性能吞吐评测框架\n5. ByzerEvaluation, 一套大模型效果评测框架 （未开源）\n6. Byzer-SQL， 一套全SQL方言，支持ETL，数据分析等工作，并且支持大模型的管理和调用（model as UDF）,方便数据预处理。\n\nQuestion: Byzer SQL是什么？\nAnswer:\n```\n\nPrompt function support two kinds of return value:\n\n1. str\n2. pydanitc.BaseModel\n\nHere is the example for pydanitc.BaseModel:\n\n```python\n\nimport ray\nimport functools\nimport inspect\nimport byzerllm\nimport pydantic\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nclass ByzerProductDesc(pydantic.BaseModel):\n byzer_retrieval: str\n byzer_llm: str\n byzer_agent: str\n byzer_perf: str\n byzer_evaluation: str\n byzer_sql: str\n\nclass RAG():\n def __init__(self): \n self.llm = ByzerLLM()\n self.llm.setup_template(model=\"sparkdesk_chat\",template=\"auto\")\n self.llm.setup_default_model_name(\"sparkdesk_chat\") \n \n @byzerllm.prompt(lambda self: self.llm,render=\"simple\")\n def generate_answer(self,context:str,question:str)->ByzerProductDesc:\n '''\n Answer the question based on only the following context:\n {context}\n Question: {question}\n Answer:\n '''\n pass\n\nt = RAG()\n\nbyzer_product = t.generate_answer(context=context,question=\"Byzer 产品列表\")\nprint(byzer_product.byzer_sql)\n## output: 一套全SQL方言，支持ETL，数据分析等工作，并且支持大模型的\n```\n\nAs we know, the prompt function is a function implemented by text instead of code, the lucky thing is that we can `program`\nin the text, we introduce the jinjia2 to support this feature. Here is the example:\n\n```python\nimport ray\nimport functools\nimport inspect\nimport byzerllm\nimport pydantic\nfrom byzerllm.utils.client import ByzerLLM\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\ndata = {\n 'name': 'Jane Doe',\n 'task_count': 3,\n 'tasks': [\n {'name': 'Submit report', 'due_date': '2024-03-10'},\n {'name': 'Finish project', 'due_date': '2024-03-15'},\n {'name': 'Reply to emails', 'due_date': '2024-03-08'}\n ]\n}\n\n\nclass RAG():\n def __init__(self): \n self.llm = ByzerLLM()\n self.llm.setup_template(model=\"sparkdesk_chat\",template=\"auto\")\n self.llm.setup_default_model_name(\"sparkdesk_chat\") \n \n @byzerllm.prompt(lambda self:self.llm,render=\"jinja2\")\n def generate_answer(self,name,task_count,tasks)->str:\n '''\n Hello {{ name }},\n\n This is a reminder that you have {{ task_count }} pending tasks:\n {% for task in tasks %}\n - Task: {{ task.name }} | Due: {{ task.due_date }}\n {% endfor %}\n\n Best regards,\n Your Reminder System\n '''\n pass\n\nt = RAG()\n\nresponse = t.generate_answer(**data)\nprint(response)\n\n## output:Hello! Is there anything else I can assist you with?\n```\n\nIf you want to handle the parameters in the prompt function using Python code instead of Jinjia2, you can use the following code:\n\n```python\nimport byzerllm\n\nbyzerllm.connect_cluster()\n\ndata = {\n 'name': 'Jane Doe',\n 'task_count': 3,\n 'tasks': [\n {'name': 'Submit report', 'due_date': '2024-03-10'},\n {'name': 'Finish project', 'due_date': '2024-03-15'},\n {'name': 'Reply to emails', 'due_date': '2024-03-08'}\n ]\n}\n\n\nclass RAG():\n def __init__(self): \n self.llm = byzerllm.ByzerLLM()\n self.llm.setup_template(model=\"sparkdesk_chat\",template=\"auto\")\n self.llm.setup_default_model_name(\"sparkdesk_chat\") \n \n @byzerllm.prompt(lambda self:self.llm,render=\"jinja2\")\n def generate_answer(self,name,task_count,tasks)->str:\n '''\n Hello {{ name }},\n\n This is a reminder that you have {{ task_count }} pending tasks:\n \n {{ tasks }}\n\n Best regards,\n Your Reminder System\n '''\n \n tasks_str = \"\\n\".join([f\"- Task: {task['name']} | Due: { task['due_date'] }\" for task in tasks])\n return {\"tasks\": tasks_str}\n\nt = RAG()\n\nresponse = t.generate_answer(**data)\nprint(response)\n```\n\nMake sure the return value is a dict, and the key is the parameter name in the prompt function. \nThe return value will override the parameter value and then use the new value to render the prompt function.\n\n\n\n## LLM Default Generation Parameters\n\nThe Byzer-llm also support setup the default generation parameters for the model. The following code will setup the default generation parameters for the model instance called `chat`:\n\n```python\nllm.setup_extra_generation_params(\"chat\",{\n \"generation.stop_token_ids\":[7]\n})\n```\n\nIn this case, the `generation.stop_token_ids` will be set to `[7]` for the model instance `chat`. Every time you call the `chat` model, the `generation.stop_token_ids` will be set to `[7]` automatically.\n\n## Pin Model Worker Mapping\n\nByzer-LLM have multi-type requests to the model instance.\n\n1. embedding\n2. tokenizer\n3. apply_chat_template\n4. meta\n5. complete/chat\n\nAnd in Byzer-LLM you can start one model instance which have multi workers,e.g.\n\n```python\nllm.setup_gpus_per_worker(2).setup_num_workers(4).setup_infer_backend(InferBackend.VLLM)\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/auto\",\n udf_name=\"chat\",\n infer_params={\"backend.gpu_memory_utilization\":0.8,\n \"backend.enforce_eager\":False,\n \"backend.trust_remote_code\":True,\n \"backend.max_model_len\":1024*4,\n \"backend.quantization\":\"gptq\",\n }\n)\n```\n\nHere you deploy one model instance with 4 workers, and each worker have 2 gpus, each worker is a vLLM instance.\nThe ByzerLLM will auto route the request to the worker, and the worker is selected by the LRU algorithm.\n\nIf you want to the request like embedding,tokenizer,apply_chat_template,meta to be processed by the same worker, you can use the following code:\n\n```python\nllm.setup_pin_model_worker_mapping({\n \"embedding\":0,\n \"tokenizer\":0,\n \"apply_chat_template\":0,\n \"meta\":0,\n } \n```\n\nthis will make sure the request like embedding,tokenizer,apply_chat_template,meta to be processed by the worker 0.\n\n## Model Worker Load Balance\n\nIn Byzer-LLM you can start one model instance which have multi workers,e.g.\n\n```python\nllm.setup_gpus_per_worker(2).setup_num_workers(4).setup_infer_backend(InferBackend.VLLM)\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/auto\",\n udf_name=\"chat\",\n infer_params={\"backend.gpu_memory_utilization\":0.8,\n \"backend.enforce_eager\":False,\n \"backend.trust_remote_code\":True,\n \"backend.max_model_len\":1024*4,\n \"backend.quantization\":\"gptq\",\n }\n)\n```\n\nHere you deploy one model instance with 4 workers, each worker have 2 gpus, and each worker is a vLLM instance.\nBy default, the ByzerLLM will auto route the request to the worker, and the worker is selected by the LRU algorithm.\nYou can change this behavior by using the following code:\n\n```python\nllm.setup_gpus_per_worker(8).setup_num_workers(1).setup_infer_backend(InferBackend.VLLM)\nllm.setup_worker_concurrency(999)\nllm.setup_load_balance_way(\"round_robin\")\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/auto\",\n udf_name=chat_model_name,\n infer_params={\"backend.gpu_memory_utilization\":0.8,\n \"backend.enforce_eager\":False,\n \"backend.trust_remote_code\":True,\n \"backend.max_model_len\":1024*4,\n \"backend.quantization\":\"gptq\",\n }\n)\n```\n\nHere you can use the `setup_worker_concurrency` to setup the worker concurrency, and use the `setup_load_balance_way` to setup the load balance way. For now, the Byzer-LLM support the following load balance way:\n\n1. round_robin\n2. lru\n\n\n## Multi Modal \n\nThe Byzer-llm also support multi modal. The following code will deploy a multi modal model and then use the model to infer the input text.\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,InferBackend\n\nray.init(address=\"auto\",namespace=\"default\") \n\nllm = ByzerLLM()\nchat_model_name = \"qwen_vl_chat\"\nmodel_location = \"/home/byzerllm/models/Qwen-VL-Chat\"\n\nllm.setup_gpus_per_worker(1).setup_num_workers(1).setup_infer_backend(InferBackend.Transformers)\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/qwen_vl_chat\",\n udf_name=chat_model_name,\n infer_params={}\n) \n```\n\nThen you can use the model to chat:\n\n```python\nimport base64\nimage_path = \"/home/byzerllm/projects/jupyter-workspace/1.jpg\"\nwith open(image_path, \"rb\") as f:\n image_content = base64.b64encode(f.read()).decode(\"utf-8\")\n\nt = llm.chat_oai(conversations=[{\n \"role\": \"user\",\n \"content\": \"这是什么\"\n}],model=chat_model_name,llm_config={\"image\":image_content})\n\nt[0].output\n\n# '{\"response\": \"图中是一名女子在沙滩上和狗玩耍，旁边的狗是一只拉布拉多犬，它坐在沙滩上，面对着一名身穿格子衬衫的女子。女子的腿有些残疾，但是她依然坚持坐在沙滩上和狗玩耍。她的右手拿着一个小玩具，这个玩具上面有两行黑色字母，具体是什么内容看不清楚。她打算把玩具扔给拉布拉多犬。\", \"history\": [{\"role\": \"user\", \"content\": \"Picture 1: <img>/tmp/byzerllm/visualglm/images/23eb4cea-cb6e-4f55-8adf-3179ca92ab42.jpg</img>\\\\n这是什么\"}, {\"role\": \"assistant\", \"content\": \"图中是一名女子在沙滩上和狗玩耍，旁边的狗是一只拉布拉多犬，它坐在沙滩上，面对着一名身穿格子衬衫的女子。女子的腿有些残疾，但是她依然坚持坐在沙滩上和狗玩耍。她的右手拿着一个小玩具，这个玩具上面有两行黑色字母，具体是什么内容看不清楚。她打算把玩具扔给拉布拉多犬。\"}]}'\n```\n\nYou can chat multi rounds with the following code:\n\n```python\nimport json\nhistory = json.loads(t[0].output)[\"history\"]\n\nllm.chat_oai(conversations=history+[{\n \"role\": \"user\",\n \"content\": \"能圈出狗么？\"\n}],model=chat_model_name,llm_config={\"image\":image_content})\n\n# [LLMResponse(output='{\"response\": \"<ref>狗</ref><box>(221,425),(511,889)</box>\", \"history\": [{\"role\"\n```\n\nGet the history from the previous chat, then add the hisotry to new conversation, then chat again.\n\n## StableDiffusion\n\nThe Byzer-llm also support StableDiffusion as the inference backend. The following code will deploy a StableDiffusion model and then use the model to infer the input text.\n\n```python\nimport ray\nfrom byzerllm.utils.client import ByzerLLM,InferBackend\n\nray.init(address=\"auto\",namespace=\"default\") \n\nllm = ByzerLLM()\nchat_model_name = \"sd_chat\"\nmodel_location = \"/home/byzerllm/models/stable-diffusion-v1-5\"\n\nllm.setup_gpus_per_worker(2).setup_num_workers(1).setup_infer_backend(InferBackend.Transformers)\nllm.deploy(\n model_path=model_location,\n pretrained_model_type=\"custom/stable_diffusion\",\n udf_name=chat_model_name,\n infer_params={}\n)\n\ndef show_image(content):\n from IPython.display import display, Image\n import base64 \n img = Image(base64.b64decode(content))\n display(img) \n \n```\n\nThen you can chat with the model:\n\n```python\nimport json\nt = llm.chat_oai(\n conversations=[\n {\n \"role\":\"user\",\n \"content\":\"画一只猫\"\n }\n ],model=chat_model_name,llm_config={\"gen.batch_size\":3}\n)\n\ncats = json.loads(t[0].output)\nfor res in cats:\n show_image(res[\"img64\"])\n```\n\nThe output:\n\n\n\nThe parameters:\n\n| 参数 | 含义 | 默认值 |\n| --------------------------- | ------------------------------------------------------------ | -------- |\n| Instruction | prompt | 非空 |\n| generation.negative_prompt | 反向的prompt | \"\" |\n| generation.sampler_name | 调度名(unpic, euler_a,euler,ddim,ddpm,deis,dpm2,dpm2-a,dpm++_2m,dpm++_2m_karras,heun,heun_karras,lms,pndm:w) | euler_a |\n| generation.sampling_steps | 生成的步骤数 | 25 |\n| generation.batch_size | 一次生成几张 | 1 |\n| generation.batch_count | 生成几次 | 1 |\n| generation.cfg_scale | 随机或贴合程度值,值越小生成的图片离你的Tags描述的内容差距越大 | 7.5 |\n| generation.seed | 随机种子 | -1 |\n| generation.width | 图片宽度 | 768 |\n| generation.height | 图片高度 | 768 |\n| generation.enable_hires | 开启高分辨率修复功能(和下面两个一组) | false |\n| generation.upscaler_mode | 放大算法(bilinear, bilinear-antialiased,bicubic,bicubic-antialiased,nearest,nearest-exact) | bilinear |\n| generation.scale_slider | 放大比例 | 1.5 |\n| generation.enable_multidiff | 图片分割处理(减少显存销耗)(和下面3个一组) | false |\n| generation.views_batch_size | 分批处理规模 | 4 |\n| generation.window_size | 切割大小，宽，高 | 64 |\n| generation.stride | 步长 | 16 |\n| generation.init_image | 初始化图片，基于这个图片处理(必须传输base64加密的图片) (和下面的一组) | None |\n| generation.strength | 重绘幅度: 图像模仿自由度，越高越自由发挥，越低和参考图像越接近，通常小于0.3基本就是加滤镜 | 0.5 |\n\n\n\n## SQL Support\n\nIn addition to the Python API, Byzer-llm also support SQL API. In order to use the SQL API, you should install Byzer-SQL language first.\n\nTry to install the Byzer-SQL language with the following command:\n\n```bash\ngit clone https://gitee.com/allwefantasy/byzer-llm\ncd byzer-llm/setup-machine\nsudo -i \nROLE=master ./setup-machine.sh\n```\n\nAfter the installation, you can visit the Byzer Console at http://localhost:9002. \n\nIn the Byzer Console, you can run the following SQL to deploy a llama2 model which have the same effect as the Python code above.\n\n```sql\n!byzerllm setup single;\n!byzerllm setup \"num_gpus=4\";\n!byzerllm setup \"maxConcurrency=1\";\n!byzerllm setup \"infer_backend=transformers\";\n\nrun command as LLM.`` where \naction=\"infer\"\nand pretrainedModelType=\"custom/llama2\"\nand localModelDir=\"/home/byzerllm/models/openbuddy-llama-13b-v5-fp16\"\nand reconnect=\"false\"\nand udfName=\"llama2_chat\"\nand modelTable=\"command\";\n\n```\n\nThen you can invoke the model with UDF `llama2_chat`:\n\n```sql\n\nselect \nllama2_chat(llm_param(map(\n \"user_role\",\"User\",\n \"assistant_role\",\"Assistant\",\n \"system_msg\",'You are a helpful assistant. Think it over and answer the user question correctly.',\n \"instruction\",llm_prompt('\nPlease remenber my name: {0} \n',array(\"Zhu William\"))\n\n))) as q \nas q1;\n```\n\nOnce you deploy the model with `run command as LLM`, then you can ues the model as a SQL function. This feature is very useful for data scientists who want to use LLM in their data analysis or data engineers who want to use LLM in their data pipeline.\n\n---\n\n### QWen\n\nIf you use QWen in ByzerLLM, you should sepcify the following parameters mannualy:\n\n1. the role mapping \n2. the stop_token_ids\n3. trim the stop tokens from the output\n\nHowever, we provide a template for this, try to the following code:\n\n```python\nfrom byzerllm.utils.client import Templates\n\n### Here,we setup the template for qwen\nllm.setup_template(\"chat\",Templates.qwen())\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\":\"你好,给我讲个100字的笑话吧?\"\n}])\nprint(t)\n```\n\n---\n## SaaS Models\n\nSince the different SaaS models have different parameters, here we provide some templates for the SaaS models to help you deploy the SaaS models.\n\n\n### baichuan/百川\n\n```python\n\nimport ray\nfrom byzerllm.utils.client import ByzerLLM\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nllm = ByzerLLM(verbose=True)\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nchat_name = \"baichuan_chat2\"\nif llm.is_model_exist(chat_name):\n llm.undeploy(udf_name=chat_name)\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/baichuan\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"xxxxxxx\", \n \"saas.model\":\"Baichuan2-Turbo\"\n })\n\nllm.chat_oai(model=chat_name,conversations=[{\n \"role\":\"user\",\n \"content\":\"你好\",\n}]) \n```\nThere are some enum values for the `saas.model`:\n\n1. Baichuan2-Turbo\n2. Baichuan-Text-Embedding\n\n### qianwen_通义千问\n\n#### Command\n\n```shell\nbyzerllm deploy --pretrained_model_type saas/qianwen \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--worker_concurrency 10 \\\n--infer_params saas.api_key=${MODEL_QIANWEN_TOKEN} saas.model=qwen-max \\\n--model qianwen_chat\n```\n\n#### Python\n\n```python\nfrom byzerllm.utils.client import ByzerLLM\nllm = ByzerLLM()\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nchat_name = \"qianwen_chat\"\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/qianwen\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"xxxxxxx\", \n \"saas.model\":\"qwen-turbo\"\n })\n\n## here you can use `stream_chat_oai`\nv = llm.stream_chat_oai(model=chat_name,conversations=[{\n \"role\":\"user\",\n \"content\":\"你好，你是谁\",\n}],llm_config={\"gen.incremental_output\":False})\n\nfor t in v:\n print(t,flush=True) \n```\n\nThere are some enum values for the `saas.model`:\n\n1. qwen-turbo\n2. qwen-max\n\n### qianwen_emb_通义千问\n\n#### Command\n\n```shell\nbyzerllm deploy --pretrained_model_type saas/qianwen \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 2 \\\n--infer_params saas.api_key=${MODEL_QIANWEN_TOKEN} saas.model=text-embedding-v2 \\\n--model qianwen_emb\n```\n\n### yi_vl_plus/01万物多模态\n\nDeploy:\n\n```shell\nbyzerllm deploy --pretrained_model_type saas/official_openai --infer_params saas.api_key=xxxxx saas.model=yi-vl-plus saas.base_url=https://api.lingyiwanwu.com/v1 --model yi_vl_chat\n``` \n\nChat:\n \n```python\nfrom dataclasses import asdict\nimport json\nimport base64\n\nimage_url = base64.b64encode(open(\"/home/winubuntu/projects/jupyter-workspace/H/jupyter-workspace/1.jpg\",\"rb\").read()).decode()\n\nv = llm.chat_oai(model=\"yi_vl_chat\",conversations=[{\n \"role\":\"user\",\n \"content\":json.dumps([{ \n \"text\":\"详细描述图片里都有啥\"\n },{ \n \"image\":f'data:image/jpg,{image_url}'\n }],ensure_ascii=False)\n}])\n\nv\n```\n\n\n\n### qianwen_vl/通义千问多模态\n\n#### Command\n\n```shell\nbyzerllm deploy --pretrained_model_type saas/qianwen_vl \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--worker_concurrency 10 \\\n--infer_params saas.api_key=${MODEL_QIANWEN_TOKEN} saas.model=qwen-vl-max \\\n--model qianwen_vl_max_chat\n```\n\n#### Python\n\n```python\nimport os\nos.environ[\"RAY_DEDUP_LOGS\"] = \"0\" \n\nimport ray\nfrom byzerllm.utils.retrieval import ByzerRetrieval\nfrom byzerllm.utils.client import ByzerLLM,LLMRequest,LLMResponse,LLMHistoryItem,InferBackend\nfrom byzerllm.utils.client import Templates\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nllm = ByzerLLM(verbose=True)\n\nllm.setup_cpus_per_worker(0.001).setup_num_workers(1).setup_gpus_per_worker(0)\n\n\nchat_name = \"qianwen_vl_chat\"\nif llm.is_model_exist(chat_name):\n llm.undeploy(udf_name=chat_name)\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/qianwen_vl\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"xxxxx\" ,\n \"saas.model\":\"qwen-vl-plus\" \n })\n```\n\nThere are some enum values for the `saas.model`:\n\n1. qwen-vl-plus\n2. qwen-vl-max\n\nYou can call the model like this:\n\n```python\nfrom dataclasses import asdict\nimport json\nimport base64\n\nimport byzerllm\n\nbyzerllm.connect_cluster()\n\nmodel_name = \"qianwen_vl_chat\"\nllm = byzerllm.ByzerLLM()\nllm.setup_template(model=model_name,template=\"auto\")\nllm.setup_default_model_name(model_name)\n\npath = \"/home/winubuntu/projects/jupyter-workspace/H/jupyter-workspace/serving/t.png\"\n\nwith open(path, 'rb') as image_file:\n image = base64.b64encode(image_file.read()).decode('utf-8')\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\":json.dumps([{\n \"image\":image,\n \"text\":\"图片里有什么\"\n }],ensure_ascii=False)\n}])\nprint(t[0].output)\n```\n\n\n### azure openai\n\n```sql\n\n!byzerllm setup single;\n!byzerllm setup \"num_gpus=0\";\n!byzerllm setup \"maxConcurrency=10\";\n\nrun command as LLM.`` where\naction=\"infer\"\nand pretrainedModelType=\"saas/azure_openai\"\nand `saas.api_type`=\"azure\"\nand `saas.api_key`=\"xxx\"\nand `saas.api_base`=\"xxx\"\nand `saas.api_version`=\"2023-07-01-preview\"\nand `saas.deployment_id`=\"xxxxx\"\nand udfName=\"azure_openai\"\nand modelTable=\"command\";\n```\n\n### openai\n\n#### Command Line\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/openai \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--worker_concurrency 10 \\\n--num_workers 1 \\\n--infer_params saas.api_key=${MODEL_OPENAI_TOKEN} saas.model=gpt-3.5-turbo-0125 \\\n--model gpt3_5_chat\n```\n\n#### Python\n\n```python\n\nimport ray\n\nfrom byzerllm.utils.client import ByzerLLM\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nllm = ByzerLLM()\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nchat_name = \"openai_chat\"\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/openai\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"xxxx\", \n \"saas.model\":\"gpt-3.5-turbo-1106\"\n })\n```\n\nIf you need to use proxy, you can setup the proxy with the following code:\n\n```python\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/official_openai\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"xxxx\", \n \"saas.model\":\"gpt-3.5-turbo-1106\"\n \"saas.base_url\": \"http://my.test.server.example.com:8083\",\n \"saas.proxies\":\"http://my.test.proxy.example.com\"\n \"saas.local_address\":\"0.0.0.0\"\n })\n```\n\n\n### zhipu/智谱\n\n```python\nimport ray\n\nfrom byzerllm.utils.client import ByzerLLM\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True) \n\nllm = ByzerLLM()\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nchat_name = \"zhipu_chat\"\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/zhipu\",\n udf_name=chat_name,\n infer_params={\n \"saas.api_key\":\"xxxx\", \n \"saas.model\":\"glm-4\"\n })\n```\n\nThere are some enum values for the `saas.model`:\n\n1. glm-4\n2. embedding-2\n\n### minimax\n\n```sql\n\n!byzerllm setup single;\n!byzerllm setup \"num_gpus=0\";\n!byzerllm setup \"maxConcurrency=10\";\n\nrun command as LLM.`` where\naction=\"infer\"\nand pretrainedModelType=\"saas/minimax\"\nand `saas.api_key`=\"xxxxxxxxxxxxxxxxxx\"\nand `saas.group_id`=\"xxxxxxxxxxxxxxxx\"\nand `saas.model`=\"abab5.5-chat\"\nand `saas.api_url`=\"https://api.minimax.chat/v1/text/chatcompletion_pro\"\nand udfName=\"minimax_saas\"\nand modelTable=\"command\";\n\n```\n\n### sparkdesk_星火\n\n#### Command\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/sparkdesk \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.appid=xxxx saas.api_key=xxxxx saas.api_secret=xxxxxx saas.gpt_url=\"wss://spark-api.xf-yun.com/v3.5/chat\" \\\n--model sparkdesk_chat\n```\n\nvalidate the model:\n\n```bash\nbyzerllm query --model sparkdesk_chat --query '你好'\n```\n\nun-deploy the model:\n\n```bash\nbyzerllm undeploy --model sparkdesk_chat\n```\n\n#### Python\n\n```python\nimport ray\n\nfrom byzerllm.utils.client import ByzerLLM\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\n\nllm = ByzerLLM()\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nchat_name = \"sparkdesk_saas\"\n\nif llm.is_model_exist(chat_name):\n llm.undeploy(udf_name=chat_name)\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/sparkdesk\",\n udf_name=chat_name,\n infer_params={\n \"saas.appid\":\"xxxxxxx\",\n \"saas.api_key\":\"xxxxxxx\",\n \"saas.api_secret\":\"xxxxxxx\",\n \"saas.gpt_url\":\"wss://spark-api.xf-yun.com/v3.1/chat\",\n \"saas.domain\":\"generalv3\"\n })\n\nv = llm.chat_oai(model=chat_name,conversations=[{\n \"role\":\"user\",\n \"content\":\"your prompt content\",\n}])\n```\nsparkdesk V1.5 request URL，associated domain parameter is general： \n`wss://spark-api.xf-yun.com/v1.1/chat` \nsparkdesk V2 request URL，associated domain parameter is generalv2： \n`wss://spark-api.xf-yun.com/v2.1/chat` \nsparkdesk V3 request URL，associated domain parameter is generalv3 (Function Call feature is now supported.)： \n`wss://spark-api.xf-yun.com/v3.1/chat` \n\n#### Byzer-SQL\n\n```sql\n!byzerllm setup single;\n!byzerllm setup \"num_gpus=0\";\n!byzerllm setup \"maxConcurrency=10\";\n\nrun command as LLM.`` where\naction=\"infer\"\nand pretrainedModelType=\"saas/sparkdesk\"\nand `saas.appid`=\"xxxxxxxxxxxxxxxxxx\"\nand `saas.api_key`=\"xxxxxxxxxxxxxxxx\"\nand `saas.api_secret`=\"xxxx\"\nand `gpt_url`=\"ws://spark-api.xf-yun.com/v1.1/chat\"\nand udfName=\"sparkdesk_saas\"\nand modelTable=\"command\";\n```\n\n### AmazonBedrock\n\n```python\nimport ray\n\nfrom byzerllm.utils.client import ByzerLLM, Templates\n\nray.init(address=\"auto\",namespace=\"default\",ignore_reinit_error=True)\n\nllm = ByzerLLM()\n\nchat_name = \"aws_bedrock_llama2_70b_chat\"\n\nllm.setup_num_workers(1).setup_gpus_per_worker(0)\n\nif llm.is_model_exist(chat_name):\n llm.undeploy(udf_name=chat_name)\n\nllm.deploy(model_path=\"\",\n pretrained_model_type=\"saas/aws_bedrock\",\n udf_name=chat_name,\n infer_params={\n \"saas.aws_access_key\": \"your access key\",\n \"saas.aws_secret_key\": \"your secret key\",\n \"saas.region_name\": \"your region name\",\n \"saas.model_api_version\": \"model api version\",\n \"saas.model\": \"meta.llama2-70b-chat-v1\"\n })\n\nv = llm.chat_oai(model=chat_name,conversations=[{\n \"role\":\"user\",\n \"content\":\"your prompt content\",\n}])\n```\n\nThere are some enum values for the `saas.model`:\n\n1. meta.llama2-70b-chat-v1\n2. meta.llama2-13b-chat-v1\n3. anthropic.claude-3-sonnet-20240229-v1:0\n4. anthropic.claude-3-haiku-20240307-v1:0\n\n### Claude\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/claude \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=xxxxx saas.model=claude-3-haiku-20240307 \\\n--model haiku_chat\n```\n\nHere is the list of the claude models:\n\n```\nClaude 3 Opus\tclaude-3-opus-20240229\nClaude 3 Sonnet\tclaude-3-sonnet-20240229\nClaude 3 Haiku\tclaude-3-haiku-20240307\n```\n### Gemini\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/gemini \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=xxxxx saas.model=gemini-pro \\\n--model gemini_chat\n```\n\n### Moonshot_Kimi\n\nYou can use the saas/offical_openai to deploy the Kimi model.\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/openai \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 2 \\\n--infer_params saas.api_key=xxxxx saas.base_url=\"https://api.moonshot.cn/v1\" saas.model=moonshot-v1-8k \\\n--model kimi_8k_chat\n```\n\n### DeepSeek\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/openai \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--worker_concurrency 10 \\\n--num_workers 1 \\\n--infer_params saas.base_url=\"https://api.deepseek.com/v1\" saas.api_key=${MODEL_DEEPSEEK_TOKEN} saas.model=deepseek-chat \\\n--model deepseek_chat\n```\n\n### volcano_tts\n\nDeploy:\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/volcengine \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=xxxxx saas.app_id=xxxx saas.model=volcano_tts \\\n--model volcano_tts\n```\n\nUsage:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"volcano_tts\")\n\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"你好，大帅哥\",\n \"voice\": \"BV705_streaming\",\n \"response_format\": \"mp3\"\n },ensure_ascii=False)\n}])\n\nwith open(\"voice.mp3\",\"wb\") as f:\n f.write(base64.b64decode(t[0].output))\n```\n\nOr stream mode:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"volcano_tts\")\n\nt = llm.stream_chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"你好，大帅哥\",\n \"voice\": \"BV705_streaming\",\n \"response_format\": \"mp3\"\n },ensure_ascii=False)\n}])\n\nwith open(\"voice.mp3\",\"ab\") as f:\n for s in t: \n f.write(s[0])\n```\n\nVoice List: https://www.volcengine.com/docs/6561/97465\n\n\n\n### azure_tts\n\nDeploy:\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/azure \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=${MODEL_AZURE_TTS_TOKEN} saas.service_region=eastus \\\n--model azure_tts\n```\n\n\nPython:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"azure_tts\")\n\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"你好，azure_tts\",\n \"voice\": \"zh-CN-XiaoxiaoNeural\",\n \"response_format\": \"mp3\"\n },ensure_ascii=False)\n}])\n\nwith open(\"voice.mp3\",\"wb\") as f:\n f.write(base64.b64decode(t[0].output))\n```\n\nOr stream mode:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"azure_tts\")\n\nt = llm.stream_chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"你好，流式 azure_tts\",\n \"voice\": \"zh-CN-XiaoxiaoNeural\",\n \"response_format\": \"mp3\"\n },ensure_ascii=False)\n}])\n\nwith open(\"voice.mp3\",\"ab\") as f:\n for s in t: \n f.write(s[0])\n```\n\nLanguage list: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=stt\nVoice List: https://speech.microsoft.com/portal/voicegallery\n\n### openai_image_generation\n\nDeploy:\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/openai \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=${MODEL_OPENAI_TOKEN} saas.model=dall-e-3 \\\n--model openai_image_gen\n```\n\nPython Usage:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"openai_image_gen\")\n\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"a white siamese cat\",\n \"size\": \"1024x1024\",\n \"quality\": \"standard\"\n },ensure_ascii=False)\n}])\n\nwith open(\"output1.jpg\",\"wb\") as f:\n f.write(base64.b64decode(t[0].output))\n```\n\n\n\n### openai_tts\n\nDeploy:\n\n```bash\nbyzerllm deploy --pretrained_model_type saas/openai \\\n--cpus_per_worker 0.001 \\\n--gpus_per_worker 0 \\\n--num_workers 1 \\\n--infer_params saas.api_key=${MODEL_OPENAI_TOKEN} saas.model=tts-1 \\\n--model openai_tts\n```\n\nPython:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"open_tts\")\n\n\nt = llm.chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"hello, open_tts\",\n \"voice\": \"alloy\",\n \"response_format\": \"mp3\"\n },ensure_ascii=False)\n}])\n\nwith open(\"voice.mp3\",\"wb\") as f:\n f.write(base64.b64decode(t[0].output))\n```\n\nOr stream mode:\n\n```python\nimport byzerllm\nimport base64\nimport json\n\nbyzerllm.connect_cluster()\n\nllm = byzerllm.ByzerLLM()\nllm.setup_default_model_name(\"openai_tts\")\n\nt = llm.stream_chat_oai(conversations=[{\n \"role\":\"user\",\n \"content\": json.dumps({\n \"input\":\"hello openai_tts\",\n \"voice\": \"alloy\",\n \"response_format\": \"mp3\"\n },ensure_ascii=False)\n}])\n\nwith open(\"voice.mp3\",\"ab\") as f:\n for s in t: \n f.write(s[0])\n```\n\nVoice List: https://platform.openai.com/docs/guides/text-to-speech\n\n\n---\n\n## Pretrain\n\nThis section will introduce how to pretrain a LLM model with Byzer-llm. However, for now, the pretrain feature is more mature in Byzer-SQL, so we will introduce the pretrain feature in Byzer-SQL.\n\n```sql\n-- Deepspeed Config\nset ds_config='''\n{\n \"gradient_accumulation_steps\": 1,\n \"train_micro_batch_size_per_gpu\": 1,\n \"prescale_gradients\": false,\n \"zero_allow_untested_optimizer\": true,\n \"optimizer\": {\n \"type\": \"AdamW\",\n \"params\": {\n \"lr\": 1e-8,\n \"eps\": 1.0e-8,\n \"betas\": [\n 0.9,\n 0.95\n ],\n \"weight_decay\": 0.1\n }\n },\n \"tensorboard\": {\n \"enabled\": true\n },\n \"zero_optimization\": {\n \"stage\": 3,\n \"offload_optimizer\": {\n \"device\": \"cpu\" \n }, \n \"offload_param\": {\n \"device\": \"cpu\"\n },\n \"contiguous_gradients\": true,\n \"allgather_bucket_size\": 1e8,\n \"reduce_bucket_size\": 1e8,\n \"overlap_comm\": true,\n \"reduce_scatter\": true\n },\n \"steps_per_print\": 16,\n \"gradient_clipping\": 1.0,\n \"wall_clock_breakdown\": true,\n \"bf16\": {\n \"enabled\": true\n }\n}\n''';\n\n-- load data\nload text.`file:///home/byzerllm/data/raw_data/*`\nwhere wholetext=\"true\" as trainData;\n\nselect value as text,file from trainData as newTrainData;\n\n-- split the data into 12 partitions\nrun newTrainData as TableRepartition.`` where partitionNum=\"12\" and partitionCols=\"file\" \nas finalTrainData;\n\n\n-- setup env, we use 12 gpus to pretrain the model\n!byzerllm setup sfft;\n!byzerllm setup \"num_gpus=12\";\n\n-- specify the pretrain model type and the pretrained model path\nrun command as LLM.`` where \nand localPathPrefix=\"/home/byzerllm/models/sfft/jobs\"\nand pretrainedModelType=\"sfft/llama2\"\n-- original model is from\nand localModelDir=\"/home/byzerllm/models/Llama-2-7b-chat-hf\"\n-- and localDataDir=\"/home/byzerllm/data/raw_data\"\n\n-- we use async mode to pretrain the model, since the pretrain process will take several days or weeks\n-- Ray Dashboard will show the tensorboard address, and then you can monitor the loss\nand detached=\"true\"\nand keepPartitionNum=\"true\"\n\n-- use deepspeed config, this is optional\nand deepspeedConfig='''${ds_config}'''\n\n\n-- the pretrain data is from finalTrainData table\nand inputTable=\"finalTrainData\"\nand outputTable=\"llama2_cn\"\nand model=\"command\"\n-- some hyper parameters\nand `sfft.int.max_length`=\"128\"\nand `sfft.bool.setup_nccl_socket_ifname_by_ip`=\"true\"\n;\n```\n\nSince the deepspeed checkpoint is not compatible with the huggingface checkpoint, we need to convert the deepspeed checkpoint to the huggingface checkpoint. The following code will convert the deepspeed checkpoint to the huggingface checkpoint.\n\n```sql\n!byzerllm setup single;\n\nrun command as LLM.`` where \naction=\"convert\"\nand pretrainedModelType=\"deepspeed/llama3b\"\nand modelNameOrPath=\"/home/byzerllm/models/base_model\"\nand checkpointDir=\"/home/byzerllm/data/checkpoints\"\nand tag=\"Epoch-1\"\nand savePath=\"/home/byzerllm/models/my_3b_test2\";\n```\n\n\nNow you can deploy the converted model :\n\n```sql\n-- 部署hugginface 模型\n!byzerllm setup single;\n\nset node=\"master\";\n!byzerllm setup \"num_gpus=2\";\n!byzerllm setup \"workerMaxConcurrency=1\";\n\nrun command as LLM.`` where \naction=\"infer\"\nand pretrainedModelType=\"custom/auto\"\nand localModelDir=\"/home/byzerllm/models/my_3b_test2\"\nand reconnect=\"false\"\nand udfName=\"my_3b_chat\"\nand modelTable=\"command\";\n```\n\n## Finetune\n\n```sql\n-- load data, we use the dummy data for finetune\n-- data format supported by Byzer-SQL：https://docs.byzer.org/#/byzer-lang/zh-cn/byzer-llm/model-sft\n\nload json.`/tmp/upload/dummy_data.jsonl` where\ninferSchema=\"true\"\nas sft_data;\n\n-- Fintune Llama2\n!byzerllm setup sft;\n!byzerllm setup \"num_gpus=4\";\n\nrun command as LLM.`` where \nand localPathPrefix=\"/home/byzerllm/models/sft/jobs\"\n\n-- 指定模型类型\nand pretrainedModelType=\"sft/llama2\"\n\n-- 指定模型\nand localModelDir=\"/home/byzerllm/models/Llama-2-7b-chat-hf\"\nand model=\"command\"\n\n-- 指定微调数据表\nand inputTable=\"sft_data\"\n\n-- 输出新模型表\nand outputTable=\"llama2_300\"\n\n-- 微调参数\nand detached=\"true\"\nand `sft.int.max_seq_length`=\"512\";\n```\n\nYou can check the finetune actor in the Ray Dashboard, the name of the actor is `sft-william-xxxxx`.\n\nAfter the finetune actor is finished, you can get the model path, so you can deploy the finetuned model.\n\n\nHere is the log of the finetune actor:\n\n```\nLoading data: /home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_data/data.jsonl3\n2\nthere are 33 data in dataset\n*** starting training ***\n{'train_runtime': 19.0203, 'train_samples_per_second': 1.735, 'train_steps_per_second': 0.105, 'train_loss': 3.0778136253356934, 'epoch': 0.97}35\n\n***** train metrics *****36 \nepoch = 0.9737 \ntrain_loss = 3.077838 \ntrain_runtime = 0:00:19.0239 \ntrain_samples_per_second = 1.73540 \ntrain_steps_per_second = 0.10541\n\n[sft-william] Copy /home/byzerllm/models/Llama-2-7b-chat-hf to /home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_model/final/pretrained_model4243 \n[sft-william] Train Actor is already finished. You can check the model in: /home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_model/final \n```\n\nYou can download the finetuned model from the path `/home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_model/final`, or copy the model to all other node in the Ray cluster.\n\nTry to deploy the finetuned model:\n\n```sql\n!byzerllm setup single;\nrun command as LLM.`` where \naction=\"infer\"\nand localPathPrefix=\"/home/byzerllm/models/infer/jobs\"\nand localModelDir=\"/home/byzerllm/models/sft/jobs/sft-william-llama2-alpaca-data-ccb8fb55-382c-49fb-af04-5cbb3966c4e6/finetune_model/final\"\nand pretrainedModelType=\"custom/llama2\"\nand udfName=\"fintune_llama2_chat\"\nand modelTable=\"command\";\n```\n\nByzer-LLM use QLora to finetune the model, you can merge the finetuned model with the original model with the following code:\n\n```sql\n-- 合并lora model + base model\n\n!byzerllm setup single;\n\nrun command as LLM.`` where \naction=\"convert\"\nand pretrainedModelType=\"deepspeed/llama\"\nand model_dir=\"/home/byzerllm/models/sft/jobs/sft-william-20230912-21-50-10-2529bf9f-493e-40a3-b20f-0369bd01d75d/finetune_model/final/pretrained_model\"\nand checkpoint_dir=\"/home/byzerllm/models/sft/jobs/sft-william-20230912-21-50-10-2529bf9f-493e-40a3-b20f-0369bd01d75d/finetune_model/final\"\nand savePath=\"/home/byzerllm/models/sft/jobs/sft-william-20230912-21-50-10-2529bf9f-493e-40a3-b20f-0369bd01d75d/finetune_model/merge\";\n\n```\n\n## Articles\n\n1. [一口气集成那些个大模型你也试试](https://www.51xpage.com/ai/yi-kou-qi-ji-cheng-na-xie-ge-da-mo-xing-ni-ye-shi-shi-unknown-unknown-man-man-xue-ai006/)\n2. [Byzer-LLM 快速体验智谱 GLM-4](https://mp.weixin.qq.com/s/Zhzn_C9-dKP4Nq49h8yUxw)\n3. [函数实现越通用越好？来看看 Byzer-LLM 的 Function Implementation 带来的编程思想大变化](https://mp.weixin.qq.com/s/_Sx0eC0WqC2M4K1JY9f49Q)\n4. [Byzer-LLM 之 QWen-VL-Chat/StableDiffusion多模态读图，生图](https://mp.weixin.qq.com/s/x4g66QvocE5dUlnL1yF9Dw)\n5. [基于Byzer-Agent 框架开发智能数据分析工具](https://mp.weixin.qq.com/s/BcoHUEXF24wTjArc7mwNaw)\n6. [Byzer-LLM 支持同时开源和SaaS版通义千问](https://mp.weixin.qq.com/s/VvzMUV654D7IO0He47nv3A)\n7. [给开源大模型带来Function Calling、 Respond With Class](https://mp.weixin.qq.com/s/GTVCYUhR_atYMX9ymp0eCg)"}}, "src/byzerllm/utils/modelinfo.py": {"8d28eee121fe7d2048dfb70aaf0ff32d31d396f5": {"t": 1785746502.293939, "n": 12706, "h": "2df589a8697bb2db", "c": "import os\nimport json\nfrom typing import List, Dict\nfrom urllib.parse import urlparse\n\nMODELS_JSON = os.path.expanduser(\"~/.auto-coder/keys/models.json\")\n\n# Default built-in models\ndefault_models_list = [\n {\n \"name\": \"deepseek/r1\",\n \"description\": \"DeepSeek Reasoner is for design/review\",\n \"model_name\": \"deepseek-reasoner\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://api.deepseek.com/v1\",\n \"api_key_path\": \"api.deepseek.com\",\n \"is_reasoning\": True,\n \"input_price\": 0.0, # 单位:M/百万 input tokens\n \"output_price\": 0.0, # 单位:M/百万 output tokens\n \"average_speed\": 0.0, # 单位:秒/请求\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"deepseek/v3\",\n \"description\": \"DeepSeek Chat is for coding\",\n \"model_name\": \"deepseek-chat\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://api.deepseek.com/v1\",\n \"api_key_path\": \"api.deepseek.com\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n },\n {\n \"name\": \"ark/deepseek-v3-250324\",\n \"description\": \"DeepSeek Chat is for coding\",\n \"model_name\": \"deepseek-v3-250324\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://ark.cn-beijing.volces.com/api/v3\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 2.0,\n \"output_price\": 8.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"ark/deepseek-r1-250120\",\n \"description\": \"DeepSeek Reasoner is for design/review\",\n \"model_name\": \"deepseek-r1-250120\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://ark.cn-beijing.volces.com/api/v3\",\n \"api_key_path\": \"\",\n \"is_reasoning\": True,\n \"input_price\": 4.0,\n \"output_price\": 16.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"openai/gpt-4.1-mini\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1-mini\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 2.8,\n \"output_price\": 11.2,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n },\n {\n \"name\": \"openai/gpt-4.1\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 14.0,\n \"output_price\": 42.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n }, \n {\n \"name\": \"openai/gpt-4.1-nano\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1-nano\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n }, \n {\n \"name\": \"openrouter/google/gemini-2.5-pro-preview-03-25\",\n \"description\": \"\",\n \"model_name\": \"google/gemini-2.5-pro-preview-03-25\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*2\n }\n]\n\ndef process_api_key_path(base_url: str) -> str:\n \"\"\"\n 从 base_url 中提取 host 部分并处理特殊字符\n 例如: https://api.example.com:8080/v1 -> api.example.com_8080\n \"\"\"\n if not base_url:\n return \"\"\n \n parsed = urlparse(base_url)\n host = parsed.netloc\n \n # 将冒号替换为下划线\n host = host.replace(\":\", \"_\")\n \n return host\n\ndef load_models() -> List[Dict]:\n \"\"\"\n Load models from ~/.auto-coder/keys/models.json and merge with default_models_list.\n Models are merged and deduplicated based on their name field.\n If file doesn't exist or is invalid, use default_models_list.\n \"\"\"\n os.makedirs(os.path.dirname(MODELS_JSON), exist_ok=True)\n \n # Start with default models\n models_dict = {model[\"name\"]: model for model in default_models_list}\n \n # If JSON file exists, read and merge with defaults\n if os.path.exists(MODELS_JSON):\n try:\n with open(MODELS_JSON, 'r', encoding='utf-8') as f:\n custom_models = json.load(f)\n # Custom models will override defaults with same name\n for model in custom_models:\n model[\"is_reasoning\"] = model.get(\"is_reasoning\", False)\n models_dict[model[\"name\"]] = model\n\n except json.JSONDecodeError:\n # If JSON is invalid, just use defaults\n print(\"JSON is invalid, using defaults\")\n save_models(default_models_list)\n else:\n # If file doesn't exist, create it with defaults\n save_models(default_models_list)\n \n # Convert merged dictionary back to list\n target_models = list(models_dict.values())\n api_key_dir = os.path.expanduser(\"~/.auto-coder/keys\")\n for model in target_models: \n if model.get(\"api_key_path\",\"\"): \n api_key_file = os.path.join(api_key_dir, model[\"api_key_path\"])\n if os.path.exists(api_key_file):\n with open(api_key_file, \"r\",encoding=\"utf-8\") as f:\n model[\"api_key\"] = f.read().strip() \n return target_models\n\ndef save_models(models: List[Dict]) -> None:\n \"\"\"\n Save models to ~/.auto-coder/keys/models.json\n \"\"\"\n os.makedirs(os.path.dirname(MODELS_JSON), exist_ok=True)\n with open(MODELS_JSON, 'w', encoding='utf-8') as f:\n json.dump(models, f, indent=2, ensure_ascii=False)\n\n\ndef add_and_activate_models(models: List[Dict]) -> None:\n \"\"\"\n 添加模型\n \"\"\"\n exits_models = load_models()\n for model in models: \n if model[\"name\"] not in [m[\"name\"] for m in exits_models]:\n exits_models.append(model)\n save_models(exits_models)\n\n for model in models:\n if \"api_key\" in model:\n update_model_with_api_key(model[\"name\"], model[\"api_key\"])\n\ndef get_model_by_name(name: str) -> Dict:\n \"\"\"\n 根据模型名称查找模型\n \"\"\"\n from autocoder.common.auto_coder_lang import get_message_with_format\n models = load_models()\n v = [m for m in models if m[\"name\"] == name.strip()]\n \n if len(v) == 0: \n raise Exception(get_message_with_format(\"model_not_found\", model_name=name))\n return v[0]\n\n\ndef update_model_input_price(name: str, price: float) -> bool:\n \"\"\"更新模型输入价格\n \n Args:\n name (str): 要更新的模型名称，必须与models.json中的记录匹配\n price (float): 新的输入价格，单位：美元/百万tokens。必须大于等于0\n \n Returns:\n bool: 是否成功找到并更新了模型价格\n \n Raises:\n ValueError: 如果price为负数时抛出\n \n Example:\n >>> update_model_input_price(\"gpt-4\", 3.0)\n True\n \n Notes:\n 1. 价格设置后会立即生效并保存到models.json\n 2. 实际费用计算时会按实际使用量精确到小数点后6位\n 3. 设置价格为0表示该模型当前不可用\n \"\"\"\n if price < 0:\n raise ValueError(\"Price cannot be negative\")\n \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"input_price\"] = float(price)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef update_model_output_price(name: str, price: float) -> bool:\n \"\"\"更新模型输出价格\n \n Args:\n name (str): 要更新的模型名称，必须与models.json中的记录匹配\n price (float): 新的输出价格，单位：美元/百万tokens。必须大于等于0\n \n Returns:\n bool: 是否成功找到并更新了模型价格\n \n Raises:\n ValueError: 如果price为负数时抛出\n \n Example:\n >>> update_model_output_price(\"gpt-4\", 6.0)\n True\n \n Notes:\n 1. 输出价格通常比输入价格高30%-50%\n 2. 对于按token计费的API，实际收费按(input_tokens * input_price + output_tokens * output_price)计算\n 3. 价格变更会影响所有依赖模型计费的功能（如成本预测、用量监控等）\n \"\"\"\n if price < 0:\n raise ValueError(\"Price cannot be negative\")\n \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"output_price\"] = float(price)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef update_model_speed(name: str, speed: float) -> bool:\n \"\"\"更新模型平均速度\n \n Args:\n name: 模型名称\n speed: 速度(秒/请求)\n \n Returns:\n bool: 是否更新成功\n \"\"\" \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"average_speed\"] = float(speed)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef check_model_exists(name: str) -> bool:\n \"\"\"\n 检查模型是否存在\n \"\"\"\n models = load_models()\n return any(m[\"name\"] == name.strip() for m in models)\n\ndef update_model_with_api_key(name: str, api_key: str) -> Dict:\n \"\"\"\n 根据模型名称查找并更新模型的 api_key_path。\n 如果找到模型,会根据其 base_url 处理 api_key_path。\n \n Args:\n name: 模型名称\n api_key: API密钥\n \n Returns:\n Dict: 更新后的模型信息,如果未找到则返回None\n \"\"\"\n models = load_models()\n \n # 在现有模型中查找\n found_model = None\n for model in models:\n if model[\"name\"] == name.strip():\n found_model = model\n break\n \n if not found_model:\n return None\n \n api_key_path = name.replace(\"/\", \"_\") # 替换 / 为 _，保证文件名合法\n if api_key_path:\n found_model[\"api_key_path\"] = api_key_path\n \n # 保存 API 密钥\n api_key_dir = os.path.expanduser(\"~/.auto-coder/keys\")\n os.makedirs(api_key_dir, exist_ok=True)\n api_key_file = os.path.join(api_key_dir, api_key_path)\n with open(api_key_file, \"w\",encoding=\"utf-8\") as f:\n f.write(api_key.strip())\n \n # 如果是新模型，添加到模型列表中\n if all(model[\"name\"] != name for model in models):\n models.append(found_model)\n else:\n # 更新现有模型\n for i, model in enumerate(models):\n if model[\"name\"] == name:\n models[i] = found_model\n break\n \n save_models(models)\n \n return found_model\n\ndef update_model(name: str, model_data: Dict) -> Dict:\n \"\"\"\n 更新模型信息\n \n Args:\n name: 要更新的模型名称\n model_data: 包含模型新信息的字典，可以包含以下字段:\n - name: 模型名称\n - description: 模型描述\n - model_name: 模型实际名称\n - model_type: 模型类型\n - base_url: 基础URL\n - api_key: API密钥\n - is_reasoning: 是否为推理模型\n - input_price: 输入价格\n - output_price: 输出价格\n - max_output_tokens: 最大输出tokens\n - average_speed: 平均速度\n \n Returns:\n Dict: 更新后的模型信息，如果未找到则返回None\n \"\"\"\n models = load_models()\n \n # 查找要更新的模型\n found = False\n for i, model in enumerate(models):\n if model[\"name\"] == name:\n # 更新模型字段\n if \"description\" in model_data:\n model[\"description\"] = model_data[\"description\"]\n if \"model_name\" in model_data:\n model[\"model_name\"] = model_data[\"model_name\"]\n if \"model_type\" in model_data:\n model[\"model_type\"] = model_data[\"model_type\"]\n if \"base_url\" in model_data:\n model[\"base_url\"] = model_data[\"base_url\"]\n if \"is_reasoning\" in model_data:\n model[\"is_reasoning\"] = model_data[\"is_reasoning\"]\n if \"input_price\" in model_data:\n model[\"input_price\"] = float(model_data[\"input_price\"])\n if \"output_price\" in model_data:\n model[\"output_price\"] = float(model_data[\"output_price\"])\n if \"max_output_tokens\" in model_data:\n model[\"max_output_tokens\"] = int(model_data[\"max_output_tokens\"])\n if \"average_speed\" in model_data:\n model[\"average_speed\"] = float(model_data[\"average_speed\"])\n \n # 保存更新后的模型\n models[i] = model\n found = True\n \n # 如果提供了API密钥，则更新\n if \"api_key\" in model_data and model_data[\"api_key\"]:\n update_model_with_api_key(name, model_data[\"api_key\"])\n \n break\n \n if found:\n save_models(models)\n return models[i]\n \n return None\n\n"}, "3ff0c8ef8e4928a872ed99d28485f1735bbfacba": {"t": 1785765013.847297, "n": 12706, "h": "2df589a8697bb2db", "c": "import os\nimport json\nfrom typing import List, Dict\nfrom urllib.parse import urlparse\n\nMODELS_JSON = os.path.expanduser(\"~/.auto-coder/keys/models.json\")\n\n# Default built-in models\ndefault_models_list = [\n {\n \"name\": \"deepseek/r1\",\n \"description\": \"DeepSeek Reasoner is for design/review\",\n \"model_name\": \"deepseek-reasoner\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://api.deepseek.com/v1\",\n \"api_key_path\": \"api.deepseek.com\",\n \"is_reasoning\": True,\n \"input_price\": 0.0, # 单位:M/百万 input tokens\n \"output_price\": 0.0, # 单位:M/百万 output tokens\n \"average_speed\": 0.0, # 单位:秒/请求\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"deepseek/v3\",\n \"description\": \"DeepSeek Chat is for coding\",\n \"model_name\": \"deepseek-chat\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://api.deepseek.com/v1\",\n \"api_key_path\": \"api.deepseek.com\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n },\n {\n \"name\": \"ark/deepseek-v3-250324\",\n \"description\": \"DeepSeek Chat is for coding\",\n \"model_name\": \"deepseek-v3-250324\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://ark.cn-beijing.volces.com/api/v3\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 2.0,\n \"output_price\": 8.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"ark/deepseek-r1-250120\",\n \"description\": \"DeepSeek Reasoner is for design/review\",\n \"model_name\": \"deepseek-r1-250120\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://ark.cn-beijing.volces.com/api/v3\",\n \"api_key_path\": \"\",\n \"is_reasoning\": True,\n \"input_price\": 4.0,\n \"output_price\": 16.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"openai/gpt-4.1-mini\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1-mini\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 2.8,\n \"output_price\": 11.2,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n },\n {\n \"name\": \"openai/gpt-4.1\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 14.0,\n \"output_price\": 42.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n }, \n {\n \"name\": \"openai/gpt-4.1-nano\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1-nano\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n }, \n {\n \"name\": \"openrouter/google/gemini-2.5-pro-preview-03-25\",\n \"description\": \"\",\n \"model_name\": \"google/gemini-2.5-pro-preview-03-25\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*2\n }\n]\n\ndef process_api_key_path(base_url: str) -> str:\n \"\"\"\n 从 base_url 中提取 host 部分并处理特殊字符\n 例如: https://api.example.com:8080/v1 -> api.example.com_8080\n \"\"\"\n if not base_url:\n return \"\"\n \n parsed = urlparse(base_url)\n host = parsed.netloc\n \n # 将冒号替换为下划线\n host = host.replace(\":\", \"_\")\n \n return host\n\ndef load_models() -> List[Dict]:\n \"\"\"\n Load models from ~/.auto-coder/keys/models.json and merge with default_models_list.\n Models are merged and deduplicated based on their name field.\n If file doesn't exist or is invalid, use default_models_list.\n \"\"\"\n os.makedirs(os.path.dirname(MODELS_JSON), exist_ok=True)\n \n # Start with default models\n models_dict = {model[\"name\"]: model for model in default_models_list}\n \n # If JSON file exists, read and merge with defaults\n if os.path.exists(MODELS_JSON):\n try:\n with open(MODELS_JSON, 'r', encoding='utf-8') as f:\n custom_models = json.load(f)\n # Custom models will override defaults with same name\n for model in custom_models:\n model[\"is_reasoning\"] = model.get(\"is_reasoning\", False)\n models_dict[model[\"name\"]] = model\n\n except json.JSONDecodeError:\n # If JSON is invalid, just use defaults\n print(\"JSON is invalid, using defaults\")\n save_models(default_models_list)\n else:\n # If file doesn't exist, create it with defaults\n save_models(default_models_list)\n \n # Convert merged dictionary back to list\n target_models = list(models_dict.values())\n api_key_dir = os.path.expanduser(\"~/.auto-coder/keys\")\n for model in target_models: \n if model.get(\"api_key_path\",\"\"): \n api_key_file = os.path.join(api_key_dir, model[\"api_key_path\"])\n if os.path.exists(api_key_file):\n with open(api_key_file, \"r\",encoding=\"utf-8\") as f:\n model[\"api_key\"] = f.read().strip() \n return target_models\n\ndef save_models(models: List[Dict]) -> None:\n \"\"\"\n Save models to ~/.auto-coder/keys/models.json\n \"\"\"\n os.makedirs(os.path.dirname(MODELS_JSON), exist_ok=True)\n with open(MODELS_JSON, 'w', encoding='utf-8') as f:\n json.dump(models, f, indent=2, ensure_ascii=False)\n\n\ndef add_and_activate_models(models: List[Dict]) -> None:\n \"\"\"\n 添加模型\n \"\"\"\n exits_models = load_models()\n for model in models: \n if model[\"name\"] not in [m[\"name\"] for m in exits_models]:\n exits_models.append(model)\n save_models(exits_models)\n\n for model in models:\n if \"api_key\" in model:\n update_model_with_api_key(model[\"name\"], model[\"api_key\"])\n\ndef get_model_by_name(name: str) -> Dict:\n \"\"\"\n 根据模型名称查找模型\n \"\"\"\n from autocoder.common.auto_coder_lang import get_message_with_format\n models = load_models()\n v = [m for m in models if m[\"name\"] == name.strip()]\n \n if len(v) == 0: \n raise Exception(get_message_with_format(\"model_not_found\", model_name=name))\n return v[0]\n\n\ndef update_model_input_price(name: str, price: float) -> bool:\n \"\"\"更新模型输入价格\n \n Args:\n name (str): 要更新的模型名称，必须与models.json中的记录匹配\n price (float): 新的输入价格，单位：美元/百万tokens。必须大于等于0\n \n Returns:\n bool: 是否成功找到并更新了模型价格\n \n Raises:\n ValueError: 如果price为负数时抛出\n \n Example:\n >>> update_model_input_price(\"gpt-4\", 3.0)\n True\n \n Notes:\n 1. 价格设置后会立即生效并保存到models.json\n 2. 实际费用计算时会按实际使用量精确到小数点后6位\n 3. 设置价格为0表示该模型当前不可用\n \"\"\"\n if price < 0:\n raise ValueError(\"Price cannot be negative\")\n \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"input_price\"] = float(price)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef update_model_output_price(name: str, price: float) -> bool:\n \"\"\"更新模型输出价格\n \n Args:\n name (str): 要更新的模型名称，必须与models.json中的记录匹配\n price (float): 新的输出价格，单位：美元/百万tokens。必须大于等于0\n \n Returns:\n bool: 是否成功找到并更新了模型价格\n \n Raises:\n ValueError: 如果price为负数时抛出\n \n Example:\n >>> update_model_output_price(\"gpt-4\", 6.0)\n True\n \n Notes:\n 1. 输出价格通常比输入价格高30%-50%\n 2. 对于按token计费的API，实际收费按(input_tokens * input_price + output_tokens * output_price)计算\n 3. 价格变更会影响所有依赖模型计费的功能（如成本预测、用量监控等）\n \"\"\"\n if price < 0:\n raise ValueError(\"Price cannot be negative\")\n \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"output_price\"] = float(price)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef update_model_speed(name: str, speed: float) -> bool:\n \"\"\"更新模型平均速度\n \n Args:\n name: 模型名称\n speed: 速度(秒/请求)\n \n Returns:\n bool: 是否更新成功\n \"\"\" \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"average_speed\"] = float(speed)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef check_model_exists(name: str) -> bool:\n \"\"\"\n 检查模型是否存在\n \"\"\"\n models = load_models()\n return any(m[\"name\"] == name.strip() for m in models)\n\ndef update_model_with_api_key(name: str, api_key: str) -> Dict:\n \"\"\"\n 根据模型名称查找并更新模型的 api_key_path。\n 如果找到模型,会根据其 base_url 处理 api_key_path。\n \n Args:\n name: 模型名称\n api_key: API密钥\n \n Returns:\n Dict: 更新后的模型信息,如果未找到则返回None\n \"\"\"\n models = load_models()\n \n # 在现有模型中查找\n found_model = None\n for model in models:\n if model[\"name\"] == name.strip():\n found_model = model\n break\n \n if not found_model:\n return None\n \n api_key_path = name.replace(\"/\", \"_\") # 替换 / 为 _，保证文件名合法\n if api_key_path:\n found_model[\"api_key_path\"] = api_key_path\n \n # 保存 API 密钥\n api_key_dir = os.path.expanduser(\"~/.auto-coder/keys\")\n os.makedirs(api_key_dir, exist_ok=True)\n api_key_file = os.path.join(api_key_dir, api_key_path)\n with open(api_key_file, \"w\",encoding=\"utf-8\") as f:\n f.write(api_key.strip())\n \n # 如果是新模型，添加到模型列表中\n if all(model[\"name\"] != name for model in models):\n models.append(found_model)\n else:\n # 更新现有模型\n for i, model in enumerate(models):\n if model[\"name\"] == name:\n models[i] = found_model\n break\n \n save_models(models)\n \n return found_model\n\ndef update_model(name: str, model_data: Dict) -> Dict:\n \"\"\"\n 更新模型信息\n \n Args:\n name: 要更新的模型名称\n model_data: 包含模型新信息的字典，可以包含以下字段:\n - name: 模型名称\n - description: 模型描述\n - model_name: 模型实际名称\n - model_type: 模型类型\n - base_url: 基础URL\n - api_key: API密钥\n - is_reasoning: 是否为推理模型\n - input_price: 输入价格\n - output_price: 输出价格\n - max_output_tokens: 最大输出tokens\n - average_speed: 平均速度\n \n Returns:\n Dict: 更新后的模型信息，如果未找到则返回None\n \"\"\"\n models = load_models()\n \n # 查找要更新的模型\n found = False\n for i, model in enumerate(models):\n if model[\"name\"] == name:\n # 更新模型字段\n if \"description\" in model_data:\n model[\"description\"] = model_data[\"description\"]\n if \"model_name\" in model_data:\n model[\"model_name\"] = model_data[\"model_name\"]\n if \"model_type\" in model_data:\n model[\"model_type\"] = model_data[\"model_type\"]\n if \"base_url\" in model_data:\n model[\"base_url\"] = model_data[\"base_url\"]\n if \"is_reasoning\" in model_data:\n model[\"is_reasoning\"] = model_data[\"is_reasoning\"]\n if \"input_price\" in model_data:\n model[\"input_price\"] = float(model_data[\"input_price\"])\n if \"output_price\" in model_data:\n model[\"output_price\"] = float(model_data[\"output_price\"])\n if \"max_output_tokens\" in model_data:\n model[\"max_output_tokens\"] = int(model_data[\"max_output_tokens\"])\n if \"average_speed\" in model_data:\n model[\"average_speed\"] = float(model_data[\"average_speed\"])\n \n # 保存更新后的模型\n models[i] = model\n found = True\n \n # 如果提供了API密钥，则更新\n if \"api_key\" in model_data and model_data[\"api_key\"]:\n update_model_with_api_key(name, model_data[\"api_key\"])\n \n break\n \n if found:\n save_models(models)\n return models[i]\n \n return None\n\n"}, "2547a0a1b4342aa04b25046e20537042b3ad1e11": {"t": 1785765015.2743263, "n": 12698, "h": "ef2de3985afd12e9", "c": "import os\nimport json\nfrom typing import List, Dict\nfrom urllib.parse import urlparse\n\nMODELS_JSON = os.path.expanduser(\"~/.auto-coder/keys/models.json\")\n\n# Default built-in models\ndefault_models_list = [\n {\n \"name\": \"deepseek/r1\",\n \"description\": \"DeepSeek Reasoner is for design/review\",\n \"model_name\": \"deepseek-reasoner\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://api.deepseek.com/v1\",\n \"api_key_path\": \"api.deepseek.com\",\n \"is_reasoning\": True,\n \"input_price\": 0.0, # 单位:M/百万 input tokens\n \"output_price\": 0.0, # 单位:M/百万 output tokens\n \"average_speed\": 0.0, # 单位:秒/请求\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"deepseek/v3\",\n \"description\": \"DeepSeek Chat is for coding\",\n \"model_name\": \"deepseek-chat\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://api.deepseek.com/v1\",\n \"api_key_path\": \"api.deepseek.com\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n },\n {\n \"name\": \"ark/deepseek-v3-250324\",\n \"description\": \"DeepSeek Chat is for coding\",\n \"model_name\": \"deepseek-v3-250324\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://ark.cn-beijing.volces.com/api/v3\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 2.0,\n \"output_price\": 8.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"ark/deepseek-r1-250120\",\n \"description\": \"DeepSeek Reasoner is for design/review\",\n \"model_name\": \"deepseek-r1-250120\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://ark.cn-beijing.volces.com/api/v3\",\n \"api_key_path\": \"\",\n \"is_reasoning\": True,\n \"input_price\": 4.0,\n \"output_price\": 16.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096\n }, \n {\n \"name\": \"openai/gpt-4.1-mini\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1-mini\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 2.8,\n \"output_price\": 11.2,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n },\n {\n \"name\": \"openai/gpt-4.1\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 14.0,\n \"output_price\": 42.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n }, \n {\n \"name\": \"openai/gpt-4.1-nano\",\n \"description\": \"\",\n \"model_name\": \"openai/gpt-4.1-nano\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*3\n }, \n {\n \"name\": \"openrouter/google/gemini-2.5-pro-preview-03-25\",\n \"description\": \"\",\n \"model_name\": \"google/gemini-2.5-pro-preview-03-25\",\n \"model_type\": \"saas/openai\",\n \"base_url\": \"https://openrouter.ai/api/v1\",\n \"api_key_path\": \"\",\n \"is_reasoning\": False,\n \"input_price\": 0.0,\n \"output_price\": 0.0,\n \"average_speed\": 0.0,\n \"max_output_tokens\": 8096*2\n }\n]\n\ndef process_api_key_path(base_url: str) -> str:\n \"\"\"\n 从 base_url 中提取 host 部分并处理特殊字符\n 例如: https://api.example.com:8080/v1 -> api.example.com_8080\n \"\"\"\n if not base_url:\n return \"\"\n \n parsed = urlparse(base_url)\n host = parsed.netloc\n \n # 将冒号替换为下划线\n host = host.replace(\":\", \"_\")\n \n return host\n\ndef load_models() -> List[Dict]:\n \"\"\"\n Load models from ~/.auto-coder/keys/models.json and merge with default_models_list.\n Models are merged and deduplicated based on their name field.\n If file doesn't exist or is invalid, use default_models_list.\n \"\"\"\n os.makedirs(os.path.dirname(MODELS_JSON), exist_ok=True)\n \n # Start with default models\n models_dict = {model[\"name\"]: model for model in default_models_list}\n \n # If JSON file exists, read and merge with defaults\n if os.path.exists(MODELS_JSON):\n try:\n with open(MODELS_JSON, 'r', encoding='utf-8') as f:\n custom_models = json.load(f)\n # Custom models will override defaults with same name\n for model in custom_models:\n model[\"is_reasoning\"] = model.get(\"is_reasoning\", False)\n models_dict[model[\"name\"]] = model\n\n except json.JSONDecodeError:\n # If JSON is invalid, just use defaults\n print(\"JSON is invalid, using defaults\")\n save_models(default_models_list)\n else:\n # If file doesn't exist, create it with defaults\n save_models(default_models_list)\n \n # Convert merged dictionary back to list\n target_models = list(models_dict.values())\n api_key_dir = os.path.expanduser(\"~/.auto-coder/keys\")\n for model in target_models: \n if model.get(\"api_key_path\",\"\"): \n api_key_file = os.path.join(api_key_dir, model[\"api_key_path\"])\n if os.path.exists(api_key_file):\n with open(api_key_file, \"r\",encoding=\"utf-8\") as f:\n model[\"api_key\"] = f.read() \n return target_models\n\ndef save_models(models: List[Dict]) -> None:\n \"\"\"\n Save models to ~/.auto-coder/keys/models.json\n \"\"\"\n os.makedirs(os.path.dirname(MODELS_JSON), exist_ok=True)\n with open(MODELS_JSON, 'w', encoding='utf-8') as f:\n json.dump(models, f, indent=2, ensure_ascii=False)\n\n\ndef add_and_activate_models(models: List[Dict]) -> None:\n \"\"\"\n 添加模型\n \"\"\"\n exits_models = load_models()\n for model in models: \n if model[\"name\"] not in [m[\"name\"] for m in exits_models]:\n exits_models.append(model)\n save_models(exits_models)\n\n for model in models:\n if \"api_key\" in model:\n update_model_with_api_key(model[\"name\"], model[\"api_key\"])\n\ndef get_model_by_name(name: str) -> Dict:\n \"\"\"\n 根据模型名称查找模型\n \"\"\"\n from autocoder.common.auto_coder_lang import get_message_with_format\n models = load_models()\n v = [m for m in models if m[\"name\"] == name.strip()]\n \n if len(v) == 0: \n raise Exception(get_message_with_format(\"model_not_found\", model_name=name))\n return v[0]\n\n\ndef update_model_input_price(name: str, price: float) -> bool:\n \"\"\"更新模型输入价格\n \n Args:\n name (str): 要更新的模型名称，必须与models.json中的记录匹配\n price (float): 新的输入价格，单位：美元/百万tokens。必须大于等于0\n \n Returns:\n bool: 是否成功找到并更新了模型价格\n \n Raises:\n ValueError: 如果price为负数时抛出\n \n Example:\n >>> update_model_input_price(\"gpt-4\", 3.0)\n True\n \n Notes:\n 1. 价格设置后会立即生效并保存到models.json\n 2. 实际费用计算时会按实际使用量精确到小数点后6位\n 3. 设置价格为0表示该模型当前不可用\n \"\"\"\n if price < 0:\n raise ValueError(\"Price cannot be negative\")\n \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"input_price\"] = float(price)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef update_model_output_price(name: str, price: float) -> bool:\n \"\"\"更新模型输出价格\n \n Args:\n name (str): 要更新的模型名称，必须与models.json中的记录匹配\n price (float): 新的输出价格，单位：美元/百万tokens。必须大于等于0\n \n Returns:\n bool: 是否成功找到并更新了模型价格\n \n Raises:\n ValueError: 如果price为负数时抛出\n \n Example:\n >>> update_model_output_price(\"gpt-4\", 6.0)\n True\n \n Notes:\n 1. 输出价格通常比输入价格高30%-50%\n 2. 对于按token计费的API，实际收费按(input_tokens * input_price + output_tokens * output_price)计算\n 3. 价格变更会影响所有依赖模型计费的功能（如成本预测、用量监控等）\n \"\"\"\n if price < 0:\n raise ValueError(\"Price cannot be negative\")\n \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"output_price\"] = float(price)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef update_model_speed(name: str, speed: float) -> bool:\n \"\"\"更新模型平均速度\n \n Args:\n name: 模型名称\n speed: 速度(秒/请求)\n \n Returns:\n bool: 是否更新成功\n \"\"\" \n models = load_models()\n updated = False\n for model in models:\n if model[\"name\"] == name:\n model[\"average_speed\"] = float(speed)\n updated = True\n break\n if updated:\n save_models(models)\n return updated\n\ndef check_model_exists(name: str) -> bool:\n \"\"\"\n 检查模型是否存在\n \"\"\"\n models = load_models()\n return any(m[\"name\"] == name.strip() for m in models)\n\ndef update_model_with_api_key(name: str, api_key: str) -> Dict:\n \"\"\"\n 根据模型名称查找并更新模型的 api_key_path。\n 如果找到模型,会根据其 base_url 处理 api_key_path。\n \n Args:\n name: 模型名称\n api_key: API密钥\n \n Returns:\n Dict: 更新后的模型信息,如果未找到则返回None\n \"\"\"\n models = load_models()\n \n # 在现有模型中查找\n found_model = None\n for model in models:\n if model[\"name\"] == name.strip():\n found_model = model\n break\n \n if not found_model:\n return None\n \n api_key_path = name.replace(\"/\", \"_\") # 替换 / 为 _，保证文件名合法\n if api_key_path:\n found_model[\"api_key_path\"] = api_key_path\n \n # 保存 API 密钥\n api_key_dir = os.path.expanduser(\"~/.auto-coder/keys\")\n os.makedirs(api_key_dir, exist_ok=True)\n api_key_file = os.path.join(api_key_dir, api_key_path)\n with open(api_key_file, \"w\",encoding=\"utf-8\") as f:\n f.write(api_key.strip())\n \n # 如果是新模型，添加到模型列表中\n if all(model[\"name\"] != name for model in models):\n models.append(found_model)\n else:\n # 更新现有模型\n for i, model in enumerate(models):\n if model[\"name\"] == name:\n models[i] = found_model\n break\n \n save_models(models)\n \n return found_model\n\ndef update_model(name: str, model_data: Dict) -> Dict:\n \"\"\"\n 更新模型信息\n \n Args:\n name: 要更新的模型名称\n model_data: 包含模型新信息的字典，可以包含以下字段:\n - name: 模型名称\n - description: 模型描述\n - model_name: 模型实际名称\n - model_type: 模型类型\n - base_url: 基础URL\n - api_key: API密钥\n - is_reasoning: 是否为推理模型\n - input_price: 输入价格\n - output_price: 输出价格\n - max_output_tokens: 最大输出tokens\n - average_speed: 平均速度\n \n Returns:\n Dict: 更新后的模型信息，如果未找到则返回None\n \"\"\"\n models = load_models()\n \n # 查找要更新的模型\n found = False\n for i, model in enumerate(models):\n if model[\"name\"] == name:\n # 更新模型字段\n if \"description\" in model_data:\n model[\"description\"] = model_data[\"description\"]\n if \"model_name\" in model_data:\n model[\"model_name\"] = model_data[\"model_name\"]\n if \"model_type\" in model_data:\n model[\"model_type\"] = model_data[\"model_type\"]\n if \"base_url\" in model_data:\n model[\"base_url\"] = model_data[\"base_url\"]\n if \"is_reasoning\" in model_data:\n model[\"is_reasoning\"] = model_data[\"is_reasoning\"]\n if \"input_price\" in model_data:\n model[\"input_price\"] = float(model_data[\"input_price\"])\n if \"output_price\" in model_data:\n model[\"output_price\"] = float(model_data[\"output_price\"])\n if \"max_output_tokens\" in model_data:\n model[\"max_output_tokens\"] = int(model_data[\"max_output_tokens\"])\n if \"average_speed\" in model_data:\n model[\"average_speed\"] = float(model_data[\"average_speed\"])\n \n # 保存更新后的模型\n models[i] = model\n found = True\n \n # 如果提供了API密钥，则更新\n if \"api_key\" in model_data and model_data[\"api_key\"]:\n update_model_with_api_key(name, model_data[\"api_key\"])\n \n break\n \n if found:\n save_models(models)\n return models[i]\n \n return None\n\n"}}}} |