1 line
19 KiB
JSON
1 line
19 KiB
JSON
{"repo": "01choco/nlp-grad-2", "files": {"eval/FastChat/docs/model_support.md": {"7a7b1e9dfe164025c656bca00de1b142f725788c": {"t": 1785747170.2154539, "n": 9278, "h": "734ef38ee4d62f10", "c": "# Model Support\nThis document describes how to support a new model in FastChat.\n\n## Content\n- [Local Models](#local-models)\n- [API-Based Models](#api-based-models)\n\n## Local Models\nTo support a new local model in FastChat, you need to correctly handle its prompt template and model loading.\nThe goal is to make the following command run with the correct prompts.\n\n```\npython3 -m fastchat.serve.cli --model [YOUR_MODEL_PATH]\n```\n\nYou can run this example command to learn the code logic.\n\n```\npython3 -m fastchat.serve.cli --model lmsys/vicuna-7b-v1.5\n```\n\nYou can add `--debug` to see the actual prompt sent to the model.\n\n### Steps\n\nFastChat uses the `Conversation` class to handle prompt templates and `BaseModelAdapter` class to handle model loading.\n\n1. Implement a conversation template for the new model at [fastchat/conversation.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py). You can follow existing examples and use `register_conv_template` to add a new one. Please also add a link to the official reference code if possible.\n2. Implement a model adapter for the new model at [fastchat/model/model_adapter.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/model_adapter.py). You can follow existing examples and use `register_model_adapter` to add a new one.\n3. (Optional) add the model name to the \"Supported models\" [section](#supported-models) above and add more information in [fastchat/model/model_registry.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/model_registry.py).\n\nAfter these steps, the new model should be compatible with most FastChat features, such as CLI, web UI, model worker, and OpenAI-compatible API server. Please do some testing with these features as well.\n\n### Supported models\n\n- [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)\n - example: `python3 -m fastchat.serve.cli --model-path meta-llama/Llama-2-7b-chat-hf`\n- Vicuna, Alpaca, LLaMA, Koala\n - example: `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5`\n- [allenai/tulu-2-dpo-7b](https://huggingface.co/allenai/tulu-2-dpo-7b)\n- [BAAI/AquilaChat-7B](https://huggingface.co/BAAI/AquilaChat-7B)\n- [BAAI/AquilaChat2-7B](https://huggingface.co/BAAI/AquilaChat2-7B)\n- [BAAI/AquilaChat2-34B](https://huggingface.co/BAAI/AquilaChat2-34B)\n- [BAAI/bge-large-en](https://huggingface.co/BAAI/bge-large-en#using-huggingface-transformers)\n- [argilla/notus-7b-v1](https://huggingface.co/argilla/notus-7b-v1)\n- [baichuan-inc/baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B)\n- [BlinkDL/RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven)\n - example: `python3 -m fastchat.serve.cli --model-path ~/model_weights/RWKV-4-Raven-7B-v11x-Eng99%-Other1%-20230429-ctx8192.pth`\n- [bofenghuang/vigogne-2-7b-instruct](https://huggingface.co/bofenghuang/vigogne-2-7b-instruct)\n- [bofenghuang/vigogne-2-7b-chat](https://huggingface.co/bofenghuang/vigogne-2-7b-chat)\n- [camel-ai/CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data)\n- [codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf)\n- [databricks/dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b)\n- [deepseek-ai/deepseek-llm-67b-chat](https://huggingface.co/deepseek-ai/deepseek-llm-67b-chat)\n- [deepseek-ai/deepseek-coder-33b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct)\n- [FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat)\n- [FreedomIntelligence/phoenix-inst-chat-7b](https://huggingface.co/FreedomIntelligence/phoenix-inst-chat-7b)\n- [FreedomIntelligence/ReaLM-7b-v1](https://huggingface.co/FreedomIntelligence/Realm-7b)\n- [h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b)\n- [HuggingFaceH4/starchat-beta](https://huggingface.co/HuggingFaceH4/starchat-beta)\n- [HuggingFaceH4/zephyr-7b-alpha](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha)\n- [internlm/internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b)\n- [cllm/consistency-llm-7b-codesearchnet/consistency-llm-7b-gsm8k/consistency-llm-7b-sharegpt48k/consistency-llm-7b-spider](https://huggingface.co/cllm)\n- [IEITYuan/Yuan2-2B/51B/102B-hf](https://huggingface.co/IEITYuan)\n- [lcw99/polyglot-ko-12.8b-chang-instruct-chat](https://huggingface.co/lcw99/polyglot-ko-12.8b-chang-instruct-chat)\n- [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5)\n- [meta-math/MetaMath-7B-V1.0](https://huggingface.co/meta-math/MetaMath-7B-V1.0)\n- [Microsoft/Orca-2-7b](https://huggingface.co/microsoft/Orca-2-7b)\n- [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat)\n - example: `python3 -m fastchat.serve.cli --model-path mosaicml/mpt-7b-chat`\n- [Neutralzz/BiLLa-7B-SFT](https://huggingface.co/Neutralzz/BiLLa-7B-SFT)\n- [nomic-ai/gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy)\n- [NousResearch/Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b)\n- [openaccess-ai-collective/manticore-13b-chat-pyg](https://huggingface.co/openaccess-ai-collective/manticore-13b-chat-pyg)\n- [OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5)\n- [openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5)\n- [Open-Orca/Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca)\n- [OpenLemur/lemur-70b-chat-v1](https://huggingface.co/OpenLemur/lemur-70b-chat-v1)\n- [Phind/Phind-CodeLlama-34B-v2](https://huggingface.co/Phind/Phind-CodeLlama-34B-v2)\n- [project-baize/baize-v2-7b](https://huggingface.co/project-baize/baize-v2-7b)\n- [Qwen/Qwen-7B-Chat](https://huggingface.co/Qwen/Qwen-7B-Chat)\n- [rishiraj/CatPPT](https://huggingface.co/rishiraj/CatPPT)\n- [Salesforce/codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b)\n- [StabilityAI/stablelm-tuned-alpha-7b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b)\n- [tenyx/TenyxChat-7B-v1](https://huggingface.co/tenyx/TenyxChat-7B-v1)\n- [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)\n- [THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)\n- [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)\n- [tiiuae/falcon-40b](https://huggingface.co/tiiuae/falcon-40b)\n- [tiiuae/falcon-180B-chat](https://huggingface.co/tiiuae/falcon-180B-chat)\n- [timdettmers/guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged)\n- [togethercomputer/RedPajama-INCITE-7B-Chat](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat)\n- [VMware/open-llama-7b-v2-open-instruct](https://huggingface.co/VMware/open-llama-7b-v2-open-instruct)\n- [WizardLM/WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0)\n- [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0)\n- [Xwin-LM/Xwin-LM-7B-V0.1](https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1)\n- Any [EleutherAI](https://huggingface.co/EleutherAI) pythia model such as [pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b)\n- Any [Peft](https://github.com/huggingface/peft) adapter trained on top of a\n model above. To activate, must have `peft` in the model path. Note: If\n loading multiple peft models, you can have them share the base model weights by\n setting the environment variable `PEFT_SHARE_BASE_WEIGHTS=true` in any model\n worker.\n\n\n## API-Based Models\nTo support an API-based model, consider learning from the existing OpenAI example.\nIf the model is compatible with OpenAI APIs, then a configuration file is all that's needed without any additional code.\nFor custom protocols, implementation of a streaming generator in [fastchat/serve/api_provider.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/api_provider.py) is required, following the provided examples. Currently, FastChat is compatible with OpenAI, Anthropic, Google Vertex AI, Mistral, Nvidia NGC, YandexGPT and Reka.\n\n### Steps to Launch a WebUI with an API Model\n1. Specify the endpoint information in a JSON configuration file. For instance, create a file named `api_endpoints.json`:\n```json\n{\n \"gpt-3.5-turbo\": {\n \"model_name\": \"gpt-3.5-turbo\",\n \"api_type\": \"openai\",\n \"api_base\": \"https://api.openai.com/v1\",\n \"api_key\": \"sk-******\",\n \"anony_only\": false,\n \"recommended_config\": {\n \"temperature\": 0.7,\n \"top_p\": 1.0\n },\n \"text-arena\": true,\n \"vision-arena\": false,\n }\n}\n```\n - \"api_type\" can be one of the following: openai, anthropic, gemini, mistral, yandexgpt or reka. For custom APIs, add a new type and implement it accordingly.\n - \"anony_only\" indicates whether to display this model in anonymous mode only.\n - \"recommended_config\" indicates the recommended generation parameters for temperature and top_p.\n - \"text-arena\" indicates whether the model should be displayed in the Text Arena.\n - \"vision-arena\" indicates whether the model should be displayed in the Vision Arena.\n\n2. Launch the Gradio web server with the argument `--register api_endpoints.json`:\n```\npython3 -m fastchat.serve.gradio_web_server --controller \"\" --share --register api_endpoints.json\n```\n\nNow, you can open a browser and interact with the model.\n"}, "93ce2999b4031e40e8dc614bd2868f86d684e9fd": {"t": 1785770922.336952, "n": 9278, "h": "734ef38ee4d62f10", "c": "# Model Support\nThis document describes how to support a new model in FastChat.\n\n## Content\n- [Local Models](#local-models)\n- [API-Based Models](#api-based-models)\n\n## Local Models\nTo support a new local model in FastChat, you need to correctly handle its prompt template and model loading.\nThe goal is to make the following command run with the correct prompts.\n\n```\npython3 -m fastchat.serve.cli --model [YOUR_MODEL_PATH]\n```\n\nYou can run this example command to learn the code logic.\n\n```\npython3 -m fastchat.serve.cli --model lmsys/vicuna-7b-v1.5\n```\n\nYou can add `--debug` to see the actual prompt sent to the model.\n\n### Steps\n\nFastChat uses the `Conversation` class to handle prompt templates and `BaseModelAdapter` class to handle model loading.\n\n1. Implement a conversation template for the new model at [fastchat/conversation.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py). You can follow existing examples and use `register_conv_template` to add a new one. Please also add a link to the official reference code if possible.\n2. Implement a model adapter for the new model at [fastchat/model/model_adapter.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/model_adapter.py). You can follow existing examples and use `register_model_adapter` to add a new one.\n3. (Optional) add the model name to the \"Supported models\" [section](#supported-models) above and add more information in [fastchat/model/model_registry.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/model/model_registry.py).\n\nAfter these steps, the new model should be compatible with most FastChat features, such as CLI, web UI, model worker, and OpenAI-compatible API server. Please do some testing with these features as well.\n\n### Supported models\n\n- [meta-llama/Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)\n - example: `python3 -m fastchat.serve.cli --model-path meta-llama/Llama-2-7b-chat-hf`\n- Vicuna, Alpaca, LLaMA, Koala\n - example: `python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5`\n- [allenai/tulu-2-dpo-7b](https://huggingface.co/allenai/tulu-2-dpo-7b)\n- [BAAI/AquilaChat-7B](https://huggingface.co/BAAI/AquilaChat-7B)\n- [BAAI/AquilaChat2-7B](https://huggingface.co/BAAI/AquilaChat2-7B)\n- [BAAI/AquilaChat2-34B](https://huggingface.co/BAAI/AquilaChat2-34B)\n- [BAAI/bge-large-en](https://huggingface.co/BAAI/bge-large-en#using-huggingface-transformers)\n- [argilla/notus-7b-v1](https://huggingface.co/argilla/notus-7b-v1)\n- [baichuan-inc/baichuan-7B](https://huggingface.co/baichuan-inc/baichuan-7B)\n- [BlinkDL/RWKV-4-Raven](https://huggingface.co/BlinkDL/rwkv-4-raven)\n - example: `python3 -m fastchat.serve.cli --model-path ~/model_weights/RWKV-4-Raven-7B-v11x-Eng99%-Other1%-20230429-ctx8192.pth`\n- [bofenghuang/vigogne-2-7b-instruct](https://huggingface.co/bofenghuang/vigogne-2-7b-instruct)\n- [bofenghuang/vigogne-2-7b-chat](https://huggingface.co/bofenghuang/vigogne-2-7b-chat)\n- [camel-ai/CAMEL-13B-Combined-Data](https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data)\n- [codellama/CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf)\n- [databricks/dolly-v2-12b](https://huggingface.co/databricks/dolly-v2-12b)\n- [deepseek-ai/deepseek-llm-67b-chat](https://huggingface.co/deepseek-ai/deepseek-llm-67b-chat)\n- [deepseek-ai/deepseek-coder-33b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct)\n- [FlagAlpha/Llama2-Chinese-13b-Chat](https://huggingface.co/FlagAlpha/Llama2-Chinese-13b-Chat)\n- [FreedomIntelligence/phoenix-inst-chat-7b](https://huggingface.co/FreedomIntelligence/phoenix-inst-chat-7b)\n- [FreedomIntelligence/ReaLM-7b-v1](https://huggingface.co/FreedomIntelligence/Realm-7b)\n- [h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b)\n- [HuggingFaceH4/starchat-beta](https://huggingface.co/HuggingFaceH4/starchat-beta)\n- [HuggingFaceH4/zephyr-7b-alpha](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha)\n- [internlm/internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b)\n- [cllm/consistency-llm-7b-codesearchnet/consistency-llm-7b-gsm8k/consistency-llm-7b-sharegpt48k/consistency-llm-7b-spider](https://huggingface.co/cllm)\n- [IEITYuan/Yuan2-2B/51B/102B-hf](https://huggingface.co/IEITYuan)\n- [lcw99/polyglot-ko-12.8b-chang-instruct-chat](https://huggingface.co/lcw99/polyglot-ko-12.8b-chang-instruct-chat)\n- [lmsys/fastchat-t5-3b-v1.0](https://huggingface.co/lmsys/fastchat-t5)\n- [meta-math/MetaMath-7B-V1.0](https://huggingface.co/meta-math/MetaMath-7B-V1.0)\n- [Microsoft/Orca-2-7b](https://huggingface.co/microsoft/Orca-2-7b)\n- [mosaicml/mpt-7b-chat](https://huggingface.co/mosaicml/mpt-7b-chat)\n - example: `python3 -m fastchat.serve.cli --model-path mosaicml/mpt-7b-chat`\n- [Neutralzz/BiLLa-7B-SFT](https://huggingface.co/Neutralzz/BiLLa-7B-SFT)\n- [nomic-ai/gpt4all-13b-snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy)\n- [NousResearch/Nous-Hermes-13b](https://huggingface.co/NousResearch/Nous-Hermes-13b)\n- [openaccess-ai-collective/manticore-13b-chat-pyg](https://huggingface.co/openaccess-ai-collective/manticore-13b-chat-pyg)\n- [OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5)\n- [openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5)\n- [Open-Orca/Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca)\n- [OpenLemur/lemur-70b-chat-v1](https://huggingface.co/OpenLemur/lemur-70b-chat-v1)\n- [Phind/Phind-CodeLlama-34B-v2](https://huggingface.co/Phind/Phind-CodeLlama-34B-v2)\n- [project-baize/baize-v2-7b](https://huggingface.co/project-baize/baize-v2-7b)\n- [Qwen/Qwen-7B-Chat](https://huggingface.co/Qwen/Qwen-7B-Chat)\n- [rishiraj/CatPPT](https://huggingface.co/rishiraj/CatPPT)\n- [Salesforce/codet5p-6b](https://huggingface.co/Salesforce/codet5p-6b)\n- [StabilityAI/stablelm-tuned-alpha-7b](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b)\n- [tenyx/TenyxChat-7B-v1](https://huggingface.co/tenyx/TenyxChat-7B-v1)\n- [TinyLlama/TinyLlama-1.1B-Chat-v1.0](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)\n- [THUDM/chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)\n- [THUDM/chatglm2-6b](https://huggingface.co/THUDM/chatglm2-6b)\n- [tiiuae/falcon-40b](https://huggingface.co/tiiuae/falcon-40b)\n- [tiiuae/falcon-180B-chat](https://huggingface.co/tiiuae/falcon-180B-chat)\n- [timdettmers/guanaco-33b-merged](https://huggingface.co/timdettmers/guanaco-33b-merged)\n- [togethercomputer/RedPajama-INCITE-7B-Chat](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat)\n- [VMware/open-llama-7b-v2-open-instruct](https://huggingface.co/VMware/open-llama-7b-v2-open-instruct)\n- [WizardLM/WizardLM-13B-V1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0)\n- [WizardLM/WizardCoder-15B-V1.0](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0)\n- [Xwin-LM/Xwin-LM-7B-V0.1](https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1)\n- Any [EleutherAI](https://huggingface.co/EleutherAI) pythia model such as [pythia-6.9b](https://huggingface.co/EleutherAI/pythia-6.9b)\n- Any [Peft](https://github.com/huggingface/peft) adapter trained on top of a\n model above. To activate, must have `peft` in the model path. Note: If\n loading multiple peft models, you can have them share the base model weights by\n setting the environment variable `PEFT_SHARE_BASE_WEIGHTS=true` in any model\n worker.\n\n\n## API-Based Models\nTo support an API-based model, consider learning from the existing OpenAI example.\nIf the model is compatible with OpenAI APIs, then a configuration file is all that's needed without any additional code.\nFor custom protocols, implementation of a streaming generator in [fastchat/serve/api_provider.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/api_provider.py) is required, following the provided examples. Currently, FastChat is compatible with OpenAI, Anthropic, Google Vertex AI, Mistral, Nvidia NGC, YandexGPT and Reka.\n\n### Steps to Launch a WebUI with an API Model\n1. Specify the endpoint information in a JSON configuration file. For instance, create a file named `api_endpoints.json`:\n```json\n{\n \"gpt-3.5-turbo\": {\n \"model_name\": \"gpt-3.5-turbo\",\n \"api_type\": \"openai\",\n \"api_base\": \"https://api.openai.com/v1\",\n \"api_key\": \"sk-******\",\n \"anony_only\": false,\n \"recommended_config\": {\n \"temperature\": 0.7,\n \"top_p\": 1.0\n },\n \"text-arena\": true,\n \"vision-arena\": false,\n }\n}\n```\n - \"api_type\" can be one of the following: openai, anthropic, gemini, mistral, yandexgpt or reka. For custom APIs, add a new type and implement it accordingly.\n - \"anony_only\" indicates whether to display this model in anonymous mode only.\n - \"recommended_config\" indicates the recommended generation parameters for temperature and top_p.\n - \"text-arena\" indicates whether the model should be displayed in the Text Arena.\n - \"vision-arena\" indicates whether the model should be displayed in the Vision Arena.\n\n2. Launch the Gradio web server with the argument `--register api_endpoints.json`:\n```\npython3 -m fastchat.serve.gradio_web_server --controller \"\" --share --register api_endpoints.json\n```\n\nNow, you can open a browser and interact with the model.\n"}}}} |