1 line
38 KiB
JSON
1 line
38 KiB
JSON
{"repo": "amit78kum/litellm--main", "files": {"litellm-main/docs/my-website/docs/providers/ollama.md": {"be99303440183c79aeb97e65f980fafb6455f0f3": {"t": 1785747209.5041351, "n": 18479, "h": "541e008de4217e68", "c": "import Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\n# Ollama \nLiteLLM supports all models from [Ollama](https://github.com/ollama/ollama)\n\n<a target=\"_blank\" href=\"https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_Ollama.ipynb\">\n <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n</a>\n\n:::info \n\nWe recommend using [ollama_chat](#using-ollama-apichat) for better responses.\n\n:::\n\n## Pre-requisites\nEnsure you have your ollama server running\n\n## Example usage\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama/llama2\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n api_base=\"http://localhost:11434\"\n)\nprint(response)\n\n```\n\n## Example usage - Streaming\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama/llama2\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n api_base=\"http://localhost:11434\",\n stream=True\n)\nprint(response)\nfor chunk in response:\n print(chunk['choices'][0]['delta'])\n\n```\n\n## Example usage - Streaming + Acompletion\nEnsure you have async_generator installed for using ollama acompletion with streaming\n```shell\npip install async_generator\n```\n\n```python\nasync def async_ollama():\n response = await litellm.acompletion(\n model=\"ollama/llama2\", \n messages=[{ \"content\": \"what's the weather\" ,\"role\": \"user\"}], \n api_base=\"http://localhost:11434\", \n stream=True\n )\n async for chunk in response:\n print(chunk)\n\n# call async_ollama\nimport asyncio\nasyncio.run(async_ollama())\n\n```\n\n## Example Usage - JSON Mode\nTo use ollama JSON Mode pass `format=\"json\"` to `litellm.completion()`\n\n```python\nfrom litellm import completion\nresponse = completion(\n model=\"ollama/llama2\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": \"respond in json, what's the weather\"\n }\n ],\n max_tokens=10,\n format = \"json\"\n)\n```\n\n## Example Usage - Tool Calling \n\nTo use ollama tool calling, pass `tools=[{..}]` to `litellm.completion()` \n\n<Tabs>\n<TabItem value=\"sdk\" label=\"SDK\">\n\n```python\nfrom litellm import completion\nimport litellm \n\n## [OPTIONAL] REGISTER MODEL - not all ollama models support function calling, litellm defaults to json mode tool calls if native tool calling not supported.\n\n# litellm.register_model(model_cost={\n# \"ollama_chat/llama3.1\": { \n# \"supports_function_calling\": true\n# },\n# })\n\ntools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_current_weather\",\n \"description\": \"Get the current weather in a given location\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\",\n \"description\": \"The city and state, e.g. San Francisco, CA\",\n },\n \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n },\n \"required\": [\"location\"],\n },\n }\n }\n]\n\nmessages = [{\"role\": \"user\", \"content\": \"What's the weather like in Boston today?\"}]\n\n\nresponse = completion(\n model=\"ollama_chat/llama3.1\",\n messages=messages,\n tools=tools\n)\n```\n\n</TabItem>\n<TabItem value=\"proxy\" label=\"PROXY\">\n\n1. Setup config.yaml \n\n```yaml\nmodel_list:\n - model_name: \"llama3.1\" \n litellm_params:\n model: \"ollama_chat/llama3.1\"\n keep_alive: \"8m\" # Optional: Overrides default keep_alive, use -1 for Forever\n model_info:\n supports_function_calling: true\n```\n\n2. Start proxy \n\n```bash\nlitellm --config /path/to/config.yaml\n```\n\n3. Test it! \n\n```bash\ncurl -X POST 'http://0.0.0.0:4000/chat/completions' \\\n-H 'Content-Type: application/json' \\\n-H 'Authorization: Bearer sk-1234' \\\n-d '{\n \"model\": \"llama3.1\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"What'\\''s the weather like in Boston today?\"\n }\n ],\n \"tools\": [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_current_weather\",\n \"description\": \"Get the current weather in a given location\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\",\n \"description\": \"The city and state, e.g. San Francisco, CA\"\n },\n \"unit\": {\n \"type\": \"string\",\n \"enum\": [\"celsius\", \"fahrenheit\"]\n }\n },\n \"required\": [\"location\"]\n }\n }\n }\n ],\n \"tool_choice\": \"auto\",\n \"stream\": true\n}'\n```\n</TabItem>\n</Tabs>\n\n\n## Using Ollama FIM on `/v1/completions`\n\nLiteLLM supports calling Ollama's `/api/generate` endpoint on `/v1/completions` requests. \n\n<Tabs>\n<TabItem value=\"sdk\" label=\"SDK\">\n\n```python\nimport litellm \nlitellm._turn_on_debug() # turn on debug to see the request\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama/llama3.1\",\n prompt=\"Hello, world!\",\n api_base=\"http://localhost:11434\"\n)\nprint(response)\n```\n</TabItem>\n<TabItem value=\"proxy\" label=\"PROXY\">\n\n1. Setup config.yaml \n\n```yaml\nmodel_list:\n - model_name: \"llama3.1\" \n litellm_params:\n model: \"ollama/llama3.1\"\n api_base: \"http://localhost:11434\"\n```\n\n2. Start proxy \n\n```bash\nlitellm --config /path/to/config.yaml --detailed_debug\n\n# RUNNING ON http://0.0.0.0:4000 \n```\n\n3. Test it! \n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"anything\", # 👈 PROXY KEY (can be anything, if master_key not set)\n base_url=\"http://0.0.0.0:4000\" # 👈 PROXY BASE URL\n)\n\nresponse = client.completions.create(\n model=\"ollama/llama3.1\",\n prompt=\"Hello, world!\",\n api_base=\"http://localhost:11434\"\n)\nprint(response)\n```\n</TabItem>\n</Tabs>\n\n## Using ollama `api/chat` \nIn order to send ollama requests to `POST /api/chat` on your ollama server, set the model prefix to `ollama_chat`\n\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama_chat/llama2\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n)\nprint(response)\n```\n## Ollama Models\nOllama supported models: https://github.com/ollama/ollama\n\n| Model Name | Function Call |\n|----------------------|-----------------------------------------------------------------------------------\n| Mistral | `completion(model='ollama/mistral', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Mistral-7B-Instruct-v0.1 | `completion(model='ollama/mistral-7B-Instruct-v0.1', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Mistral-7B-Instruct-v0.2 | `completion(model='ollama/mistral-7B-Instruct-v0.2', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Mixtral-8x7B-Instruct-v0.1 | `completion(model='ollama/mistral-8x7B-Instruct-v0.1', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Mixtral-8x22B-Instruct-v0.1 | `completion(model='ollama/mixtral-8x22B-Instruct-v0.1', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Llama2 7B | `completion(model='ollama/llama2', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 13B | `completion(model='ollama/llama2:13b', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 70B | `completion(model='ollama/llama2:70b', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 Uncensored | `completion(model='ollama/llama2-uncensored', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Code Llama | `completion(model='ollama/codellama', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 Uncensored | `completion(model='ollama/llama2-uncensored', messages, api_base=\"http://localhost:11434\", stream=True)` |\n|Meta LLaMa3 8B | `completion(model='ollama/llama3', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Meta LLaMa3 70B | `completion(model='ollama/llama3:70b', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Orca Mini | `completion(model='ollama/orca-mini', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Vicuna | `completion(model='ollama/vicuna', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Nous-Hermes | `completion(model='ollama/nous-hermes', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Nous-Hermes 13B | `completion(model='ollama/nous-hermes:13b', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Wizard Vicuna Uncensored | `completion(model='ollama/wizard-vicuna', messages, api_base=\"http://localhost:11434\", stream=True)` |\n\n\n### JSON Schema support \n\n<Tabs>\n<TabItem value=\"sdk\" label=\"SDK\">\n\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama_chat/deepseek-r1\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n response_format={\"type\": \"json_schema\", \"json_schema\": {\"schema\": {\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"}}}}},\n)\nprint(response)\n```\n</TabItem>\n<TabItem value=\"proxy\" label=\"PROXY\">\n\n1. Setup config.yaml \n\n```yaml\nmodel_list:\n - model_name: \"deepseek-r1\" \n litellm_params:\n model: \"ollama_chat/deepseek-r1\"\n api_base: \"http://localhost:11434\"\n```\n\n2. Start proxy \n\n```bash\nlitellm --config /path/to/config.yaml\n\n# RUNNING ON http://0.0.0.0:4000\n```\n\n3. Test it! \n\n```python\nfrom pydantic import BaseModel\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"anything\", # 👈 PROXY KEY (can be anything, if master_key not set)\n base_url=\"http://0.0.0.0:4000\" # 👈 PROXY BASE URL\n)\n\nclass Step(BaseModel):\n explanation: str\n output: str\n\nclass MathReasoning(BaseModel):\n steps: list[Step]\n final_answer: str\n\ncompletion = client.beta.chat.completions.parse(\n model=\"deepseek-r1\",\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful math tutor. Guide the user through the solution step by step.\"},\n {\"role\": \"user\", \"content\": \"how can I solve 8x + 7 = -23\"}\n ],\n response_format=MathReasoning,\n)\n\nmath_reasoning = completion.choices[0].message.parsed\n```\n</TabItem>\n</Tabs>\n\n## Ollama Vision Models\n| Model Name | Function Call |\n|------------------|--------------------------------------|\n| llava | `completion('ollama/llava', messages)` |\n\n#### Using Ollama Vision Models\n\nCall `ollama/llava` in the same input/output format as OpenAI [`gpt-4-vision`](https://docs.litellm.ai/docs/providers/openai#openai-vision-models)\n\nLiteLLM Supports the following image types passed in `url`\n- Base64 encoded svgs\n\n**Example Request**\n```python\nimport litellm\n\nresponse = litellm.completion(\n model = \"ollama/llava\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": \"Whats in this image?\"\n },\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": \"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"\n }\n }\n ]\n }\n ],\n)\nprint(response)\n```\n\n\n\n## LiteLLM/Ollama Docker Image \n\nFor Ollama LiteLLM Provides a Docker Image for an OpenAI API compatible server for local LLMs - llama2, mistral, codellama\n\n\n[](https://wa.link/huol9n) [](https://discord.gg/wuPM9dRgDw) \n### An OpenAI API compatible server for local LLMs - llama2, mistral, codellama\n\n### Quick Start:\nDocker Hub: \nFor ARM Processors: https://hub.docker.com/repository/docker/litellm/ollama/general\nFor Intel/AMD Processors: to be added\n```shell\ndocker pull litellm/ollama\n```\n\n```shell\ndocker run --name ollama litellm/ollama\n```\n\n#### Test the server container\nOn the docker container run the `test.py` file using `python3 test.py`\n\n\n### Making a request to this server\n```python\nimport openai\n\napi_base = f\"http://0.0.0.0:4000\" # base url for server\n\nopenai.api_base = api_base\nopenai.api_key = \"temp-key\"\nprint(openai.api_base)\n\n\nprint(f'LiteLLM: response from proxy with streaming')\nresponse = openai.chat.completions.create(\n model=\"ollama/llama2\", \n messages = [\n {\n \"role\": \"user\",\n \"content\": \"this is a test request, acknowledge that you got it\"\n }\n ],\n stream=True\n)\n\nfor chunk in response:\n print(f'LiteLLM: streaming response from proxy {chunk}')\n```\n\n### Responses from this server \n```json\n{\n \"object\": \"chat.completion\",\n \"choices\": [\n {\n \"finish_reason\": \"stop\",\n \"index\": 0,\n \"message\": {\n \"content\": \" Hello! I acknowledge receipt of your test request. Please let me know if there's anything else I can assist you with.\",\n \"role\": \"assistant\",\n \"logprobs\": null\n }\n }\n ],\n \"id\": \"chatcmpl-403d5a85-2631-4233-92cb-01e6dffc3c39\",\n \"created\": 1696992706.619709,\n \"model\": \"ollama/llama2\",\n \"usage\": {\n \"prompt_tokens\": 18,\n \"completion_tokens\": 25,\n \"total_tokens\": 43\n }\n}\n```\n\n## Calling Docker Container (host.docker.internal)\n\n[Follow these instructions](https://github.com/BerriAI/litellm/issues/1517#issuecomment-1922022209/)\n"}, "8feb21faeb389f32c8359ebcd48a1c48619bcde8": {"t": 1785771384.0923574, "n": 18479, "h": "541e008de4217e68", "c": "import Tabs from '@theme/Tabs';\nimport TabItem from '@theme/TabItem';\n\n# Ollama \nLiteLLM supports all models from [Ollama](https://github.com/ollama/ollama)\n\n<a target=\"_blank\" href=\"https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_Ollama.ipynb\">\n <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n</a>\n\n:::info \n\nWe recommend using [ollama_chat](#using-ollama-apichat) for better responses.\n\n:::\n\n## Pre-requisites\nEnsure you have your ollama server running\n\n## Example usage\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama/llama2\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n api_base=\"http://localhost:11434\"\n)\nprint(response)\n\n```\n\n## Example usage - Streaming\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama/llama2\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n api_base=\"http://localhost:11434\",\n stream=True\n)\nprint(response)\nfor chunk in response:\n print(chunk['choices'][0]['delta'])\n\n```\n\n## Example usage - Streaming + Acompletion\nEnsure you have async_generator installed for using ollama acompletion with streaming\n```shell\npip install async_generator\n```\n\n```python\nasync def async_ollama():\n response = await litellm.acompletion(\n model=\"ollama/llama2\", \n messages=[{ \"content\": \"what's the weather\" ,\"role\": \"user\"}], \n api_base=\"http://localhost:11434\", \n stream=True\n )\n async for chunk in response:\n print(chunk)\n\n# call async_ollama\nimport asyncio\nasyncio.run(async_ollama())\n\n```\n\n## Example Usage - JSON Mode\nTo use ollama JSON Mode pass `format=\"json\"` to `litellm.completion()`\n\n```python\nfrom litellm import completion\nresponse = completion(\n model=\"ollama/llama2\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": \"respond in json, what's the weather\"\n }\n ],\n max_tokens=10,\n format = \"json\"\n)\n```\n\n## Example Usage - Tool Calling \n\nTo use ollama tool calling, pass `tools=[{..}]` to `litellm.completion()` \n\n<Tabs>\n<TabItem value=\"sdk\" label=\"SDK\">\n\n```python\nfrom litellm import completion\nimport litellm \n\n## [OPTIONAL] REGISTER MODEL - not all ollama models support function calling, litellm defaults to json mode tool calls if native tool calling not supported.\n\n# litellm.register_model(model_cost={\n# \"ollama_chat/llama3.1\": { \n# \"supports_function_calling\": true\n# },\n# })\n\ntools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_current_weather\",\n \"description\": \"Get the current weather in a given location\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\",\n \"description\": \"The city and state, e.g. San Francisco, CA\",\n },\n \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n },\n \"required\": [\"location\"],\n },\n }\n }\n]\n\nmessages = [{\"role\": \"user\", \"content\": \"What's the weather like in Boston today?\"}]\n\n\nresponse = completion(\n model=\"ollama_chat/llama3.1\",\n messages=messages,\n tools=tools\n)\n```\n\n</TabItem>\n<TabItem value=\"proxy\" label=\"PROXY\">\n\n1. Setup config.yaml \n\n```yaml\nmodel_list:\n - model_name: \"llama3.1\" \n litellm_params:\n model: \"ollama_chat/llama3.1\"\n keep_alive: \"8m\" # Optional: Overrides default keep_alive, use -1 for Forever\n model_info:\n supports_function_calling: true\n```\n\n2. Start proxy \n\n```bash\nlitellm --config /path/to/config.yaml\n```\n\n3. Test it! \n\n```bash\ncurl -X POST 'http://0.0.0.0:4000/chat/completions' \\\n-H 'Content-Type: application/json' \\\n-H 'Authorization: Bearer sk-1234' \\\n-d '{\n \"model\": \"llama3.1\",\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"What'\\''s the weather like in Boston today?\"\n }\n ],\n \"tools\": [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_current_weather\",\n \"description\": \"Get the current weather in a given location\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"location\": {\n \"type\": \"string\",\n \"description\": \"The city and state, e.g. San Francisco, CA\"\n },\n \"unit\": {\n \"type\": \"string\",\n \"enum\": [\"celsius\", \"fahrenheit\"]\n }\n },\n \"required\": [\"location\"]\n }\n }\n }\n ],\n \"tool_choice\": \"auto\",\n \"stream\": true\n}'\n```\n</TabItem>\n</Tabs>\n\n\n## Using Ollama FIM on `/v1/completions`\n\nLiteLLM supports calling Ollama's `/api/generate` endpoint on `/v1/completions` requests. \n\n<Tabs>\n<TabItem value=\"sdk\" label=\"SDK\">\n\n```python\nimport litellm \nlitellm._turn_on_debug() # turn on debug to see the request\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama/llama3.1\",\n prompt=\"Hello, world!\",\n api_base=\"http://localhost:11434\"\n)\nprint(response)\n```\n</TabItem>\n<TabItem value=\"proxy\" label=\"PROXY\">\n\n1. Setup config.yaml \n\n```yaml\nmodel_list:\n - model_name: \"llama3.1\" \n litellm_params:\n model: \"ollama/llama3.1\"\n api_base: \"http://localhost:11434\"\n```\n\n2. Start proxy \n\n```bash\nlitellm --config /path/to/config.yaml --detailed_debug\n\n# RUNNING ON http://0.0.0.0:4000 \n```\n\n3. Test it! \n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"anything\", # 👈 PROXY KEY (can be anything, if master_key not set)\n base_url=\"http://0.0.0.0:4000\" # 👈 PROXY BASE URL\n)\n\nresponse = client.completions.create(\n model=\"ollama/llama3.1\",\n prompt=\"Hello, world!\",\n api_base=\"http://localhost:11434\"\n)\nprint(response)\n```\n</TabItem>\n</Tabs>\n\n## Using ollama `api/chat` \nIn order to send ollama requests to `POST /api/chat` on your ollama server, set the model prefix to `ollama_chat`\n\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama_chat/llama2\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n)\nprint(response)\n```\n## Ollama Models\nOllama supported models: https://github.com/ollama/ollama\n\n| Model Name | Function Call |\n|----------------------|-----------------------------------------------------------------------------------\n| Mistral | `completion(model='ollama/mistral', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Mistral-7B-Instruct-v0.1 | `completion(model='ollama/mistral-7B-Instruct-v0.1', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Mistral-7B-Instruct-v0.2 | `completion(model='ollama/mistral-7B-Instruct-v0.2', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Mixtral-8x7B-Instruct-v0.1 | `completion(model='ollama/mistral-8x7B-Instruct-v0.1', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Mixtral-8x22B-Instruct-v0.1 | `completion(model='ollama/mixtral-8x22B-Instruct-v0.1', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Llama2 7B | `completion(model='ollama/llama2', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 13B | `completion(model='ollama/llama2:13b', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 70B | `completion(model='ollama/llama2:70b', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 Uncensored | `completion(model='ollama/llama2-uncensored', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Code Llama | `completion(model='ollama/codellama', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Llama2 Uncensored | `completion(model='ollama/llama2-uncensored', messages, api_base=\"http://localhost:11434\", stream=True)` |\n|Meta LLaMa3 8B | `completion(model='ollama/llama3', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Meta LLaMa3 70B | `completion(model='ollama/llama3:70b', messages, api_base=\"http://localhost:11434\", stream=False)` |\n| Orca Mini | `completion(model='ollama/orca-mini', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Vicuna | `completion(model='ollama/vicuna', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Nous-Hermes | `completion(model='ollama/nous-hermes', messages, api_base=\"http://localhost:11434\", stream=True)` |\n| Nous-Hermes 13B | `completion(model='ollama/nous-hermes:13b', messages, api_base=\"http://localhost:11434\", stream=True)` | \n| Wizard Vicuna Uncensored | `completion(model='ollama/wizard-vicuna', messages, api_base=\"http://localhost:11434\", stream=True)` |\n\n\n### JSON Schema support \n\n<Tabs>\n<TabItem value=\"sdk\" label=\"SDK\">\n\n```python\nfrom litellm import completion\n\nresponse = completion(\n model=\"ollama_chat/deepseek-r1\", \n messages=[{ \"content\": \"respond in 20 words. who are you?\",\"role\": \"user\"}], \n response_format={\"type\": \"json_schema\", \"json_schema\": {\"schema\": {\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"}}}}},\n)\nprint(response)\n```\n</TabItem>\n<TabItem value=\"proxy\" label=\"PROXY\">\n\n1. Setup config.yaml \n\n```yaml\nmodel_list:\n - model_name: \"deepseek-r1\" \n litellm_params:\n model: \"ollama_chat/deepseek-r1\"\n api_base: \"http://localhost:11434\"\n```\n\n2. Start proxy \n\n```bash\nlitellm --config /path/to/config.yaml\n\n# RUNNING ON http://0.0.0.0:4000\n```\n\n3. Test it! \n\n```python\nfrom pydantic import BaseModel\nfrom openai import OpenAI\n\nclient = OpenAI(\n api_key=\"anything\", # 👈 PROXY KEY (can be anything, if master_key not set)\n base_url=\"http://0.0.0.0:4000\" # 👈 PROXY BASE URL\n)\n\nclass Step(BaseModel):\n explanation: str\n output: str\n\nclass MathReasoning(BaseModel):\n steps: list[Step]\n final_answer: str\n\ncompletion = client.beta.chat.completions.parse(\n model=\"deepseek-r1\",\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful math tutor. Guide the user through the solution step by step.\"},\n {\"role\": \"user\", \"content\": \"how can I solve 8x + 7 = -23\"}\n ],\n response_format=MathReasoning,\n)\n\nmath_reasoning = completion.choices[0].message.parsed\n```\n</TabItem>\n</Tabs>\n\n## Ollama Vision Models\n| Model Name | Function Call |\n|------------------|--------------------------------------|\n| llava | `completion('ollama/llava', messages)` |\n\n#### Using Ollama Vision Models\n\nCall `ollama/llava` in the same input/output format as OpenAI [`gpt-4-vision`](https://docs.litellm.ai/docs/providers/openai#openai-vision-models)\n\nLiteLLM Supports the following image types passed in `url`\n- Base64 encoded svgs\n\n**Example Request**\n```python\nimport litellm\n\nresponse = litellm.completion(\n model = \"ollama/llava\",\n messages=[\n {\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": \"Whats in this image?\"\n },\n {\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": \"iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tnT58+dXXCJDdECBE2Ojrqjh071hpNECjx4cMHVycM1Uhbv359B2F79+51586daxN/+pyRkRFXKyRDAqxEp4yMlDDzXG1NPnnyJKkThoK0VFd1ELZu3TrzXKxKfW7dMBQ6bcuWLW2v0VlHjx41z717927ba22U9APcw7Nnz1oGEPeL3m3p2mTAYYnFmMOMXybPPXv2bNIPpFZr1NHn4HMw0KRBjg9NuRw95s8PEcz/6DZELQd/09C9QGq5RsmSRybqkwHGjh07OsJSsYYm3ijPpyHzoiacg35MLdDSIS/O1yM778jOTwYUkKNHWUzUWaOsylE00MyI0fcnOwIdjvtNdW/HZwNLGg+sR1kMepSNJXmIwxBZiG8tDTpEZzKg0GItNsosY8USkxDhD0Rinuiko2gfL/RbiD2LZAjU9zKQJj8RDR0vJBR1/Phx9+PHj9Z7REF4nTZkxzX4LCXHrV271qXkBAPGfP/atWvu/PnzHe4C97F48eIsRLZ9+3a3f/9+87dwP1JxaF7/3r17ba+5l4EcaVo0lj3SBq5kGTJSQmLWMjgYNei2GPT1MuMqGTDEFHzeQSP2wi/jGnkmPJ/nhccs44jvDAxpVcxnq0F6eT8h4ni/iIWpR5lPyA6ETkNXoSukvpJAD3AsXLiwpZs49+fPn5ke4j10TqYvegSfn0OnafC+Tv9ooA/JPkgQysqQNBzagXY55nO/oa1F7qvIPWkRL12WRpMWUvpVDYmxAPehxWSe8ZEXL20sadYIozfmNch4QJPAfeJgW3rNsnzphBKNJM2KKODo1rVOMRYik5ETy3ix4qWNI81qAAirizgMIc+yhTytx0JWZuNI03qsrgWlGtwjoS9XwgUhWGyhUaRZZQNNIEwCiXD16tXcAHUs79co0vSD8rrJCIW98pzvxpAWyyo3HYwqS0+H0BjStClcZJT5coMm6D2LOF8TolGJtK9fvyZpyiC5ePFi9nc/oJU4eiEP0jVoAnHa9wyJycITMP78+eMeP37sXrx44d6+fdt6f82aNdkx1pg9e3Zb5W+RSRE+n+VjksQWifvVaTKFhn5O8my63K8Qabdv33b379/PiAP//vuvW7BggZszZ072/+TJk91YgkafPn166zXB1rQHFvouAWHq9z3SEevSUerqCn2/dDCeta2jxYbr69evk4MHDyY7d+7MjhMnTiTPnz9Pfv/+nfQT2ggpO2dMF8cghuoM7Ygj5iWCqRlGFml0QC/ftGmTmzt3rmsaKDsgBSPh0/8yPeLLBihLkOKJc0jp8H8vUzcxIA1k6QJ/c78tWEyj5P3o4u9+jywNPdJi5rAH9x0KHcl4Hg570eQp3+vHXGyrmEeigzQsQsjavXt38ujRo44LQuDDhw+TW7duRS1HGgMxhNXHgflaNTOsHyKvHK5Ijo2jbFjJBQK9YwFd6RVMzfgRBmEfP37suBBm/p49e1qjEP2mwTViNRo0VJWH1deMXcNK08uUjVUu7s/zRaL+oLNxz1bpANco4npUgX4G2eFbpDFyQoQxojBCpEGSytmOH8qrH5Q9vuzD6ofQylkCUmh8DBAr+q8JCyVNtWQIidKQE9wNtLSQnS4jDSsxNHogzFuQBw4cyM61UKVsjfr3ooBkPSqqQHesUPWVtzi9/vQi1T+rJj7WiTz4Pt/l3LxUkr5P2VYZaZ4URpsE+st/dujQoaBBYokbrz/8TJNQYLSonrPS9kUaSkPeZyj1AWSj+d+VBoy1pIWVNed8P0Ll/ee5HdGRhrHhR5GGN0r4LGZBaj8oFDJitBTJzIZgFcmU0Y8ytWMZMzJOaXUSrUs5RxKnrxmbb5YXO9VGUhtpXldhEUogFr3IzIsvlpmdosVcGVGXFWp2oU9kLFL3dEkSz6NHEY1sjSRdIuDFWEhd8KxFqsRi1uM/nz9/zpxnwlESONdg6dKlbsaMGS4EHFHtjFIDHwKOo46l4TxSuxgDzi+rE2jg+BaFruOX4HXa0Nnf1lwAPufZeF8/r6zD97WK2qFnGjBxTw5qNGPxT+5T/r7/7RawFC3j4vTp09koCxkeHjqbHJqArmH5UrFKKksnxrK7FuRIs8STfBZv+luugXZ2pR/pP9Ois4z+TiMzUUkUjD0iEi1fzX8GmXyuxUBRcaUfykV0YZnlJGKQpOiGB76x5GeWkWWJc3mOrK6S7xdND+W5N6XyaRgtWJFe13GkaZnKOsYqGdOVVVbGupsyA/l7emTLHi7vwTdirNEt0qxnzAvBFcnQF16xh/TMpUuXHDowhlA9vQVraQhkudRdzOnK+04ZSP3DUhVSP61YsaLtd/ks7ZgtPcXqPqEafHkdqa84X6aCeL7YWlv6edGFHb+ZFICPlljHhg0bKuk0CSvVznWsotRu433alNdFrqG45ejoaPCaUkWERpLXjzFL2Rpllp7PJU2a/v7Ab8N05/9t27Z16KUqoFGsxnI9EosS2niSYg9SpU6B4JgTrvVW1flt1sT+0ADIJU2maXzcUTraGCRaL1Wp9rUMk16PMom8QhruxzvZIegJjFU7LLCePfS8uaQdPny4jTTL0dbee5mYokQsXTIWNY46kuMbnt8Kmec+LGWtOVIl9cT1rCB0V8WqkjAsRwta93TbwNYoGKsUSChN44lgBNCoHLHzquYKrU6qZ8lolCIN0Rh6cP0Q3U6I6IXILYOQI513hJaSKAorFpuHXJNfVlpRtmYBk1Su1obZr5dnKAO+L10Hrj3WZW+E3qh6IszE37F6EB+68mGpvKm4eb9bFrlzrok7fvr0Kfv727dvWRmdVTJHw0qiiCUSZ6wCK+7XL/AcsgNyL74DQQ730sv78Su7+t/A36MdY0sW5o40ahslXr58aZ5HtZB8GH64m9EmMZ7FpYw4T6QnrZfgenrhFxaSiSGXtPnz57e9TkNZLvTjeqhr734CNtrK41L40sUQckmj1lGKQ0rC37x544r8eNXRpnVE3ZZY7zXo8NomiO0ZUCj2uHz58rbXoZ6gc0uA+F6ZeKS/jhRDUq8MKrTho9fEkihMmhxtBI1DxKFY9XLpVcSkfoi8JGnToZO5sU5aiDQIW716ddt7ZLYtMQlhECdBGXZZMWldY5BHm5xgAroWj4C0hbYkSc/jBmggIrXJWlZM6pSETsEPGqZOndr2uuuR5rF169a2HoHPdurUKZM4CO1WTPqaDaAd+GFGKdIQkxAn9RuEWcTRyN2KSUgiSgF5aWzPTeA/lN5rZubMmR2bE4SIC4nJoltgAV/dVefZm72AtctUCJU2CMJ327hxY9t7EHbkyJFseq+EJSY16RPo3Dkq1kkr7+q0bNmyDuLQcZBEPYmHVdOBiJyIlrRDq41YPWfXOxUysi5fvtyaj+2BpcnsUV/oSoEMOk2CQGlr4ckhBwaetBhjCwH0ZHtJROPJkyc7UjcYLDjmrH7ADTEBXFfOYmB0k9oYBOjJ8b4aOYSe7QkKcYhFlq3QYLQhSidNmtS2RATwy8YOM3EQJsUjKiaWZ+vZToUQgzhkHXudb/PW5YMHD9yZM2faPsMwoc7RciYJXbGuBqJ1UIGKKLv915jsvgtJxCZDubdXr165mzdvtr1Hz5LONA8jrUwKPqsmVesKa49S3Q4WxmRPUEYdTjgiUcfUwLx589ySJUva3oMkP6IYddq6HMS4o55xBJBUeRjzfa4Zdeg56QZ43LhxoyPo7Lf1kNt7oO8wWAbNwaYjIv5lhyS7kRf96dvm5Jah8vfvX3flyhX35cuX6HfzFHOToS1H4BenCaHvO8pr8iDuwoUL7tevX+b5ZdbBair0xkFIlFDlW4ZknEClsp/TzXyAKVOmmHWFVSbDNw1l1+4f90U6IY/q4V27dpnE9bJ+v87QEydjqx/UamVVPRG+mwkNTYN+9tjkwzEx+atCm/X9WvWtDtAb68Wy9LXa1UmvCDDIpPkyOQ5ZwSzJ4jMrvFcr0rSjOUh+GcT4LSg5ugkW1Io0/SCDQBojh0hPlaJdah+tkVYrnTZowP8iq1F1TgMBBauufyB33x1v+NWFYmT5KmppgHC+NkAgbmRkpD3yn9QIseXymoTQFGQmIOKTxiZIWpvAatenVqRVXf2nTrAWMsPnKrMZHz6bJq5jvce6QK8J1cQNgKxlJapMPdZSR64/UivS9NztpkVEdKcrs5alhhWP9NeqlfWopzhZScI6QxseegZRGeg5a8C3Re1Mfl1ScP36ddcUaMuv24iOJtz7sbUjTS4qBvKmstYJoUauiuD3k5qhyr7QdUHMeCgLa1Ear9NquemdXgmum4fvJ6w1lqsuDhNrg1qSpleJK7K3TF0Q2jSd94uSZ60kK1e3qyVpQK6PVWXp2/FC3mp6jBhKKOiY2h3gtUV64TWM6wDETRPLDfSakXmH3w8g9Jlug8ZtTt4kVF0kLUYYmCCtD/DrQ5YhMGbA9L3ucdjh0y8kOHW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC\"\n }\n }\n ]\n }\n ],\n)\nprint(response)\n```\n\n\n\n## LiteLLM/Ollama Docker Image \n\nFor Ollama LiteLLM Provides a Docker Image for an OpenAI API compatible server for local LLMs - llama2, mistral, codellama\n\n\n[](https://wa.link/huol9n) [](https://discord.gg/wuPM9dRgDw) \n### An OpenAI API compatible server for local LLMs - llama2, mistral, codellama\n\n### Quick Start:\nDocker Hub: \nFor ARM Processors: https://hub.docker.com/repository/docker/litellm/ollama/general\nFor Intel/AMD Processors: to be added\n```shell\ndocker pull litellm/ollama\n```\n\n```shell\ndocker run --name ollama litellm/ollama\n```\n\n#### Test the server container\nOn the docker container run the `test.py` file using `python3 test.py`\n\n\n### Making a request to this server\n```python\nimport openai\n\napi_base = f\"http://0.0.0.0:4000\" # base url for server\n\nopenai.api_base = api_base\nopenai.api_key = \"temp-key\"\nprint(openai.api_base)\n\n\nprint(f'LiteLLM: response from proxy with streaming')\nresponse = openai.chat.completions.create(\n model=\"ollama/llama2\", \n messages = [\n {\n \"role\": \"user\",\n \"content\": \"this is a test request, acknowledge that you got it\"\n }\n ],\n stream=True\n)\n\nfor chunk in response:\n print(f'LiteLLM: streaming response from proxy {chunk}')\n```\n\n### Responses from this server \n```json\n{\n \"object\": \"chat.completion\",\n \"choices\": [\n {\n \"finish_reason\": \"stop\",\n \"index\": 0,\n \"message\": {\n \"content\": \" Hello! I acknowledge receipt of your test request. Please let me know if there's anything else I can assist you with.\",\n \"role\": \"assistant\",\n \"logprobs\": null\n }\n }\n ],\n \"id\": \"chatcmpl-403d5a85-2631-4233-92cb-01e6dffc3c39\",\n \"created\": 1696992706.619709,\n \"model\": \"ollama/llama2\",\n \"usage\": {\n \"prompt_tokens\": 18,\n \"completion_tokens\": 25,\n \"total_tokens\": 43\n }\n}\n```\n\n## Calling Docker Container (host.docker.internal)\n\n[Follow these instructions](https://github.com/BerriAI/litellm/issues/1517#issuecomment-1922022209/)\n"}}}} |