1 line
93 KiB
JSON
1 line
93 KiB
JSON
{"repo": "alexsidebar/docs", "files": {"configuration/litellm-setup.mdx": {"ed31fcb6bfd5586cc96127b0d00a574330443a64": {"t": 1785747144.6350427, "n": 18061, "h": "648e6eef1c287911", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n<Info>\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n</Info>\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n<Note>\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n</Note>\n\n## Why Use LiteLLM?\n\n<CardGroup cols={2}>\n <Card title=\"Use Your Company's AI\" icon=\"building\">\n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n </Card>\n \n <Card title=\"Data Never Leaves Your Infrastructure\" icon=\"shield\">\n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n </Card>\n \n <Card title=\"Track Costs by Project\" icon=\"chart-line\">\n See exactly how much each project costs. Set budgets and get alerts\n </Card>\n \n <Card title=\"One Interface for All Models\" icon=\"plug\">\n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n </Card>\n</CardGroup>\n\n## Quick Start\n\n<Steps>\n <Step title=\"Install LiteLLM\">\n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n </Step>\n \n <Step title=\"Configure Your Providers\">\n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n api_key: \"your-openai-key\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n </Step>\n \n <Step title=\"Connect Alex Sidebar\">\n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n </Step>\n</Steps>\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n </Tab>\n</Tabs>\n\n### Google Vertex AI\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n </Tab>\n</Tabs>\n\n### Azure OpenAI\n\n<Tabs>\n <Tab title=\"Setup\">\n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"azure/o4-mini-2025-04-16\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro-2025-06-10\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n </Tab>\n</Tabs>\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n \n - model_name: \"o3-pro-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n reasoning_effort: \"high\"\n \n - model_name: \"o4-mini-reasoning\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n reasoning_effort: \"medium\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n<Tip>\n All AI requests from your team go through your infrastructure. You control the data and costs.\n</Tip>\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\", \"o4-mini-2025-04-16\", \"o3-pro-2025-06-10\"]\n```\n\n## Troubleshooting\n\n<AccordionGroup>\n <Accordion title=\"Connection refused error\">\n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n </Accordion>\n \n <Accordion title=\"Authentication errors\">\n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n </Accordion>\n \n <Accordion title=\"Model not found or deprecated\">\n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n </Accordion>\n \n <Accordion title=\"High latency or rate limiting\">\n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n </Accordion>\n \n <Accordion title=\"Cost tracking issues\">\n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n </Accordion>\n</AccordionGroup>\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n thinking: true\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact team@alexcodes.app for business support and enterprise features\n\n<Note>\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n</Note>"}, "c53c861da0d860ec0936ec60f566e7963806d8cb": {"t": 1785770700.0183918, "n": 18063, "h": "f590f1c9b870d4c9", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n<Info>\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n</Info>\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n<Note>\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n</Note>\n\n## Why Use LiteLLM?\n\n<CardGroup cols={2}>\n <Card title=\"Use Your Company's AI\" icon=\"building\">\n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n </Card>\n \n <Card title=\"Data Never Leaves Your Infrastructure\" icon=\"shield\">\n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n </Card>\n \n <Card title=\"Track Costs by Project\" icon=\"chart-line\">\n See exactly how much each project costs. Set budgets and get alerts\n </Card>\n \n <Card title=\"One Interface for All Models\" icon=\"plug\">\n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n </Card>\n</CardGroup>\n\n## Quick Start\n\n<Steps>\n <Step title=\"Install LiteLLM\">\n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n </Step>\n \n <Step title=\"Configure Your Providers\">\n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n api_key: \"your-openai-key\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n </Step>\n \n <Step title=\"Connect Alex Sidebar\">\n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n </Step>\n</Steps>\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n </Tab>\n</Tabs>\n\n### Google Vertex AI\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n </Tab>\n</Tabs>\n\n### Azure OpenAI\n\n<Tabs>\n <Tab title=\"Setup\">\n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"azure/o4-mini-2025-04-16\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro-2025-06-10\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n </Tab>\n</Tabs>\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n \n - model_name: \"o3-pro-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n reasoning_effort: \"high\"\n \n - model_name: \"o4-mini-reasoning\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n reasoning_effort: \"medium\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n<Tip>\n All AI requests from your team go through your infrastructure. You control the data and costs.\n</Tip>\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\", \"o4-mini-2025-04-16\", \"o3-pro-2025-06-10\"]\n```\n\n## Troubleshooting\n\n<AccordionGroup>\n <Accordion title=\"Connection refused error\">\n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n </Accordion>\n \n <Accordion title=\"Authentication errors\">\n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n </Accordion>\n \n <Accordion title=\"Model not found or deprecated\">\n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n </Accordion>\n \n <Accordion title=\"High latency or rate limiting\">\n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n </Accordion>\n \n <Accordion title=\"Cost tracking issues\">\n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n </Accordion>\n</AccordionGroup>\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n thinking: true\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support and enterprise features\n\n<Note>\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n</Note>"}, "ddfbdcd79953d4ace54ffb1a19a2306b72646247": {"t": 1785770701.3612468, "n": 18179, "h": "2b99109ecbe55dfa", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n<Info>\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n</Info>\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n<Note>\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n</Note>\n\n## Why Use LiteLLM?\n\n<CardGroup cols={2}>\n <Card title=\"Use Your Company's AI\" icon=\"building\">\n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n </Card>\n \n <Card title=\"Data Never Leaves Your Infrastructure\" icon=\"shield\">\n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n </Card>\n \n <Card title=\"Track Costs by Project\" icon=\"chart-line\">\n See exactly how much each project costs. Set budgets and get alerts\n </Card>\n \n <Card title=\"One Interface for All Models\" icon=\"plug\">\n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n </Card>\n</CardGroup>\n\n## Quick Start\n\n<Steps>\n <Step title=\"Install LiteLLM\">\n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n </Step>\n \n <Step title=\"Configure Your Providers\">\n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n api_key: \"your-openai-key\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n </Step>\n \n <Step title=\"Connect Alex Sidebar\">\n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n </Step>\n</Steps>\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true # Enable reasoning capabilities\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n </Tab>\n</Tabs>\n\n### Google Vertex AI\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n </Tab>\n</Tabs>\n\n### Azure OpenAI\n\n<Tabs>\n <Tab title=\"Setup\">\n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"azure/o4-mini-2025-04-16\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro-2025-06-10\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n </Tab>\n</Tabs>\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n reasoning_effort: \"medium\" # low, medium, high\n \n - model_name: \"o3-pro-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n reasoning_effort: \"high\"\n \n - model_name: \"o4-mini-reasoning\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n reasoning_effort: \"medium\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n<Tip>\n All AI requests from your team go through your infrastructure. You control the data and costs.\n</Tip>\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\", \"o4-mini-2025-04-16\", \"o3-pro-2025-06-10\"]\n```\n\n## Troubleshooting\n\n<AccordionGroup>\n <Accordion title=\"Connection refused error\">\n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n </Accordion>\n \n <Accordion title=\"Authentication errors\">\n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n </Accordion>\n \n <Accordion title=\"Model not found or deprecated\">\n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n </Accordion>\n \n <Accordion title=\"High latency or rate limiting\">\n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n </Accordion>\n \n <Accordion title=\"Cost tracking issues\">\n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n </Accordion>\n</AccordionGroup>\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n thinking: true\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n reasoning_effort: \"high\"\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support and enterprise features\n\n<Note>\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n</Note>"}, "04cc0c4608a736f8ac7f06b9350d061929f4c3c8": {"t": 1785770703.0079153, "n": 17532, "h": "15840335d36a6439", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n<Info>\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n</Info>\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n<Note>\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n</Note>\n\n## Why Use LiteLLM?\n\n<CardGroup cols={2}>\n <Card title=\"Use Your Company's AI\" icon=\"building\">\n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n </Card>\n \n <Card title=\"Data Never Leaves Your Infrastructure\" icon=\"shield\">\n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n </Card>\n \n <Card title=\"Track Costs by Project\" icon=\"chart-line\">\n See exactly how much each project costs. Set budgets and get alerts\n </Card>\n \n <Card title=\"One Interface for All Models\" icon=\"plug\">\n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n </Card>\n</CardGroup>\n\n## Quick Start\n\n<Steps>\n <Step title=\"Install LiteLLM\">\n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n </Step>\n \n <Step title=\"Configure Your Providers\">\n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n </Step>\n \n <Step title=\"Connect Alex Sidebar\">\n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n </Step>\n</Steps>\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true # Enable reasoning capabilities\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n </Tab>\n</Tabs>\n\n### Google Vertex AI\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n </Tab>\n</Tabs>\n\n### Azure OpenAI\n\n<Tabs>\n <Tab title=\"Setup\">\n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n </Tab>\n</Tabs>\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n reasoning_effort: \"medium\" # low, medium, high\n \n - model_name: \"o3-reasoning\"\n litellm_params:\n model: \"o3\"\n reasoning_effort: \"high\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n<Tip>\n All AI requests from your team go through your infrastructure. You control the data and costs.\n</Tip>\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\"]\n```\n\n## Troubleshooting\n\n<AccordionGroup>\n <Accordion title=\"Connection refused error\">\n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n </Accordion>\n \n <Accordion title=\"Authentication errors\">\n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n </Accordion>\n \n <Accordion title=\"Model not found or deprecated\">\n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n </Accordion>\n \n <Accordion title=\"High latency or rate limiting\">\n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n </Accordion>\n \n <Accordion title=\"Cost tracking issues\">\n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n </Accordion>\n</AccordionGroup>\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n thinking: true\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n reasoning_effort: \"high\"\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support and enterprise features\n\n<Note>\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n</Note>"}, "c3fd9b461b7b90a9476ba62672d7e5ab1651d9e2": {"t": 1785770704.6295335, "n": 9028, "h": "0ff435485784e888", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n<Info>\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n</Info>\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n<Note>\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments.\n</Note>\n\n## Why Use LiteLLM?\n\n<CardGroup cols={2}>\n <Card title=\"Use Your Company's AI\" icon=\"building\">\n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n </Card>\n \n <Card title=\"Data Never Leaves Your Infrastructure\" icon=\"shield\">\n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n </Card>\n \n <Card title=\"Track Costs by Project\" icon=\"chart-line\">\n See exactly how much each project costs. Set budgets and get alerts\n </Card>\n \n <Card title=\"One Interface for All Models\" icon=\"plug\">\n Switch between Claude on Bedrock, Gemini on Vertex, or GPT-4 on Azure without changing code\n </Card>\n</CardGroup>\n\n## Quick Start\n\n<Steps>\n <Step title=\"Install LiteLLM\">\n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-3-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:main\n ```\n </Step>\n \n <Step title=\"Configure Your Providers\">\n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock\n - model_name: \"claude-3-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-sonnet-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI \n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n </Step>\n \n <Step title=\"Connect Alex Sidebar\">\n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-3-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n </Step>\n</Steps>\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"claude-3-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-opus-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"llama3-70b\"\n litellm_params:\n model: \"bedrock/meta.llama3-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n </Tab>\n</Tabs>\n\n### Google Vertex AI\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"code-bison\"\n litellm_params:\n model: \"vertex_ai/code-bison\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n </Tab>\n</Tabs>\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n<Tip>\n All AI requests from your team go through your infrastructure. You control the data and costs.\n</Tip>\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing\n\nDistribute requests across multiple model deployments:\n\n```yaml\nmodel_list:\n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint1.openai.azure.com\"\n api_key: \"key1\"\n \n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint2.openai.azure.com\"\n api_key: \"key2\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\"\n```\n\n### Cost Tracking\n\nEnable cost tracking for all requests:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \nlitellm_settings:\n success_callback: [\"langfuse\"] # or other observability tools\n track_cost_callback: true\n```\n\n### Security\n\nSecure your LiteLLM deployment:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\" # Required for all requests\n \n # IP allowlist\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n \n # Rate limiting\n max_parallel_requests: 100\n max_request_per_minute: 1000\n```\n\n## Monitoring & Observability\n\nLiteLLM provides built-in support for:\n- Request/response logging\n- Latency tracking \n- Cost monitoring per user/project\n- Error rates and retry statistics\n- Integration with MLflow, Langfuse, Helicone, Lunary, and more\n\nExample cost tracking dashboard:\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# View costs at http://localhost:4000/ui\n```\n\n## Troubleshooting\n\n<AccordionGroup>\n <Accordion title=\"Connection refused error\">\n - Verify LiteLLM is running and accessible\n - Check firewall rules\n - Ensure you're using the correct URL format with `/v1` suffix\n </Accordion>\n \n <Accordion title=\"Authentication errors\">\n - For Bedrock: Verify AWS credentials and permissions\n - For Vertex: Check service account permissions\n - Ensure the master key matches if configured\n </Accordion>\n \n <Accordion title=\"Model not found\">\n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider\n - Check region/location settings\n </Accordion>\n</AccordionGroup>\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS\nYour company has AWS Bedrock with Claude models. Instead of buying Anthropic API keys for each developer:\n1. Deploy LiteLLM on an EC2 instance\n2. Configure it to use your Bedrock models\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill\n\n### Scenario 2: Switching Between Models\nTest different models without changing code:\n```yaml\n# Your config.yaml\nmodel_list:\n - model_name: \"default-model\"\n litellm_params:\n model: \"bedrock/claude-3-sonnet\" # Switch to GPT-4, Gemini, etc.\n```\n\n### Scenario 3: Development vs Production\n```yaml\n# Dev environment - use cheaper models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/claude-3-haiku\"\n\n# Prod environment - use powerful models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/claude-3-opus\"\n```\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support\n\n<Note>\n LiteLLM gives you control over your AI infrastructure and works with all Alex Sidebar features.\n</Note>"}, "fa85e4f3ce8da4b3935c8f3166dc902307e6a73e": {"t": 1785770706.3445246, "n": 8913, "h": "d63e61fc5ce2492c", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n<Info>\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n</Info>\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n<Note>\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments.\n</Note>\n\n## Why Use LiteLLM?\n\n<CardGroup cols={2}>\n <Card title=\"Use Your Company's AI\" icon=\"building\">\n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n </Card>\n \n <Card title=\"Data Never Leaves Your Infrastructure\" icon=\"shield\">\n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n </Card>\n \n <Card title=\"Track Costs by Project\" icon=\"chart-line\">\n See exactly how much each project costs. Set budgets and get alerts\n </Card>\n \n <Card title=\"One Interface for All Models\" icon=\"plug\">\n Switch between Claude on Bedrock, Gemini on Vertex, or GPT-4 on Azure without changing code\n </Card>\n</CardGroup>\n\n## Quick Start\n\n<Steps>\n <Step title=\"Install LiteLLM\">\n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-3-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:main\n ```\n </Step>\n \n <Step title=\"Configure Your Providers\">\n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock\n - model_name: \"claude-3-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-sonnet-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI \n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n </Step>\n \n <Step title=\"Connect Alex Sidebar\">\n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-3-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n </Step>\n</Steps>\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"claude-3-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-opus-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"llama3-70b\"\n litellm_params:\n model: \"bedrock/meta.llama3-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n </Tab>\n</Tabs>\n\n### Google Vertex AI\n\n<Tabs>\n <Tab title=\"Setup\">\n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"code-bison\"\n litellm_params:\n model: \"vertex_ai/code-bison\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n </Tab>\n \n <Tab title=\"Authentication\">\n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n </Tab>\n</Tabs>\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to your team settings\n3. Override model endpoints with your LiteLLM proxy URL\n4. All team members will automatically use your proxy\n\n<Tip>\n All AI requests from your team go through your infrastructure. You control the data and costs.\n</Tip>\n\n## Advanced Configuration\n\n### Load Balancing\n\nDistribute requests across multiple model deployments:\n\n```yaml\nmodel_list:\n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint1.openai.azure.com\"\n api_key: \"key1\"\n \n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint2.openai.azure.com\"\n api_key: \"key2\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\"\n```\n\n### Cost Tracking\n\nEnable cost tracking for all requests:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \nlitellm_settings:\n success_callback: [\"langfuse\"] # or other observability tools\n track_cost_callback: true\n```\n\n### Security\n\nSecure your LiteLLM deployment:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\" # Required for all requests\n \n # IP allowlist\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n \n # Rate limiting\n max_parallel_requests: 100\n max_request_per_minute: 1000\n```\n\n## Monitoring & Observability\n\nLiteLLM provides built-in support for:\n- Request/response logging\n- Latency tracking \n- Cost monitoring per user/project\n- Error rates and retry statistics\n- Integration with MLflow, Langfuse, Helicone, Lunary, and more\n\nExample cost tracking dashboard:\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# View costs at http://localhost:4000/ui\n```\n\n## Troubleshooting\n\n<AccordionGroup>\n <Accordion title=\"Connection refused error\">\n - Verify LiteLLM is running and accessible\n - Check firewall rules\n - Ensure you're using the correct URL format with `/v1` suffix\n </Accordion>\n \n <Accordion title=\"Authentication errors\">\n - For Bedrock: Verify AWS credentials and permissions\n - For Vertex: Check service account permissions\n - Ensure the master key matches if configured\n </Accordion>\n \n <Accordion title=\"Model not found\">\n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider\n - Check region/location settings\n </Accordion>\n</AccordionGroup>\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS\nYour company has AWS Bedrock with Claude models. Instead of buying Anthropic API keys for each developer:\n1. Deploy LiteLLM on an EC2 instance\n2. Configure it to use your Bedrock models\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill\n\n### Scenario 2: Switching Between Models\nTest different models without changing code:\n```yaml\n# Your config.yaml\nmodel_list:\n - model_name: \"default-model\"\n litellm_params:\n model: \"bedrock/claude-3-sonnet\" # Switch to GPT-4, Gemini, etc.\n```\n\n### Scenario 3: Development vs Production\n```yaml\n# Dev environment - use cheaper models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/claude-3-haiku\"\n\n# Prod environment - use powerful models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/claude-3-opus\"\n```\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support\n\n<Note>\n LiteLLM gives you control over your AI infrastructure and works with all Alex Sidebar features.\n</Note>"}}}} |