{"repo": "alexsidebar/docs", "files": {"configuration/litellm-setup.mdx": {"ed31fcb6bfd5586cc96127b0d00a574330443a64": {"t": 1785747144.6350427, "n": 18061, "h": "648e6eef1c287911", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n\n\n## Why Use LiteLLM?\n\n\n \n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n \n \n \n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n \n \n \n See exactly how much each project costs. Set budgets and get alerts\n \n \n \n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n \n\n\n## Quick Start\n\n\n \n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n \n \n \n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n api_key: \"your-openai-key\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n \n \n \n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n \n\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n\n \n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n \n \n \n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n \n\n\n### Google Vertex AI\n\n\n \n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n \n \n \n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n \n\n\n### Azure OpenAI\n\n\n \n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"azure/o4-mini-2025-04-16\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro-2025-06-10\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n \n \n \n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n \n\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n \n - model_name: \"o3-pro-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n reasoning_effort: \"high\"\n \n - model_name: \"o4-mini-reasoning\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n reasoning_effort: \"medium\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n\n All AI requests from your team go through your infrastructure. You control the data and costs.\n\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\", \"o4-mini-2025-04-16\", \"o3-pro-2025-06-10\"]\n```\n\n## Troubleshooting\n\n\n \n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n \n \n \n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n \n \n \n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n \n \n \n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n \n \n \n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n \n\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n thinking: true\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact team@alexcodes.app for business support and enterprise features\n\n\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n"}, "c53c861da0d860ec0936ec60f566e7963806d8cb": {"t": 1785770700.0183918, "n": 18063, "h": "f590f1c9b870d4c9", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n\n\n## Why Use LiteLLM?\n\n\n \n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n \n \n \n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n \n \n \n See exactly how much each project costs. Set budgets and get alerts\n \n \n \n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n \n\n\n## Quick Start\n\n\n \n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n \n \n \n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n api_key: \"your-openai-key\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n \n \n \n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n \n\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n\n \n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n \n \n \n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n \n\n\n### Google Vertex AI\n\n\n \n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n \n \n \n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n \n\n\n### Azure OpenAI\n\n\n \n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"azure/o4-mini-2025-04-16\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro-2025-06-10\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n \n \n \n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n \n\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n \n - model_name: \"o3-pro-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n reasoning_effort: \"high\"\n \n - model_name: \"o4-mini-reasoning\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n reasoning_effort: \"medium\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n\n All AI requests from your team go through your infrastructure. You control the data and costs.\n\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\", \"o4-mini-2025-04-16\", \"o3-pro-2025-06-10\"]\n```\n\n## Troubleshooting\n\n\n \n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n \n \n \n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n \n \n \n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n \n \n \n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n \n \n \n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n \n\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n thinking: true\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support and enterprise features\n\n\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n"}, "ddfbdcd79953d4ace54ffb1a19a2306b72646247": {"t": 1785770701.3612468, "n": 18179, "h": "2b99109ecbe55dfa", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n\n\n## Why Use LiteLLM?\n\n\n \n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n \n \n \n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n \n \n \n See exactly how much each project costs. Set budgets and get alerts\n \n \n \n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n \n\n\n## Quick Start\n\n\n \n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n \n \n \n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n api_key: \"your-openai-key\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n \n \n \n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n \n\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n\n \n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true # Enable reasoning capabilities\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n \n \n \n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n \n\n\n### Google Vertex AI\n\n\n \n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n \n \n \n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n \n\n\n### Azure OpenAI\n\n\n \n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"azure/o4-mini-2025-04-16\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro-2025-06-10\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n \n \n \n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n \n\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n reasoning_effort: \"medium\" # low, medium, high\n \n - model_name: \"o3-pro-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\"\n reasoning_effort: \"high\"\n \n - model_name: \"o4-mini-reasoning\"\n litellm_params:\n model: \"o4-mini-2025-04-16\"\n reasoning_effort: \"medium\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n\n All AI requests from your team go through your infrastructure. You control the data and costs.\n\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\", \"o4-mini-2025-04-16\", \"o3-pro-2025-06-10\"]\n```\n\n## Troubleshooting\n\n\n \n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n \n \n \n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n \n \n \n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n \n \n \n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n \n \n \n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n \n\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro-2025-06-10\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n thinking: true\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n reasoning_effort: \"high\"\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support and enterprise features\n\n\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n"}, "04cc0c4608a736f8ac7f06b9350d061929f4c3c8": {"t": 1785770703.0079153, "n": 17532, "h": "15840335d36a6439", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments. Current stable version: **v1.73.6-stable** (June 2025)\n\n\n## Why Use LiteLLM?\n\n\n \n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n \n \n \n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n \n \n \n See exactly how much each project costs. Set budgets and get alerts\n \n \n \n Switch between Claude 4 on Bedrock, Gemini 2.5 on Vertex, or GPT-4 on Azure without changing code\n \n\n\n## Quick Start\n\n\n \n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-4-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:v1.73.6-stable\n ```\n \n \n \n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock - Latest Claude 4 Models\n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI - Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # OpenAI O-Series with Reasoning\n - model_name: \"o4-mini\"\n litellm_params:\n model: \"o4-mini\"\n api_key: \"your-openai-key\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n \n \n \n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-4-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n \n\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n\n \n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Claude 4 Models\n - model_name: \"claude-4-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Reasoning and Thinking Support\n - model_name: \"claude-4-sonnet-reasoning\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n thinking: true # Enable reasoning capabilities\n \n # Latest Llama 4 Models\n - model_name: \"llama4-70b\"\n litellm_params:\n model: \"bedrock/meta.llama4-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # DeepSeek R1 Models\n - model_name: \"deepseek-r1\"\n litellm_params:\n model: \"bedrock/deepseek.deepseek-r1-distill-llama-70b\"\n aws_region_name: \"us-east-1\"\n ```\n \n \n \n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n - Workload Identity Federation (for cross-cloud deployments)\n \n\n\n### Google Vertex AI\n\n\n \n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n # Latest Gemini 2.5 Models\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"gemini-2.5-flash\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Claude 4 on Vertex AI\n - model_name: \"claude-4-vertex\"\n litellm_params:\n model: \"vertex_ai/claude-4-opus\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n # Multimodal Image Generation\n - model_name: \"imagen-4\"\n litellm_params:\n model: \"vertex_ai/imagen-4\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n \n \n \n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n - Authorized user credentials\n \n\n\n### Azure OpenAI\n\n\n \n Configure Azure OpenAI with latest models:\n \n ```yaml\n model_list:\n # Latest O-Series Models\n - model_name: \"o3-pro\"\n litellm_params:\n model: \"azure/o3-pro\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n \n # GPT-4 with Audio Preview\n - model_name: \"gpt-4o-audio\"\n litellm_params:\n model: \"azure/gpt-4o-audio-preview-2025-06-03\"\n api_base: \"https://your-resource.openai.azure.com\"\n api_key: \"your-azure-key\"\n api_version: \"2025-01-01-preview\"\n ```\n \n \n \n Azure supports multiple authentication methods:\n - API keys\n - Azure AD tokens\n - Managed Identity\n - Certificate-based authentication\n \n\n\n## Advanced Features\n\n### Reasoning and Thinking Capabilities\n\nEnable advanced reasoning for supported models:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-reasoning\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n thinking: true\n reasoning_effort: \"medium\" # low, medium, high\n \n - model_name: \"o3-reasoning\"\n litellm_params:\n model: \"o3\"\n reasoning_effort: \"high\"\n```\n\n### Multimodal Support\n\nConfigure models for text, image, audio, and video:\n\n```yaml\nmodel_list:\n # Vision + Audio Models\n - model_name: \"gpt-4o-multimodal\"\n litellm_params:\n model: \"gpt-4o\"\n supports_vision: true\n supports_audio: true\n \n # Gemini with Enhanced Multimodal\n - model_name: \"gemini-2.5-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\"\n supports_vision: true\n supports_pdf_input: true\n supports_audio: true\n```\n\n### MCP Gateway Integration\n\nEnable Model Context Protocol for enhanced tool use:\n\n```yaml\ngeneral_settings:\n enable_mcp_gateway: true\n \nmcp_servers:\n - server_name: \"filesystem\"\n server_command: [\"uvx\", \"mcp-server-filesystem\", \"/path/to/allowed/files\"]\n \n - server_name: \"jira\"\n server_command: [\"node\", \"/path/to/jira-mcp-server\"]\n auth_type: \"api_key\"\n auth_value: \"your-jira-api-key\"\n```\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n\n All AI requests from your team go through your infrastructure. You control the data and costs.\n\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing with Fallbacks\n\nDistribute requests across multiple model deployments with intelligent routing:\n\n```yaml\nmodel_list:\n - model_name: \"claude-4-primary\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"claude-4-fallback\"\n litellm_params:\n model: \"anthropic/claude-4-sonnet-20250514\"\n api_key: \"fallback-key\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\", \"weighted-round-robin\"\n model_group_alias: \"claude-4\"\n fallbacks: [{\"claude-4-primary\": [\"claude-4-fallback\"]}]\n cooldown_time: 60 # seconds before retrying failed deployment\n```\n\n### Cost Tracking and Budget Management\n\nEnable comprehensive cost tracking:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# Budget controls\nlitellm_settings:\n max_budget: 1000 # Monthly budget in USD\n budget_duration: \"monthly\" # daily, weekly, monthly\n success_callback: [\"langfuse\", \"prometheus\"]\n track_cost_callback: true\n \n# User-level budgets\nuser_api_key_config:\n user1:\n budget_duration: \"monthly\"\n max_budget: 100\n```\n\n### Security and Rate Limiting\n\nSecure your LiteLLM deployment with advanced controls:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\"\n \n # Enhanced security\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n disable_spend_logs: false\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n # Advanced rate limiting\n max_parallel_requests: 1000\n max_request_per_minute: 10000\n rate_limiting_strategy: \"sliding-window\" # New accurate rate limiting\n \n# SCIM integration for enterprise SSO\nscim_settings:\n enabled: true\n scim_base_url: \"https://your-litellm.com/scim/v2\"\n```\n\n### Vector Store Integration\n\nConnect to knowledge bases and vector stores:\n\n```yaml\nvector_stores:\n - store_name: \"company_docs\"\n store_type: \"bedrock_knowledge_base\"\n knowledge_base_id: \"your-kb-id\"\n aws_region: \"us-east-1\"\n \n - store_name: \"technical_docs\"\n store_type: \"pinecone\"\n api_key: \"your-pinecone-key\"\n environment: \"your-environment\"\n\n# Auto-activate for specific models\nmodel_list:\n - model_name: \"claude-4-with-kb\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\"\n vector_store: \"company_docs\"\n```\n\n## Monitoring & Observability\n\nLiteLLM v1.73.6 provides enhanced monitoring capabilities:\n\n### Performance Metrics\n- **2x Higher RPS**: Enhanced aiohttp transport for improved performance\n- **50ms Median Latency**: Optimized for high-throughput applications\n- **Multi-instance Rate Limiting**: Accurate rate limiting across deployments\n\n### Dashboard Features\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n ui_access_mode: \"admin_only\" # or \"all\"\n \n# Enhanced logging\nlitellm_settings:\n success_callback: [\"langfuse\", \"prometheus\", \"datadog\"]\n failure_callback: [\"slack\", \"pagerduty\"]\n \n# Session tracking\nsession_config:\n enable_session_logs: true\n session_retention_days: 30\n```\n\n### Real-time Monitoring\n```yaml\n# Prometheus metrics\nprometheus_settings:\n enabled: true\n track_end_users: false # Opt-in to prevent large metric sets\n \n# Health checks\nhealth_check:\n enabled: true\n check_interval: 300 # seconds\n models_to_check: [\"claude-4-sonnet\", \"gemini-2.5-pro\"]\n```\n\n## Troubleshooting\n\n\n \n - Verify LiteLLM is running and accessible\n - Check firewall rules and security groups\n - Ensure you're using the correct URL format with `/v1` suffix\n - For Docker: Check port mapping and container status\n \n \n \n - **Bedrock**: Verify AWS credentials, IAM permissions, and model access\n - **Vertex**: Check service account permissions and project settings\n - **Azure**: Ensure API keys and resource endpoints are correct\n - Verify master key matches if configured\n \n \n \n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider console\n - Update to latest model versions (e.g., claude-4 instead of claude-3)\n - Check region/location settings and model availability\n \n \n \n - Enable aiohttp transport: `USE_AIOHTTP_TRANSPORT=True`\n - Implement load balancing across multiple deployments\n - Adjust `max_parallel_requests` and rate limiting settings\n - Consider regional deployment distribution\n \n \n \n - Ensure database connection is properly configured\n - Check that `track_cost_callback: true` is set\n - Verify model pricing information is up to date\n - Review spend logs retention settings\n \n\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS with Latest Models\nYour company has AWS Bedrock with Claude 4 models. Instead of buying Anthropic API keys:\n1. Deploy LiteLLM v1.73.6-stable on an EC2 instance\n2. Configure it to use your Bedrock Claude 4 models with reasoning capabilities\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill with detailed tracking\n\n### Scenario 2: Multi-Cloud Model Testing\nTest latest models across providers without changing code:\n```yaml\nmodel_list:\n - model_name: \"best-reasoning\"\n litellm_params:\n model: \"o3-pro\" # OpenAI's latest reasoning model\n reasoning_effort: \"high\"\n \n - model_name: \"best-multimodal\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-pro\" # Google's latest multimodal\n \n - model_name: \"best-coding\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-sonnet-20250514-v1:0\" # Claude 4 for coding\n```\n\n### Scenario 3: Development vs Production with New Models\n```yaml\n# Dev environment - use efficient models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-haiku-20240307-v1:0\"\n max_tokens: 1000\n\n# Staging - test new capabilities\n- model_name: \"staging-model\"\n litellm_params:\n model: \"vertex_ai/gemini-2.5-flash\"\n thinking: true\n\n# Production - use most capable models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/anthropic.claude-4-opus-20250514-v1:0\"\n reasoning_effort: \"high\"\n```\n\n### Scenario 4: Enterprise Security and Compliance\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secure-key\"\n guardrails: [\"presidio_pii\", \"bedrock_guardrails\"]\n \n# PII masking and content filtering\nguardrail_settings:\n presidio:\n mask_entities: [\"PERSON\", \"EMAIL\", \"PHONE_NUMBER\"]\n block_entities: [\"MEDICAL_LICENSE\"]\n \n bedrock_guardrails:\n guardrail_id: \"your-guardrail-id\"\n guardrail_version: \"1\"\n```\n\n## Enterprise Features (LiteLLM v1.73.6+)\n\n### SCIM Integration\nAutomatic user provisioning from your identity provider:\n- Okta, Azure AD, OneLogin support\n- Automatic team creation and user assignment\n- Deprovisioning when users are removed\n\n### Advanced Analytics\n- Team and tag-based usage tracking\n- Daily spend analysis by model and user\n- Session grouping and analysis\n- Audit logs for compliance\n\n### Enhanced Security\n- Vector store permissions by team/user\n- MCP server access controls\n- IP allowlisting and rate limiting\n- End-to-end encryption options\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage with the new dashboard\n- Explore [Vector Store integration](https://docs.litellm.ai/docs/proxy/vector_stores) for RAG applications\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support and enterprise features\n\n\n LiteLLM v1.73.6-stable gives you control over your AI infrastructure with the latest models and enterprise-grade features, working seamlessly with all Alex Sidebar capabilities.\n"}, "c3fd9b461b7b90a9476ba62672d7e5ab1651d9e2": {"t": 1785770704.6295335, "n": 9028, "h": "0ff435485784e888", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments.\n\n\n## Why Use LiteLLM?\n\n\n \n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n \n \n \n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n \n \n \n See exactly how much each project costs. Set budgets and get alerts\n \n \n \n Switch between Claude on Bedrock, Gemini on Vertex, or GPT-4 on Azure without changing code\n \n\n\n## Quick Start\n\n\n \n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-3-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:main\n ```\n \n \n \n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock\n - model_name: \"claude-3-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-sonnet-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI \n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n \n \n \n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-3-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n \n\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n\n \n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"claude-3-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-opus-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"llama3-70b\"\n litellm_params:\n model: \"bedrock/meta.llama3-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n ```\n \n \n \n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n \n\n\n### Google Vertex AI\n\n\n \n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"code-bison\"\n litellm_params:\n model: \"vertex_ai/code-bison\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n \n \n \n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n \n\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to Models tab\n3. Add your LiteLLM proxy URL as Base URL for each model type\n4. All team members automatically use your proxy\n\n\n All AI requests from your team go through your infrastructure. You control the data and costs.\n\n\nSee the [Team Configuration Guide](/configuration/team-configuration) for detailed instructions on managing team models.\n\n## Advanced Configuration\n\n### Load Balancing\n\nDistribute requests across multiple model deployments:\n\n```yaml\nmodel_list:\n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint1.openai.azure.com\"\n api_key: \"key1\"\n \n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint2.openai.azure.com\"\n api_key: \"key2\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\"\n```\n\n### Cost Tracking\n\nEnable cost tracking for all requests:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \nlitellm_settings:\n success_callback: [\"langfuse\"] # or other observability tools\n track_cost_callback: true\n```\n\n### Security\n\nSecure your LiteLLM deployment:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\" # Required for all requests\n \n # IP allowlist\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n \n # Rate limiting\n max_parallel_requests: 100\n max_request_per_minute: 1000\n```\n\n## Monitoring & Observability\n\nLiteLLM provides built-in support for:\n- Request/response logging\n- Latency tracking \n- Cost monitoring per user/project\n- Error rates and retry statistics\n- Integration with MLflow, Langfuse, Helicone, Lunary, and more\n\nExample cost tracking dashboard:\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# View costs at http://localhost:4000/ui\n```\n\n## Troubleshooting\n\n\n \n - Verify LiteLLM is running and accessible\n - Check firewall rules\n - Ensure you're using the correct URL format with `/v1` suffix\n \n \n \n - For Bedrock: Verify AWS credentials and permissions\n - For Vertex: Check service account permissions\n - Ensure the master key matches if configured\n \n \n \n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider\n - Check region/location settings\n \n\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS\nYour company has AWS Bedrock with Claude models. Instead of buying Anthropic API keys for each developer:\n1. Deploy LiteLLM on an EC2 instance\n2. Configure it to use your Bedrock models\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill\n\n### Scenario 2: Switching Between Models\nTest different models without changing code:\n```yaml\n# Your config.yaml\nmodel_list:\n - model_name: \"default-model\"\n litellm_params:\n model: \"bedrock/claude-3-sonnet\" # Switch to GPT-4, Gemini, etc.\n```\n\n### Scenario 3: Development vs Production\n```yaml\n# Dev environment - use cheaper models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/claude-3-haiku\"\n\n# Prod environment - use powerful models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/claude-3-opus\"\n```\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support\n\n\n LiteLLM gives you control over your AI infrastructure and works with all Alex Sidebar features.\n"}, "fa85e4f3ce8da4b3935c8f3166dc902307e6a73e": {"t": 1785770706.3445246, "n": 8913, "h": "d63e61fc5ce2492c", "c": "---\ntitle: \"LiteLLM Proxy Setup\"\ndescription: \"Connect Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers\"\n---\n\nLiteLLM proxy connects Alex Sidebar to Amazon Bedrock, Google Vertex AI, and other enterprise AI providers. Teams can use their existing cloud infrastructure without changing their security setup.\n\n\n **For iOS Developers**: If your company already pays for AWS Bedrock or Google Cloud AI, this guide shows how to use those models in Xcode with Alex Sidebar instead of buying separate API keys.\n\n\n## What is LiteLLM?\n\nLiteLLM is an open-source proxy that translates between the OpenAI API format and 100+ different AI providers. Alex Sidebar can work with enterprise AI services that don't support the OpenAI API format.\n\n\n LiteLLM is what Alex Sidebar uses internally for model connections, making it a well-tested solution for enterprise deployments.\n\n\n## Why Use LiteLLM?\n\n\n \n If your company uses AWS Bedrock or Google Cloud AI, LiteLLM lets you access those models through Alex Sidebar\n \n \n \n Your code stays within your company's cloud. No data goes to Alex Sidebar servers\n \n \n \n See exactly how much each project costs. Set budgets and get alerts\n \n \n \n Switch between Claude on Bedrock, Gemini on Vertex, or GPT-4 on Azure without changing code\n \n\n\n## Quick Start\n\n\n \n Choose your deployment method:\n \n **Option 1: pip install (simplest)**\n ```bash\n pip install 'litellm[proxy]'\n litellm --model bedrock/claude-3-sonnet --port 4000\n ```\n \n **Option 2: Docker (recommended for production)**\n ```bash\n docker run -p 4000:4000 ghcr.io/berriai/litellm:main\n ```\n \n \n \n Create a `config.yaml` file in your LiteLLM directory:\n \n ```yaml\n model_list:\n # Amazon Bedrock\n - model_name: \"claude-3-sonnet\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-sonnet-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n # Google Vertex AI \n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-gcp-project\"\n vertex_location: \"us-central1\"\n \n # Start proxy with config\n # litellm --config config.yaml --port 4000\n ```\n \n \n \n In Alex Sidebar, add a custom model pointing to your LiteLLM proxy:\n \n 1. Open Settings → Models → Custom Models\n 2. Click \"Add New Model\"\n 3. Configure:\n - **Model ID**: Your model name from config.yaml (e.g., `claude-3-sonnet`)\n - **Base URL**: Your LiteLLM URL + `/v1` (e.g., `https://litellm.company.com/v1`)\n - **API Key**: Your LiteLLM proxy key (if configured)\n \n\n\n## Provider-Specific Setup\n\n### Amazon Bedrock\n\n\n \n 1. Ensure your AWS credentials are configured on the LiteLLM server\n 2. Enable the models you need in the AWS Bedrock console\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"claude-3-opus\"\n litellm_params:\n model: \"bedrock/anthropic.claude-3-opus-20240229-v1:0\"\n aws_region_name: \"us-east-1\"\n \n - model_name: \"llama3-70b\"\n litellm_params:\n model: \"bedrock/meta.llama3-70b-instruct-v1:0\"\n aws_region_name: \"us-east-1\"\n ```\n \n \n \n LiteLLM supports multiple AWS authentication methods:\n - IAM roles (recommended for EC2/ECS)\n - Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)\n - AWS profiles\n - Temporary credentials with STS\n \n\n\n### Google Vertex AI\n\n\n \n 1. Enable the Vertex AI API in your GCP project\n 2. Set up authentication (service account recommended)\n 3. Add to your LiteLLM config:\n \n ```yaml\n model_list:\n - model_name: \"gemini-2.5-pro\"\n litellm_params:\n model: \"vertex_ai/gemini-1.5-pro\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n \n - model_name: \"code-bison\"\n litellm_params:\n model: \"vertex_ai/code-bison\"\n vertex_project: \"your-project-id\"\n vertex_location: \"us-central1\"\n ```\n \n \n \n Set up authentication using one of these methods:\n - Service account JSON file: `export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`\n - Workload Identity (for GKE)\n - Default application credentials\n \n\n\n## Team Configuration\n\nFor team accounts, you can override all Alex Sidebar model endpoints:\n\n1. Go to [Alex Sidebar Admin Portal](https://alexcodes.app/admin)\n2. Navigate to your team settings\n3. Override model endpoints with your LiteLLM proxy URL\n4. All team members will automatically use your proxy\n\n\n All AI requests from your team go through your infrastructure. You control the data and costs.\n\n\n## Advanced Configuration\n\n### Load Balancing\n\nDistribute requests across multiple model deployments:\n\n```yaml\nmodel_list:\n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint1.openai.azure.com\"\n api_key: \"key1\"\n \n - model_name: \"gpt-4\"\n litellm_params:\n model: \"azure/gpt-4\"\n api_base: \"https://endpoint2.openai.azure.com\"\n api_key: \"key2\"\n\nrouter_settings:\n routing_strategy: \"least-busy\" # or \"round-robin\"\n```\n\n### Cost Tracking\n\nEnable cost tracking for all requests:\n\n```yaml\ngeneral_settings:\n master_key: \"your-secret-key\"\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \nlitellm_settings:\n success_callback: [\"langfuse\"] # or other observability tools\n track_cost_callback: true\n```\n\n### Security\n\nSecure your LiteLLM deployment:\n\n```yaml\ngeneral_settings:\n master_key: \"sk-your-secret-key\" # Required for all requests\n \n # IP allowlist\n allowed_ips: [\"10.0.0.0/8\", \"172.16.0.0/12\"]\n \n # Rate limiting\n max_parallel_requests: 100\n max_request_per_minute: 1000\n```\n\n## Monitoring & Observability\n\nLiteLLM provides built-in support for:\n- Request/response logging\n- Latency tracking \n- Cost monitoring per user/project\n- Error rates and retry statistics\n- Integration with MLflow, Langfuse, Helicone, Lunary, and more\n\nExample cost tracking dashboard:\n```yaml\ngeneral_settings:\n database_url: \"postgresql://user:pass@localhost:5432/litellm\"\n \n# View costs at http://localhost:4000/ui\n```\n\n## Troubleshooting\n\n\n \n - Verify LiteLLM is running and accessible\n - Check firewall rules\n - Ensure you're using the correct URL format with `/v1` suffix\n \n \n \n - For Bedrock: Verify AWS credentials and permissions\n - For Vertex: Check service account permissions\n - Ensure the master key matches if configured\n \n \n \n - Check model name matches exactly in config.yaml\n - Verify the model is enabled in your cloud provider\n - Check region/location settings\n \n\n\n## Common Use Cases for iOS Teams\n\n### Scenario 1: Company Uses AWS\nYour company has AWS Bedrock with Claude models. Instead of buying Anthropic API keys for each developer:\n1. Deploy LiteLLM on an EC2 instance\n2. Configure it to use your Bedrock models\n3. Developers connect Alex Sidebar to your LiteLLM endpoint\n4. All costs go to your AWS bill\n\n### Scenario 2: Switching Between Models\nTest different models without changing code:\n```yaml\n# Your config.yaml\nmodel_list:\n - model_name: \"default-model\"\n litellm_params:\n model: \"bedrock/claude-3-sonnet\" # Switch to GPT-4, Gemini, etc.\n```\n\n### Scenario 3: Development vs Production\n```yaml\n# Dev environment - use cheaper models\n- model_name: \"dev-model\"\n litellm_params:\n model: \"bedrock/claude-3-haiku\"\n\n# Prod environment - use powerful models \n- model_name: \"prod-model\"\n litellm_params:\n model: \"bedrock/claude-3-opus\"\n```\n\n## Next Steps\n\n- Review [LiteLLM's official documentation](https://docs.litellm.ai) for detailed configuration options\n- Check the [Proxy UI](http://localhost:4000/ui) to monitor costs and usage\n- Join the [Alex Sidebar Discord](https://discord.gg/T5zxfReEnd) for help with enterprise setups\n- Contact daniel@alexcodes.app for business support\n\n\n LiteLLM gives you control over your AI infrastructure and works with all Alex Sidebar features.\n"}}}}