1 line
27 KiB
JSON
1 line
27 KiB
JSON
{"repo": "christian-posta/scripted-solo-demos", "files": {"deepseek-blog/resources/failover/deepseek-failover.yaml": {"71c7419c833411c3835d4a74d2998d2a78492b16": {"t": 1785746310.9371464, "n": 570, "h": "e83fd933b37f8752", "c": "apiVersion: gloo.solo.io/v1\nkind: Upstream\nmetadata:\n labels:\n app: gloo\n name: deepseek\n namespace: gloo-system\nspec:\n ai:\n multi:\n priorities:\n - pool:\n - openai:\n customHost:\n host: \"api.deepseek.com\"\n authToken:\n secretRef:\n name: deepseek-secret\n namespace: gloo-system \n - pool:\n - openai:\n customHost:\n host: ollama-deepseek-r1.ollama.svc.cluster.local\n port: 11434\n model: \"deepseek-r1:7b\"\n\n\n"}, "fc483dfc33022a745cfc0f3e6534432a560fc095": {"t": 1785761633.6229446, "n": 570, "h": "e83fd933b37f8752", "c": "apiVersion: gloo.solo.io/v1\nkind: Upstream\nmetadata:\n labels:\n app: gloo\n name: deepseek\n namespace: gloo-system\nspec:\n ai:\n multi:\n priorities:\n - pool:\n - openai:\n customHost:\n host: \"api.deepseek.com\"\n authToken:\n secretRef:\n name: deepseek-secret\n namespace: gloo-system \n - pool:\n - openai:\n customHost:\n host: ollama-deepseek-r1.ollama.svc.cluster.local\n port: 11434\n model: \"deepseek-r1:7b\"\n\n\n"}}, "deepseek-blog/README.md": {"71c7419c833411c3835d4a74d2998d2a78492b16": {"t": 1785746556.220563, "n": 3359, "h": "ac33e9fdbc842fe5", "c": "Demo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\nIn this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls. Here is an example of calling DeepSeek directly (note, using the `deepseek-chat` model because `deepseek-reasoner` is down at the time of this writing):\n\n```bash\ncurl -v https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What is 2+2\"}\n ],\n \"stream\": false\n }'\n```\n\n### Deploy Gloo Gateway\n\n```bash\n./install-gateway-1-19.sh\n./setup-observability.sh\n```\n\nSet up secrets for DeepSeek and OpenAI\n\n```bash\nsource ~/bin/ai-keys\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n\nkubectl create secret generic openai-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $OPENAI_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f - \n```\n\nTo deploy the deepseek-r1:7b model onto a cluster with GPUs, make sure you have a cluster with GPUs (see for GKE below), and install:\n\n```bash\nkubectl apply -f resources/deepseek/ollama-deepseek.yaml\n```\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\nhttps://cloud.google.com/compute/docs/gpus#l4-gpus\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}, "4f14da22c5f0c2dac935154822256fd2fe39630e": {"t": 1785761632.072444, "n": 3359, "h": "ac33e9fdbc842fe5", "c": "Demo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\nIn this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls. Here is an example of calling DeepSeek directly (note, using the `deepseek-chat` model because `deepseek-reasoner` is down at the time of this writing):\n\n```bash\ncurl -v https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What is 2+2\"}\n ],\n \"stream\": false\n }'\n```\n\n### Deploy Gloo Gateway\n\n```bash\n./install-gateway-1-19.sh\n./setup-observability.sh\n```\n\nSet up secrets for DeepSeek and OpenAI\n\n```bash\nsource ~/bin/ai-keys\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n\nkubectl create secret generic openai-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $OPENAI_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f - \n```\n\nTo deploy the deepseek-r1:7b model onto a cluster with GPUs, make sure you have a cluster with GPUs (see for GKE below), and install:\n\n```bash\nkubectl apply -f resources/deepseek/ollama-deepseek.yaml\n```\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\nhttps://cloud.google.com/compute/docs/gpus#l4-gpus\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}, "2dc827707a7d8d2669c4669c067a0a6ecc109211": {"t": 1785761633.635991, "n": 3134, "h": "527bf2ec9602d0a3", "c": "Demo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\nIn this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls. Here is an example of calling DeepSeek directly (note, using the `deepseek-chat` model because `deepseek-reasoner` is down at the time of this writing):\n\n```bash\ncurl -v https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"What is 2+2\"}\n ],\n \"stream\": false\n }'\n```\n\n### Deploy Gloo Gateway\n\n```bash\n./install-gateway-1-19.sh\n```\n\n\n### Set up Gateway To Route to DeepSeek\n\n```bash\nsource ~/bin/ai-keys\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n\nkubectl create secret generic openai-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $OPENAI_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f - \n```\n\n\n\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\nhttps://cloud.google.com/compute/docs/gpus#l4-gpus\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}, "4d4b212b5a298d1b79d518b772c842711b037307": {"t": 1785761636.216029, "n": 3424, "h": "ceed59193d428868", "c": "In this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls:\n\n```bash\ncurl -v https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"user\", \"content\": \"Hello!\"}\n ],\n \"stream\": false\n }'\n```\n\n\n### Deploy KGateway\n\n```bash\nkubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml\n\nhelm repo add gloo-ee-helm https://storage.googleapis.com/gloo-ee-helm\nhelm repo update\n\nVERSION=\"1.18.2\"\n\nsource ~/bin/glooe-license-key-env \nhelm upgrade -i gloo-gateway gloo-ee-helm/gloo-ee \\\n --version $VERSION \\\n --namespace gloo-system --create-namespace \\\n --set license_key=$GLOO_LICENSE_WITH_AI \\\n-f kgateway-values.yaml\n\nkubectl apply -f ai-gateway.yaml\n```\n\n\n### Set up Gateway To Route to DeepSeek\n\n```bash\nsource ~/bin/ai-keys\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n\nkubectl create secret generic openai-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $OPENAI_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f - \n```\n\nDemo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\nhttps://cloud.google.com/compute/docs/gpus#l4-gpus\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}, "fc483dfc33022a745cfc0f3e6534432a560fc095": {"t": 1785761638.5452776, "n": 3494, "h": "c25cfd72a5b7a9e4", "c": "In this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls:\n\n```bash\ncurl https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Hello!\"}\n ],\n \"stream\": false\n }'\n```\n\n\n### Deploy KGateway\n\n```bash\nkubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml\n\nhelm repo add gloo-ee-helm https://storage.googleapis.com/gloo-ee-helm\nhelm repo update\n\nVERSION=\"1.18.2\"\n\nsource ~/bin/glooe-license-key-env \nhelm upgrade -i gloo-gateway gloo-ee-helm/gloo-ee \\\n --version $VERSION \\\n --namespace gloo-system --create-namespace \\\n --set license_key=$GLOO_LICENSE_WITH_AI \\\n-f kgateway-values.yaml\n\nkubectl apply -f ai-gateway.yaml\n```\n\n\n### Set up Gateway To Route to DeepSeek\n\n```bash\nsource ~/bin/ai-keys\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n\nkubectl create secret generic openai-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $OPENAI_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f - \n```\n\nDemo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\nhttps://cloud.google.com/compute/docs/gpus#l4-gpus\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}, "6499d0ba3d98a16a76dbda27c0e16a0ffb477f17": {"t": 1785761641.0538905, "n": 3430, "h": "6a04832c7f20cd69", "c": "In this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls:\n\ncurl https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Hello!\"}\n ],\n \"stream\": false\n }'\n\n\n### Deploy KGateway\n\n```bash\nkubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml\n\nhelm repo add gloo-ee-helm https://storage.googleapis.com/gloo-ee-helm\nhelm repo update\n\nVERSION=\"1.18.2\"\n\nsource ~/bin/glooe-license-key-env \nhelm upgrade -i gloo-gateway gloo-ee-helm/gloo-ee \\\n --version $VERSION \\\n --namespace gloo-system --create-namespace \\\n --set license_key=$GLOO_LICENSE_WITH_AI \\\n-f kgateway-values.yaml\n\nkubectl apply -f ai-gateway.yaml\n```\n\n\n### Set up Gateway To Route to DeepSeek\n\n```bash\nsource ~/bin/ai-keys\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n\nkubectl create secret generic openai-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $OPENAI_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f - \n```\n\nDemo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}, "65516ed076c6719eac6bf8839a9c31b52edba5da": {"t": 1785761642.7370706, "n": 3544, "h": "dcbd9d6a80eb99a4", "c": "In this guide, we will deploy an AI Gateway to apply policy to DeepSeek R1 calls:\n\ncurl https://api.deepseek.com/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -H \"Authorization: Bearer $DEEPSEEK_API_KEY\" \\\n -d '{\n \"model\": \"deepseek-chat\",\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Hello!\"}\n ],\n \"stream\": false\n }'\n\n\n### Deploy KGateway\n\n```bash\nkubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml\n\nhelm repo add gloo-ee-helm https://storage.googleapis.com/gloo-ee-helm\nhelm repo update\n\nVERSION=\"1.18.2\"\n\nsource ~/bin/glooe-license-key-env \nhelm upgrade -i gloo-gateway gloo-ee-helm/gloo-ee \\\n --version $VERSION \\\n --namespace gloo-system --create-namespace \\\n --set license_key=$GLOO_LICENSE_WITH_AI \\\n-f kgateway-values.yaml\n\nkubectl apply -f ai-gateway.yaml\n```\n\n\n### Set up Gateway To Route to DeepSeek\n\n```bash\nkubectl create secret generic deepseek-secret -n gloo-system \\\n --from-literal=\"Authorization=Bearer $DEEPSEEK_API_KEY\" \\\n --dry-run=client -oyaml | kubectl apply -f -\n```\n\nAdd the DeepSeek Upstream and HTTPRoute\n\n```bash\nkubectl apply -f deepseek-upstream.yaml\nkubectl apply -f http-routes.yaml\n```\n\nTo Route Locally, add the DeepSeek Local Upstream and HTTPRoute\n\n```bash\nkubectl apply -f deepseek-local-upstream.yaml\nkubectl apply -f http-routes-local.yaml\n```\n\nTODO Fill out Demo:\n* Calling out to DeepSeek \n* Prompt Guard the call\n* Traffic Split to Local\n* All traffic to Local\n* Secure the call with JWT\n* Metrics\n\n### Setup GKE Clusters with NVIDIA L4 inferencing GPUs\n\n```bash\nsource env.sh\ngcloud container clusters create $CLUSTER_NAME \\\n --zone $ZONE \\\n --region $REGION \\\n --node-locations $ZONE \\\n --release-channel \"regular\" \\\n --machine-type \"n1-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --disk-type \"pd-standard\" \\\n --disk-size \"100\" \\\n --no-enable-intra-node-visibility \\\n --metadata disable-legacy-endpoints=true \\\n --max-pods-per-node \"110\" \\\n --num-nodes \"2\" \\\n --logging=SYSTEM,WORKLOAD \\\n --monitoring=SYSTEM \\\n --enable-ip-alias \\\n --default-max-pods-per-node \"110\" \\\n --no-enable-master-authorized-networks\n```\n\n```bash\ngcloud container node-pools create gpu-pool \\\n --cluster $CLUSTER_NAME \\\n --zone $ZONE \\\n --node-locations $ZONE \\\n --machine-type \"g2-standard-8\" \\\n --image-type \"UBUNTU_CONTAINERD\" \\\n --node-labels=\"gke-no-default-nvidia-gpu-device-plugin=true\" \\\n --metadata disable-legacy-endpoints=true \\\n --disk-size \"100\" \\\n --num-nodes \"1\" \\\n --tags=nvidia-ingress-all\n```\n\nInstall GPU Operator to manage GPU resources, containerd toolkits, and GPU device plugins.\n\n```bash\nhelm repo add nvidia https://helm.ngc.nvidia.com/nvidia\nhelm repo update\n\nexport GPU_OPERATOR_VERSION=v24.9.1\nkubectl create namespace gpu-operator\nkubectl apply -f ./gke-resourcequota.yaml\nhelm upgrade --install gpu-operator nvidia/gpu-operator --wait \\\n -n gpu-operator \\\n --version=$GPU_OPERATOR_VERSION \\\n -f ./gpu-operator-values.yaml \\\n --timeout 3m\n```\n\nVerify GPU nodes are properly recognized\n```bash\nkubectl get nodes \"-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\\.com/gpu\"\n```\n\nVerify with nvidia-smi\n```bash\nPOD=$(kubectl get pods -n gpu-operator -l app=nvidia-driver-daemonset -o jsonpath='{.items[0].metadata.name}')\nkubectl exec -it $POD -n gpu-operator -- nvidia-smi\n```"}}}} |