Deploying Distributed vLLM Clusters on Kubernetes with GPU Auto-scaling

Here is our production GitOps setup for managing high-availability GPU inference nodes across multiple Kubernetes clusters using KEDA and the NVIDIA GPU Operator.

Key Components

1. Node Provisioning: Spot GPU instances with Karpenter on AWS/GCP with automated preemption drain.
2. Metrics-Based HPA: KEDA listening to vLLM's Prometheus metric vllm:num_requests_waiting.
3. Graceful Connection Draining: Envoy Gateway terminating TLS and redistributing queued streaming requests before Pod termination.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-autoscaler
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 2
  maxReplicaCount: 16
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting{namespace="ai-prod"})
      threshold: '5'

This setup dropped our p99 token time-to-first-token (TTFT) by 65% during sudden traffic surges.