GPU Infrastructure for AI: Kubernetes, Scheduling, and Cost Optimization

GPU infrastructure is the foundation of every production AI system. Get it right, and your models serve thousands of users with low latency at reasonable cost. Get it wrong, and you burn through cloud budgets while your models sit idle or your users wait in queues.

This guide covers how we build GPU infrastructure for AI workloads at Cloudpush.

GPU Selection for AI Workloads

Current GPU Landscape

GPU	VRAM	FP16 TFLOPS*	Best For	Cloud Cost/hr
B200	192GB	2,250	Next-gen training + inference	$5-7
GB200 (Grace Blackwell)	384GB	4,500	Largest models, multi-modal	$8-12
H100 SXM	80GB	989	Training + large model inference	$3-4
H100 PCIe	80GB	756	Inference at scale	$2.5-3.5
A100 80GB	80GB	312	General purpose AI	$1.5-2.5
A100 40GB	40GB	312	Medium models, fine-tuning	$1-2
L40S	48GB	362	Inference-optimized	$1-1.5
A10G	24GB	125	Small model inference	$0.5-1

*FP16 TFLOPS figures are with sparsity. Dense FP16 performance is approximately half.

Selection Guidelines

Training at scale: B200 or GB200 for the latest generation performance and memory capacity
Training: H100 SXM with NVLink for multi-GPU training
Large model inference (70B+): B200, H100, or A100 80GB with tensor parallelism
Medium model inference (7B-13B): A100 40GB or L40S
Small model inference (under 7B): A10G for cost efficiency
Embedding models: A10G or even CPU for small embedding models

Kubernetes for GPU Workloads

NVIDIA Device Plugin

The NVIDIA device plugin enables Kubernetes to schedule GPU workloads:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin
  template:
    spec:
      containers:
      - name: nvidia-device-plugin
        image: nvcr.io/nvidia/k8s-device-plugin:v0.15.0
        securityContext:
          privileged: true
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins

GPU Node Pools

Separate GPU nodes from CPU nodes for efficient resource management:

# GPU node pool for inference
apiVersion: v1
kind: Node
metadata:
  labels:
    gpu-type: a100-80gb
    workload-type: inference
    node-pool: gpu-inference

# GPU node pool for training
apiVersion: v1
kind: Node
metadata:
  labels:
    gpu-type: h100-sxm
    workload-type: training
    node-pool: gpu-training

For smaller workloads, share GPUs between multiple pods:

# NVIDIA MIG (Multi-Instance GPU) configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-config
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-3g.40gb:
        - devices: all
          mig-enabled: true
          mig-devices:
            3g.40gb: 2  # Split A100 80GB into 2x 40GB instances

MIG is ideal for serving multiple small models on a single A100.

Resource Requests and Limits

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: llm-server
    resources:
      requests:
        nvidia.com/gpu: 1
        memory: "32Gi"
        cpu: "8"
      limits:
        nvidia.com/gpu: 1
        memory: "64Gi"
        cpu: "16"
  nodeSelector:
    gpu-type: a100-80gb
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

Cost Optimization Strategies

GPU compute is expensive. Here's how to reduce costs by 40-60%:

1. Right-Size Your GPUs

Don't use an H100 for a model that fits on an A10G. Profile your model's actual memory and compute requirements:

def estimate_gpu_memory(model_params_b: float, precision: str = "fp16") -> float:
    bytes_per_param = {"fp32": 4, "fp16": 2, "int8": 1, "int4": 0.5}
    model_memory_gb = model_params_b * bytes_per_param[precision]
    # Add ~20% overhead for KV-cache and framework
    return model_memory_gb * 1.2

2. Spot/Preemptible Instances

Use spot instances for training and non-critical inference:

Training: Checkpoint frequently and resume on interruption
Batch inference: Process queued requests on spot instances
Dev/staging: Always use spot for non-production environments

Savings: 60-70% compared to on-demand pricing.

3. Auto-Scaling to Zero

For low-traffic services, scale to zero when idle:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-scaler
spec:
  scaleTargetRef:
    name: llm-deployment
  minReplicaCount: 0      # Scale to zero
  maxReplicaCount: 5
  cooldownPeriod: 300       # 5 min before scaling down
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus:9090
      metricName: http_requests_pending
      threshold: "1"
      query: sum(http_requests_pending{service="llm"})

4. Quantization

4-bit quantization reduces GPU memory requirements by 4x with minimal quality loss:

Model	FP16 Memory	INT4 Memory	Quality Loss
Llama 3.1 8B	16GB	4GB	Under 1%
Llama 3.1 70B	140GB	35GB	1-3%
Mixtral 8x7B	90GB	23GB	1-2%

5. Multi-Tenancy

Serve multiple customers on shared infrastructure with isolation:

class MultiTenantRouter:
    def __init__(self):
        self.model_pool = SharedModelPool()
        self.tenant_configs = {}

    async def serve(self, request, tenant_id):
        config = self.tenant_configs[tenant_id]
        model = self.model_pool.get_model(config.model_name)

        # Apply tenant-specific system prompt and guardrails
        response = await model.generate(
            messages=request.messages,
            system_prompt=config.system_prompt,
            max_tokens=config.max_tokens,
            guardrails=config.guardrails
        )
        return response

MLOps Pipeline

CI/CD for Models

# Model deployment pipeline
stages:
  - validate:
      - run_evaluation_suite
      - check_safety_benchmarks
      - verify_latency_requirements
  - stage:
      - deploy_to_staging
      - run_integration_tests
      - run_load_tests
  - canary:
      - deploy_canary_10_percent
      - monitor_metrics_30_minutes
      - compare_with_baseline
  - production:
      - gradual_rollout
      - monitor_and_alert

Model Registry

Track model versions, training runs, and deployment status:

class ModelRegistry:
    def register(self, model_name, version, metadata):
        # Store model artifact reference
        # Record training hyperparameters
        # Store evaluation metrics
        # Track deployment status
        pass

    def promote(self, model_name, version, stage):
        # Move model from staging to production
        # Trigger deployment pipeline
        pass

Monitoring GPU Infrastructure

Key Metrics

gpu_metrics:
  - nvidia_gpu_utilization         # Target: 70-90%
  - nvidia_gpu_memory_used         # Alert if >95%
  - nvidia_gpu_temperature         # Alert if >85°C
  - nvidia_gpu_power_usage         # Track for cost allocation

inference_metrics:
  - request_latency_p50            # Target: <200ms
  - request_latency_p99            # Target: <2000ms
  - throughput_tokens_per_second
  - queue_depth                    # Alert if >50
  - error_rate                     # Alert if >1%

cost_metrics:
  - gpu_cost_per_request
  - gpu_idle_percentage            # Target: <20%
  - spot_instance_interruptions

Getting Started

Building GPU infrastructure for AI is a specialized discipline that combines DevOps, ML engineering, and cost optimization. The right architecture depends on your model sizes, traffic patterns, latency requirements, and budget.

Contact Cloudpush for expert help designing and managing GPU infrastructure for your AI workloads.