How to Deploy LLMs in Production: vLLM, TGI, and Triton Inference Server

Deploying an LLM in a Jupyter notebook is trivial. Deploying it to serve thousands of concurrent users with consistent latency and 99.9% uptime is an entirely different challenge.

This guide compares the three leading LLM serving frameworks and shares production deployment patterns from our work at Cloudpush.

The LLM Serving Challenge

LLM inference is fundamentally different from traditional ML serving:

• Memory-bound: A 7B parameter model needs ~14GB in FP16, ~28GB in FP32
• Autoregressive: Each token depends on all previous tokens, limiting parallelism
• Variable latency: Response time depends on output length, not just input length
• GPU-intensive: CPU-only serving is impractically slow for most use cases
• Stateful: KV-cache grows with sequence length, consuming memory dynamically

Framework Comparison

vLLM

vLLM's key innovation is PagedAttention, which manages KV-cache memory like virtual memory in an operating system. This eliminates memory fragmentation and dramatically improves throughput.

Strengths:

• Highest throughput for most workloads (2-4x over naive implementations)
• Excellent continuous batching
• PagedAttention for efficient memory management
• Built-in support for tensor parallelism across GPUs
• Active open-source community

Best for: High-throughput serving, multi-user applications, API services

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,           # 4 GPUs
    max_model_len=8192,
    gpu_memory_utilization=0.90,
    enable_prefix_caching=True,       # Cache common prefixes
    max_num_seqs=256,                 # Max concurrent requests
)

# Serve with OpenAI-compatible API
# vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4

Text Generation Inference (TGI)

Hugging Face's TGI is a production-ready solution with excellent Docker support and a focus on safety and compliance.

Strengths:

• Simple Docker-based deployment
• Built-in safety features (watermarking, content filtering)
• Excellent Hugging Face ecosystem integration
• Flash Attention and paged attention support
• Grammar/JSON output constraints

Best for: Teams already in the Hugging Face ecosystem, compliance-heavy environments

docker run --gpus all \
  -p 8080:80 \
  -v /models:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-70B-Instruct \
  --num-shard 4 \
  --max-input-tokens 4096 \
  --max-total-tokens 8192 \
  --max-batch-prefill-tokens 16384

NVIDIA Triton Inference Server

Triton is NVIDIA's enterprise-grade inference server, designed for multi-model serving and heterogeneous workloads.

Strengths:

• Multi-model serving (LLMs, vision models, embeddings on same instance)
• Ensemble pipelines (chain preprocessing → model → postprocessing)
• Advanced scheduling and batching
• Enterprise support from NVIDIA
• TensorRT-LLM backend for maximum NVIDIA GPU performance

Best for: Multi-model deployments, enterprise environments, maximum GPU utilization

# Model repository structure for Triton
# models/
#   llama-70b/
#     config.pbtxt
#     1/
#       model.plan  # TensorRT engine

Performance Comparison

Production Deployment Patterns

Pattern 1: Single-Model API Service

The simplest pattern — one model behind a load balancer:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-service
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "--model=meta-llama/Llama-3.1-8B-Instruct"
          - "--gpu-memory-utilization=0.9"
          - "--max-model-len=4096"
        resources:
          limits:
            nvidia.com/gpu: 1
      nodeSelector:
        gpu-type: a100-80gb

Pattern 2: Model Router

Route requests to different models based on complexity:

class ModelRouter:
    def __init__(self):
        self.fast_model = vLLMClient("llama-8b-endpoint")    # Simple queries
        self.smart_model = vLLMClient("llama-70b-endpoint")   # Complex queries
        self.classifier = ComplexityClassifier()

    async def route(self, request: ChatRequest) -> ChatResponse:
        complexity = await self.classifier.classify(request.messages)

        if complexity == "simple":
            return await self.fast_model.chat(request)
        else:
            return await self.smart_model.chat(request)

This can reduce costs by 60-70% while maintaining quality on complex queries.

Pattern 3: Speculative Decoding

Use a small model to draft tokens and a large model to verify:

# vLLM supports speculative decoding natively
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_model="meta-llama/Llama-3.1-8B-Instruct",
    num_speculative_tokens=5,
)

Speculative decoding can improve latency by 2-3x for compatible model pairs.

Auto-Scaling for LLM Workloads

LLM workloads are bursty and GPU-bound, requiring specialized auto-scaling:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-service
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Pods
    pods:
      metric:
        name: gpu_utilization
      target:
        type: AverageValue
        averageValue: "75"
  - type: Pods
    pods:
      metric:
        name: request_queue_depth
      target:
        type: AverageValue
        averageValue: "10"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 2
        periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300

Key Auto-Scaling Considerations

• Scale on queue depth, not CPU: GPU utilization is a lagging indicator
• Slow scale-down: Model loading takes 30-120 seconds, so keep warm replicas
• Pre-warming: Maintain minimum replicas to handle baseline traffic without cold starts
• GPU-aware scheduling: Use Kubernetes GPU device plugin and node affinity

Performance Optimization Checklist

1. Enable continuous batching: Groups requests together for GPU efficiency
2. Use prefix caching: Cache system prompts and common prefixes
3. Set appropriate max_model_len: Don't allocate for 128K context if you only need 4K
4. Use quantization: AWQ or GPTQ 4-bit reduces memory 4x with minimal quality loss
5. Enable Flash Attention: Reduces memory and improves speed for long contexts
6. Use tensor parallelism: Split large models across GPUs within a node
7. Monitor and tune gpu_memory_utilization: 0.85-0.95 is the sweet spot
8. Streaming responses: Return tokens as they're generated for better perceived latency

Monitoring in Production

Track these metrics:

• Time to First Token (TTFT): How long until the first token is generated
• Inter-Token Latency (ITL): Time between subsequent tokens
• Throughput: Total tokens per second across all requests
• GPU utilization: Should be 70-90% under normal load
• Queue depth: Number of pending requests
• Error rate: Failed requests due to OOM, timeouts, or model errors
• KV-cache utilization: Percentage of allocated KV-cache in use

Getting Started

The right serving framework depends on your scale, team expertise, and infrastructure. For most teams starting out, vLLM offers the best balance of performance and simplicity. For enterprise multi-model deployments, Triton with TensorRT-LLM is the gold standard.

Contact Cloudpush for expert help deploying LLMs in production with optimized infrastructure and monitoring.