LLM Inference in Production 2026: Quantization, KV Cache, Speculative Decoding, and the vLLM vs SGLang Decision

The gap between a demo and a production LLM deployment is measured in dollars per token and milliseconds of latency. A naive inference setup on an A100 costs 5-10x more per query than an optimized one. A poorly configured KV cache turns a 2GB model into a 16GB memory hog. The wrong serving framework means your 99th-percentile latency is 4 seconds instead of 800ms.

I have deployed LLM inference on Azure VMs for three different use cases in the last six months. Here is what actually matters.

The Optimization Stack

Production LLM inference has four optimization layers, each with real trade-offs:

┌──────────────────────────────────────────────┐
│        Layer 4: Serving Framework             │
│  vLLM / SGLang / TensorRT-LLM / llama.cpp    │
├──────────────────────────────────────────────┤
│        Layer 3: Speculative Decoding          │
│  Draft model acceleration                     │
├──────────────────────────────────────────────┤
│        Layer 2: KV Cache Optimization         │
│  GQA / MLA / PagedAttention                   │
├──────────────────────────────────────────────┤
│        Layer 1: Quantization                  │
│  GPTQ / AWQ / GGUF / FP8                     │
└──────────────────────────────────────────────┘

Most teams focus on Layer 1 (quantization) and ignore the rest. The real gains are in Layers 2-4.

Layer 1: Quantization — Which Method, When

Quantization reduces model precision from FP16/BF16 to INT8, INT4, or mixed precision. The result: lower VRAM usage with minimal quality loss.

The three production-grade quantization methods:

My decision rule:

def choose_quantization(model_size_gb: float, gpu_vram_gb: int, quality_critical: bool) -> str:
    """Select quantization method based on deployment constraints."""

    # If the model fits in FP16, do not quantize
    if model_size_gb * 2 <= gpu_vram_gb * 0.8:  # 80% VRAM threshold
        return "fp16"

    if quality_critical:
        # Quality-sensitive workloads (RAG, code generation, reasoning)
        if model_size_gb * 0.5 <= gpu_vram_gb * 0.8:
            return "awq"  # Best quality-to-compression ratio
        return "gptq"  # Fallback for tight VRAM

    if gpu_vram_gb >= 80:
        # Datacenter GPU — GPTQ is fine
        return "gptq"

    if gpu_vram_gb <= 24:
        # Consumer/prosumer GPU — need maximum compression
        return "gguf-q4_k_m"  # Best 4-bit quant for mixed CPU/GPU

    return "gptq"

The FP8 option: NVIDIA H100/H200 GPUs support FP8 natively. If you are on H-series hardware, FP8 gives you 2x throughput over FP16 with negligible quality loss. It is not quantization in the traditional sense — it is native hardware precision.

# Example: Deploy a 70B model on 2xH100 with AWQ
vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \
  --quantization awq \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --dtype auto

Layer 2: KV Cache Optimization

The KV (Key-Value) cache stores the attention state for all previous tokens. For a 70B model with 8192 context length, the KV cache alone can consume 16-24GB of VRAM. This is often the bottleneck, not the model weights.

Three KV cache techniques in production:

Grouped Query Attention (GQA)

Most modern models (Llama 3, Qwen 3, DeepSeek) use GQA by default. Multiple query heads share the same key/value heads, reducing KV cache size by 4-8x without quality loss.

# Check if your model uses GQA
from transformers import AutoConfig
config = AutoConfig.from_pretrained("meta-llama/Llama-3.1-70B-Instruct")
print(f"Num KV heads: {config.num_key_value_heads}")  # 8 (vs 64 query heads)
print(f"GQA ratio: {config.num_attention_heads // config.num_key_value_heads}")  # 8x reduction

Multi-head Latent Attention (MLA)

DeepSeek's MLA compresses KV cache into a low-rank latent representation. DeepSeek V4 and V3 use this, achieving 10x KV cache compression over standard MHA.

The trade-off: MLA requires a custom attention kernel. Not all serving frameworks support it. vLLM added MLA support in v1.x; SGLang has experimental support.

PagedAttention

vLLM's signature innovation. Instead of allocating contiguous VRAM for each sequence's KV cache (which wastes memory on fragmented sequences), PagedAttention uses block-based allocation — the same concept as virtual memory pages.

Traditional KV Cache:
[Seq 1: 8192 tokens, 16GB contiguous] [waste: 4GB gap] [Seq 2: 2048 tokens, 4GB]

PagedAttention:
[Block 1: Seq1] [Block 2: Seq1] [Block 3: Seq2] [Block 4: free]
→ No fragmentation, 30-50% better memory utilization

Practical impact: PagedAttention typically allows 2-4x more concurrent requests per GPU by eliminating KV cache fragmentation. This is the single biggest throughput lever for production deployments.

Layer 3: Speculative Decoding

The idea: use a small, fast "draft" model to generate candidate tokens, then verify them in parallel with the large model. If the draft model predicts correctly (which it does 70-90% of the time for common patterns), you get those tokens at draft-model speed.

# vLLM with speculative decoding
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --tensor-parallel-size 4

When speculative decoding helps: - High-throughput scenarios (many concurrent requests) - Draft model is fast relative to target model (8B draft for 70B target) - Tasks with predictable token patterns (code completion, structured output)

When it does not help: - Low-latency, single-request scenarios (the draft model adds overhead) - Creative generation (draft predictions are less accurate) - Small models where the draft model is not significantly faster

My benchmark results on Azure ND96amsr_A100_v4 (8xA100-80GB):

The 50-75% throughput improvement from speculative decoding is real, but it requires careful tuning of the num_speculative_tokens parameter.

Layer 4: The Serving Framework Decision

This is the most consequential choice. vLLM, SGLang, TensorRT-LLM, and llama.cpp each have distinct strengths.

vLLM (v1.x)

The default choice for most production deployments. PagedAttention, continuous batching, tensor/pipeline parallelism, and the broadest model support.

Strengths: - Widest model support (HuggingFace format out of the box) - PagedAttention for maximum memory efficiency - OpenAI-compatible API - Active community and rapid development - Multi-modal support (vision, audio) in v1.x

Weaknesses: - Higher single-request latency than SGLang - MLA support is newer and less battle-tested - Complex configuration for optimal performance

SGLang

Optimized for low-latency single requests and structured output. The SGLang runtime compiles attention kernels ahead-of-time for maximum performance.

Strengths: - Fastest single-request latency in benchmarks - Excellent structured output (JSON, regex constraints) - RadixAttention for prefix caching (reuses KV cache across requests with shared prefixes) - Smaller memory footprint

Weaknesses: - Narrower model support than vLLM - MLA support is experimental - Smaller community

TensorRT-LLM

NVIDIA's inference engine. Maximum performance on NVIDIA hardware but requires model conversion.

Strengths: - Highest throughput on NVIDIA GPUs - FP8 native support on H100/H200 - In-flight batching for maximum GPU utilization

Weaknesses: - Model conversion step adds complexity - NVIDIA-only (no AMD, Intel support) - Longer startup time (kernel compilation)

llama.cpp

CPU and edge inference. GGUF format, runs on anything from a Raspberry Pi to a Mac Studio.

Strengths: - Runs on CPU, Apple Silicon, and any GPU - GGUF format for easy distribution - Minimal resource requirements - Perfect for edge and development

Weaknesses: - Lower throughput than GPU-optimized frameworks - Limited multi-GPU support - Not suitable for high-concurrency production

Decision Matrix

My recommendation: Start with vLLM for general-purpose serving. Switch to SGLang if you need low-latency structured output. Use TensorRT-LLM only if you need maximum NVIDIA throughput and can handle the conversion complexity. Use llama.cpp for edge and development.

Practical Azure Deployment

Here is a production-ready deployment on Azure for a 70B model:

# Azure ND96amsr_A100_v4 (8xA100-80GB)
# Deploy with vLLM + AWQ quantization + tensor parallelism

pip install vllm

vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \
  --host 0.0.0.0 \
  --port 8000 \
  --quantization awq \
  --tensor-parallel-size 4 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 64 \
  --dtype auto \
  --trust-remote-code

# Health check
curl http://localhost:8000/health

# Benchmark
python -m vllm.entrypoints.openai.api_client \
  --model meta-llama/Llama-3.1-70B-Instruct-AWQ \
  --base-url http://localhost:8000

Key parameters explained: - --tensor-parallel-size 4: Split model across 4 GPUs (70B AWQ fits in 4x80GB) - --gpu-memory-utilization 0.92: Leave 8% headroom for KV cache growth - --enable-chunked-prefill: Process long prompts in chunks to avoid latency spikes - --max-num-seqs 64: Maximum concurrent requests (tune based on latency requirements)

What Surprised Me

1. KV cache is the bottleneck, not model weights. For most workloads, the KV cache consumes more VRAM than the model itself at inference time. Optimizing KV cache (PagedAttention, GQA) has more impact than quantization.
2. Speculative decoding is underused. Most teams do not know it exists. The 50-75% throughput improvement is real and requires minimal configuration.
3. Framework choice matters more than hardware choice. The same model on the same GPU with different frameworks can show 2-3x throughput difference.
4. AWQ beats GPTQ in almost every metric. Better quality, faster inference, easier deployment. GPTQ is legacy at this point.
5. Structured output is a killer feature. SGLang's constrained decoding for JSON/schema output eliminates post-processing overhead and guarantees valid output format.

Key Takeaways

1. Quantize with AWQ for GPU inference, GGUF for CPU/edge. AWQ gives the best quality-to-compression ratio; GGUF is the only option for non-NVIDIA hardware.
2. KV cache optimization (PagedAttention, GQA) gives the biggest throughput gains. Invest here before exploring more exotic optimizations.
3. Start with vLLM. It has the broadest support, the best documentation, and PagedAttention. Switch to SGLang or TensorRT-LLM only when you have a specific performance requirement.
4. Speculative decoding is a free 50-75% throughput boost. Use it for high-throughput scenarios with a small draft model.
5. The right serving framework on the same hardware can double your throughput. Benchmark before committing to a deployment architecture.

Resources

• vLLM Documentation
• SGLang Documentation
• TensorRT-LLM
• llama.cpp