Here's a number that should change how you think about AI infrastructure: 80% of AI GPU spend is now inference, not training. If your team is running LLMs in production — whether through Azure OpenAI APIs, self-hosted vLLM, or a hybrid of both — the overwhelming majority of your compute bill goes to serving models, not building them.

I've been tracking this shift across Malaysian enterprises since late 2025, and the pattern is consistent. Organisations that deployed their first LLM proof-of-concept for RM5,000–10,000 in API costs are now looking at RM50,000–200,000/month bills as they scale to production. The good news: the inference optimization stack has matured dramatically. The techniques that were research papers 18 months ago are now production-ready, and they stack together to deliver genuinely transformative cost reductions.

This article walks through the complete optimization layer cake — from quantization at the model level to prompt caching at the API level — with real numbers, real code, and a decision framework specifically tuned for the constraints Malaysian enterprises face.

The Cost Problem: Why Inference Optimization Is Now a Board-Level Concern

Before we dive into solutions, let's ground the problem in real numbers. Using current Azure OpenAI pricing (June 2026):

ModelInput (per 1M tokens)Cached InputOutput (per 1M tokens)
GPT-5.4 Global$2.50$0.25$15.00
GPT-5.4 mini Global$0.75$0.08$4.50
GPT-5.1 Global$1.25$0.13$10.00
GPT-4.1 Global$2.00$0.50$8.00
GPT-4.1 mini Global$0.40$0.10$1.60

Now consider a typical enterprise use case: a document processing pipeline that ingests 10,000 documents/day, each requiring ~2,000 input tokens and generating ~500 output tokens. At GPT-5.4 pricing, that's roughly $75/day in output tokens plus $50/day on the input side — about RM550/day or RM16,500/month. For a mid-sized Malaysian enterprise, that's a non-trivial budget line item that demands optimization.

The playbook I'm about to walk through has allowed teams I work with to cut these bills by 5-10x while maintaining or improving quality. It works through five optimization layers, each compounding the gains of the ones beneath it.

Layer 1: Quantization — Shrinking Models Without Shrinking Capabilities

Quantization is the foundational layer. It reduces model weight precision from 16-bit floating point (FP16/BF16) to 8-bit, 4-bit, or even lower, cutting memory requirements by 2-4x with surprisingly minimal quality degradation.

The Quantization Spectrum

Here's what the benchmarks actually show for a Llama 3.1 70B model (illustrative figures from community GGUF testing):

QuantizationModel SizeQuality (MMLU)Inference SpeedVRAM Required
FP16 (baseline)140 GB82.11.0x2x A100 80GB
Q8_070 GB81.81.3x1x A100 80GB
Q4_K_M40 GB81.21.8x1x A100 80GB
Q4_038 GB80.51.9x1x A100 80GB
Q2_K26 GB77.32.1x1× RTX 5090 32GB (partial offload)

The key insight: Q4_K_M gives you near-FP16 quality at half the model size and nearly 2x the inference speed. For most enterprise tasks — RAG, summarization, classification — the quality difference between Q8 and Q4 is practically undetectable.

Practical Quantization with llama.cpp

# Quantize a HuggingFace model to GGUF Q4_K_M
# Requires: llama.cpp build with CUDA support

# Step 1: Convert to GGUF
python convert_hf_to_gguf.py \
    meta-llama/Llama-3.1-70B-Instruct \
    --outfile llama-3.1-70b-f16.gguf \
    --outtype f16

# Step 2: Quantize to Q4_K_M
./llama-quantize \
    llama-3.1-70b-f16.gguf \
    llama-3.1-70b-q4_k_m.gguf \
    Q4_K_M

# Step 3: Serve with llama.cpp
./llama-server \
    -m llama-3.1-70b-q4_k_m.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 99 \  # offload all layers to GPU
    -c 8192 \  # context length
    -t 8       # CPU threads

When to Use What

  • Production RAG pipelines: Q4_K_M on a single A100 — best cost/quality ratio
  • Customer-facing chat: Q8_0 for maximum quality retention
  • Edge deployment / on-premise: Q4_0 on RTX 4090s or even smaller quantizations on laptop GPUs for internal tools
  • Maximum throughput, accept some degradation: Q3_K_M or Q2_K for high-volume classification tasks

For Malaysian enterprises with on-premise GPU infrastructure (common in banking and government), quantization is often the single highest-impact optimization. Going from FP16 to Q4 lets you serve a 70B model on a single GPU instead of two, halving your infrastructure cost immediately.

Layer 2: KV Cache Compression — Unlocking Longer Contexts

The KV (Key-Value) cache is the memory that LLMs use to maintain attention over conversation history. For a 70B model with a 4K context window, the KV cache alone consumes ~2GB of VRAM. At 128K context, that balloons to ~64GB — more than most single GPUs can handle.

Grouped Query Attention (GQA)

Modern models like Llama 3.1 and Qwen 2.5 use GQA instead of Multi-Head Attention (MHA). Instead of each attention head having its own key and value projection, groups of query heads share a single key-value pair.

The impact: 4-8x reduction in KV cache size with negligible quality loss. Llama 3.1 70B uses 8 KV heads (vs. 64 query heads), giving an immediate 8x reduction.

PagedAttention with vLLM

vLLM's PagedAttention brings OS-style virtual memory management to the KV cache. Instead of pre-allocating contiguous memory blocks for each sequence (which wastes memory on variable-length outputs), it allocates small "pages" on demand.

# vLLM with PagedAttention (enabled by default)
from vllm import LLM, SamplingParams

# Serve a Q4-quantized model with automatic KV cache management
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    quantization="awq",          # or "gptq"
    tensor_parallel_size=2,       # split across 2 GPUs
    max_model_len=32768,          # 32K context
    gpu_memory_utilization=0.90,  # 90% of GPU memory to KV cache
    enable_prefix_caching=True,   # cache common prefixes
)

# Process multiple requests with efficient KV cache sharing
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=2048,
)

prompts = ["Summarize this document: ..."] * 100  # batch of requests
outputs = llm.generate(prompts, sampling_params)

The practical result: vLLM achieves up to 24x higher throughput than HuggingFace Transformers (and 2.2-3.5x higher than HuggingFace TGI) under high-concurrency workloads, primarily through efficient KV cache management. For a Malaysian fintech processing thousands of customer queries per minute, the difference between naive serving and vLLM is the difference between needing 8 GPUs and needing 1.

Layer 3: Continuous Batching — Maximizing GPU Utilisation

Traditional "static batching" waits for all requests in a batch to complete before processing the next batch. If one request generates 50 tokens and another generates 500, the GPU sits idle for the duration of the shorter request while waiting for the longer one.

Continuous batching (also called iteration-level scheduling) inserts and removes requests at every decoding step, keeping the GPU fully utilised.

The Throughput Impact

Representative community benchmarks illustrate the difference clearly (exact figures vary by model, hardware, and workload):

Batching MethodThroughput (tok/s)GPU Utilisationp99 Latency
Static batching (HF TGI)4235%12.4s
Continuous batching (vLLM v0.6)28589%4.1s

That's a 6.8x throughput improvement from batching alone, with latency actually decreasing because requests aren't queuing behind each other.

vLLM Continuous Batching in Practice

# vLLM server with continuous batching optimized for throughput
# Run as a service
# vllm serve meta-llama/Llama-3.1-70B-Instruct \
#   --tensor-parallel-size 2 \
#   --max-num-seqs 64 \
#   --max-num-batched-tokens 8192 \
#   --enable-prefix-caching

# Client code
import openai

client = openai.OpenAI(
    base_url="http://your-vllm-server:8000/v1",
    api_key="not-needed",
)

# vLLM serves as an OpenAI-compatible endpoint
# All requests benefit from continuous batching automatically
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[{"role": "user", "content": "Analyse this financial report..."}],
    temperature=0.7,
    max_tokens=1024,
)

Layer 4: Speculative Decoding — The Biggest Latency Breakthrough

Speculative decoding is the technique that made me stop dismissing "academic" inference optimizations. A small, fast "draft" model proposes multiple tokens simultaneously, then the large "target" model verifies them in a single forward pass. Correct proposals are accepted; incorrect ones trigger a rollback to the draft model's output at that position.

The critical property: speculative decoding is lossless. The output is mathematically identical to running the large model alone — you're just getting there faster.

Real Benchmark Numbers

Snowflake's Arctic Inference team published the most comprehensive benchmarks on vLLM (Llama 3.1 70B, 8×H100 node):

WorkloadNo SpeculationN-gramEAGLELSTM + Suffix (Arctic)
ShareGPT (chat)76 tok/s91 tok/s102 tok/s179 tok/s
HumanEval (code)77 tok/s100 tok/s112 tok/s217 tok/s
SWE-Bench (agents)76 tok/s175 tok/s—302 tok/s

That's a 2.3x-4x speedup depending on workload, with zero quality degradation. For agentic workloads (the fastest-growing inference category), the gains are the largest because agent tasks contain highly repetitive patterns that draft models exploit efficiently.

Deploying Speculative Decoding with vLLM

# Deploy Llama 3.1 70B with Arctic LSTM speculative decoding
pip install "git+https://github.com/snowflakedb/ArcticInference.git#egg=arctic-inference[vllm]"

vllm serve \
    meta-llama/Llama-3.1-70B-Instruct \
    --quantization "fp8" \
    --tensor-parallel-size 2 \
    --speculative-config '{
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true
    }'

The draft model is tiny — typically 0.5-2 billion parameters — so it adds negligible memory overhead while dramatically accelerating the target model. For enterprises running inference at scale, this is essentially free performance.

N-gram Speculation for Quick Wins

If you don't want to train or deploy a draft model, vLLM's N-gram speculative decoding works out of the box — it looks for repeating token patterns in the input prompt and uses them as draft proposals:

# Zero-config N-gram speculation
# Just add to your vLLM config:
# --speculative-config '{"method": "ngram", "num_speculative_tokens": 4}'

Even this lightweight approach yields 1.2-2.3x speedups on code-heavy workloads where variable names and patterns repeat frequently.

Layer 5: Prompt Caching — The API-Level Cost Killer

If you're using managed APIs (Azure OpenAI, OpenAI, Anthropic), prompt caching is the single easiest cost reduction available. It requires zero infrastructure changes and delivers 40-85% input cost reductions for workloads with repeated prompt prefixes.

How It Works

When a request contains a prefix identical to a previously cached request (same model, same first 1024+ tokens), the provider charges only 10% of the normal input token rate for the cached portion. On Azure OpenAI, cached input tokens for GPT-5.4 cost $0.25/1M instead of $2.50/1M.

Real Cost Impact

Consider a RAG pipeline with a 3,000-token system prompt (instructions + retrieved context) and a 200-token user query:

ScenarioInput TokensStandard CostCached CostSavings
10K requests/day, 50% cache hit3,200 × 10K$80.00/day$44.00/day45%
10K requests/day, 80% cache hit3,200 × 10K$80.00/day$22.40/day72%
50K requests/day, 80% cache hit3,200 × 50K$400.00/day$128.00/day72%

That last row represents a RM11,520/month saving — from changing zero lines of code, just ensuring your prompts share a common prefix.

Azure OpenAI Prompt Caching Implementation

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="your-key",
    api_version="2024-12-01-preview",
    azure_endpoint="https://your-resource.openai.azure.com/",
)

def process_with_caching(user_query: str, context: str) -> str:
    # The system prompt and context MUST be identical prefix for caching
    # Place static content FIRST — caching works left-to-right
    response = client.chat.completions.create(
        model="gpt-5.4-global",
        messages=[
            {
                "role": "system",
                "content": f"""You are a compliance assistant for Malaysian banking regulations.
                
Standards: {context}
                
Follow these rules exactly:
1. Always cite the specific regulation section
2. Flag any conflicts with BNM Guidelines
3. Rate compliance risk as Low/Medium/High/Critical"""
            },
            {"role": "user", "content": user_query}
        ],
        temperature=0.3,
        max_tokens=1024,
    )
    
    # Check cache hit rate in response
    usage = response.usage
    if hasattr(usage, 'prompt_tokens_details'):
        cached = usage.prompt_tokens_details.cached_tokens or 0
        print(f"Cache hit: {cached}/{usage.prompt_tokens} tokens")
    
    return response.choices[0].message.content

The critical implementation detail: place your long, static content at the beginning of the prompt. Caching works left-to-right through the token stream. If your system prompt varies per request, you won't get cache hits.

The Decision Framework: Self-Host vs. API for Malaysian Enterprises

This is where it gets practical. The right choice depends on your scale, data sensitivity, and team capabilities.

Cost Comparison: Azure OpenAI vs. Self-Hosted vLLM

FactorAzure OpenAI (GPT-5.4 mini)Self-Hosted vLLM (Llama 3.1 70B Q4)
Input cost (per 1M tokens)$0.75~$0.09*
Output cost (per 1M tokens)$4.50~$0.24*
With prompt caching$0.08 inputN/A (local)
Infrastructure cost$0~$2.50/hr (H100) or ~$1.00/hr (A100)
Setup complexityMinutesDays to weeks
Data residencyAzure Malaysia West (✅ PDPA)Full control (✅)
MaintenanceZeroOngoing (model updates, GPU failures)
Break-evenBest <50M tokens/monthBest >50M tokens/month

*Self-hosted costs assume 2&times;A100 80GB at $2/hr combined serving ~274K output tokens/hr at a 76 tok/s baseline (&asymp;$0.09/1M input, &asymp;$0.24/1M output); speculative decoding roughly doubles throughput at similar cost. Raw GPU throughput alone is not the whole equation &mdash; factor in engineering, monitoring, model updates, and failure handling when comparing paths.

When to Choose Each Path

Choose Azure OpenAI when:

  • Monthly token volume is under 50M tokens
  • You need rapid deployment (days, not weeks)
  • Your team lacks ML infrastructure expertise
  • You need guaranteed SLAs and no ops burden
  • Data can leave your premise (Malaysia West region covers PDPA requirements)

Choose self-hosted vLLM/llama.cpp when:

  • Monthly volume exceeds 100M tokens (the economics flip decisively)
  • Data sovereignty is non-negotiable (some government and financial sectors)
  • You need model customisation (fine-tuning, custom system prompts baked into weights)
  • You're running high-frequency, low-latency workloads (sub-200ms p99)
  • You already have GPU infrastructure for training

The hybrid approach (increasingly common in Malaysian enterprises): Use Azure OpenAI for customer-facing applications that need SLAs and rapid scaling, self-host open models for internal high-volume processing pipelines where you control the data.

Stacking the Layers: The 10x Cost Reduction in Practice

The real power comes from combining all five layers. Here's a real scenario I helped architect for a Malaysian financial services company processing KYC documents:

Baseline: Azure OpenAI GPT-5.4, 100K documents/month, ~30K input and ~10K output tokens per document across multiple review passes.

  • Monthly cost: ~$7,500 (input) + ~$15,000 (output) = $22,500/month

After optimization stack:

  1. Self-hosted Llama 3.1 70B Q4 on 2x A100 (on-premise, existing GPUs): $0 marginal compute
  2. vLLM continuous batching: 6.8x throughput improvement → handled same load
  3. Speculative decoding (EAGLE): 2.3x latency reduction → same throughput with 1 GPU
  4. Prompt caching (system prompt reuse): 45% input reduction on remaining API fallback traffic
  5. Quantized model quality maintained at Q4_K_M (verified on internal eval set)

Optimised monthly cost: ~$350 (GPU electricity) + $500 (Azure fallback for edge cases) = ~$850/month

That's a 26x reduction — from $22,500 to $850. Even in less aggressive scenarios where you keep Azure OpenAI as primary and just add prompt caching + batching, you're looking at 3-5x reductions from day one.

Key Takeaways

  1. Quantization is table stakes, not a compromise. Q4_K_M delivers 98%+ of FP16 quality on enterprise workloads. If you're still serving unquantized models in production, you're paying 2-4x more than necessary for no measurable quality gain.
  1. Speculative decoding is the most underutilised optimization. The technology is lossless, adds negligible overhead, and delivers 2-4x speedups — especially on the agentic workloads that are driving the next wave of enterprise AI. Every production inference deployment should evaluate it.
  1. Prompt caching is free money for API users. Restructure your prompts to put static content first, and you'll immediately save 40-85% on input costs with zero code changes. This is the first thing every Azure OpenAI customer should implement.
  1. Self-hosting becomes cost-effective at scale, but don't underestimate the ops burden. For most Malaysian enterprises under 100M tokens/month, Azure OpenAI with prompt caching and Batch API pricing (50% discount) is the pragmatic choice. Go self-hosted when data sovereignty demands it or volume makes it economically compelling.
  1. The optimization layers compound. Each layer builds on the previous one. A quantized model with KV cache optimization, continuous batching, speculative decoding, and prompt caching on API fallback can deliver 10-25x cost reductions. The playbook exists — the gap is adoption.