The key data structure driving both phases is the KV cache.

What the KV Cache Is

In the attention mechanism, each token attends to every previous token in the sequence. To compute attention for token N, you need the key and value vectors for tokens 1 through N-1. Recomputing those from scratch at each step would mean re-running the full model for all previous tokens every time you generate a new one – O(N^2) compute per sequence.

The KV cache avoids this. After the prefill phase, the key and value vectors for every input token are stored in GPU memory. During the decode phase, each new token appended to the sequence just needs to compute its own K and V vectors and append them to the cache. Attending to all previous tokens is then a lookup into the cached K and V tensors, not a recomputation.

The cost of this: GPU memory. For a 7B parameter model at FP16, the KV cache for a single sequence can be hundreds of megabytes depending on context length, number of layers, and number of attention heads. For larger models serving many concurrent users, the KV cache dominates memory usage.

The Fragmentation Problem

Naive serving allocates KV cache memory statically: reserve a maximum-size block for each sequence upfront. This has the same problem as a badly written memory allocator.

Say you pre-allocate 4096 tokens per sequence and your GPU can hold 20 such blocks. A new request comes in and needs 10 blocks. 18 are free but they’re fragmented across physical memory – no contiguous run of 10. Request rejected, GPU sitting at 60% capacity.

Even worse: you don’t know the output length when a request arrives. If you allocate for the max context length immediately, most of that memory is wasted for short outputs. If you allocate conservatively, long outputs fail mid-generation.

PagedAttention

vLLM’s solution, introduced in the 2023 paper, borrows the virtual memory abstraction from operating systems.

Physical GPU memory is divided into fixed-size blocks (pages), typically 16 tokens each. Each sequence gets a logical block table that maps logical block indices to physical blocks. The blocks for one sequence don’t need to be contiguous in physical memory – the block table handles the translation.

Sequence A: logical [0,1,2] → physical [block_7, block_2, block_15]
Sequence B: logical [0,1]   → physical [block_3, block_11]

Blocks are allocated one at a time as tokens are generated, not all upfront. A short output uses a few blocks and returns them when done. A long output gradually accumulates more. Fragmentation is eliminated because the allocation granularity matches the actual growth pattern of sequences.

The result: GPU memory utilization goes from 20-40% with naive block allocation to 70-90% with PagedAttention. More sequences fit in memory simultaneously, which means higher throughput.

Continuous Batching

PagedAttention works with continuous batching to make the GPU utilization gains practical.

Traditional batching waits until a batch of N requests is assembled, runs inference for the full batch, returns all results. Any sequence that finishes early holds its slot until the full batch is done. Expensive for variable-length outputs.

Continuous batching (also called iteration-level scheduling) processes one decode step at a time across all in-flight sequences. After each step, finished sequences are removed and new requests are inserted into the freed slots. The batch is always full – there’s no waiting.

Combined with PagedAttention, the memory freed by completed sequences is immediately available for new ones. The GPU is busy at every step, not idling through early completions.

RadixAttention

SGLang introduced RadixAttention for workloads with shared prompt prefixes, common in agentic systems and structured generation.

Consider an agent that runs many LLM calls with the same system prompt and tool definitions – often 1000+ tokens. With standard PagedAttention, each request recomputes the KV vectors for those shared tokens during prefill. If you’re making 100 calls with the same 1000-token preamble, that’s 100x the prefill compute for tokens that never change.

RadixAttention maintains a radix tree of cached KV blocks keyed by token sequence prefix. When a new request arrives, it walks the tree to find the longest matching prefix, reuses those cached blocks, and only computes KV vectors for the novel suffix.

For agentic workloads where the same system prompt and tool list appear in every call, the first request pays the full prefill cost. Every subsequent request with the same prefix pays only for its unique portion. At the scale of thousands of agent steps, this is a substantial throughput difference.

The tradeoff: RadixAttention adds cache management overhead. The tree needs to be pruned when memory pressure is high. The eviction policy matters – evicting a heavily shared prefix that gets reused on the next request undoes the benefit. SGLang uses LRU eviction, which works well for stable prefixes but less well for highly dynamic ones.

Choosing Between vLLM and SGLang

The choice comes down to workload:

vLLM is the default for general inference, chat APIs, and throughput benchmarks. PagedAttention and continuous batching are well-tested across a wide range of models and hardware.

SGLang is better for agentic workflows, structured output (constrained decoding for JSON, function calling), and any case where many requests share large prompt prefixes. The RadixAttention benefit only materializes when your prefix reuse rate is high.

Both expose an OpenAI-compatible API and support the same quantization formats. Running both and profiling your actual traffic pattern is the right answer if you’re unsure – the throughput difference on real workloads is usually decisive.

The KV cache is why LLM inference is memory-bound rather than compute-bound during decode. Understanding it makes the rest of the optimization landscape – quantization to shrink the model, PagedAttention to use memory efficiently, RadixAttention to reuse computation – follow naturally from first principles.

The problem was naive serving. Standard HuggingFace pipeline() processes one request at a time, sequentially. The GPU sits idle between requests, waiting. You’re paying for hardware that’s doing nothing most of the time.

The fix is a set of four techniques, each with a different ROI profile. Here’s what they are, how they work, and when to use each one.

Quantization

This is the first thing to do because it’s a one-time change with immediate impact. AWQ 4-bit quantization cuts VRAM usage by roughly 75%. FP8 cuts it by 50% but requires H100s.

That 75% reduction matters because it changes which GPU class you need. A model that required an A100 might now fit on an A6000. That’s not a minor cost reduction – it’s a different tier of hardware entirely.

The tooling is straightforward. Use AutoAWQ to quantize yourself, or just pull a pre-quantized checkpoint from HuggingFace. Most popular models have AWQ variants already uploaded.

Serving Engine

Once you’re quantized, the next question is which serving engine to use. The two serious options are vLLM and SGLang, and they’re not interchangeable.

vLLM is the right choice for chat and general inference. Two mechanisms make the difference:

Continuous batching inserts new requests into the batch as slots open up rather than waiting for a full batch to complete. Naive serving sits at around 20% GPU utilization. vLLM pushes that to 60-85%.

PagedAttention treats VRAM like virtual memory for the KV cache. Instead of pre-allocating contiguous blocks, it pages memory on demand. More sequences can coexist in memory simultaneously, which means more throughput per dollar.

The key vLLM flags worth knowing in production:

1--max-num-seqs 64        # tune for your context length + available VRAM
2--disable-log-stats      # remove logging overhead

SGLang is the right choice for agentic workflows and structured output. Its main advantage is RadixAttention – it automatically detects shared prompt prefixes and reuses their KV cache. If every request in your system starts with the same 1000-token system prompt and tool definitions, SGLang computes that prefix once and caches it. LMSYS benchmarks show higher throughput than vLLM on structured generation workloads.

The rule of thumb: SGLang for agents and JSON, vLLM for chat.

Speculative Decoding

This one is less obvious but gets you 2-3x speedup on long-form generation. The mechanism is a draft-then-verify loop:

1. A small draft model (1-7B parameters) generates 3-12 candidate tokens per step
2. The large target model verifies all candidates in a single parallel forward pass
3. When the draft guesses correctly – which happens 70-90% of the time – you get multiple tokens for the cost of one forward pass

The canonical pairing is Llama-3-8B-Instruct-AWQ as the draft model against Llama-3-70B-Instruct-AWQ as the target.

A full deployment looks like this:

 1import runpod
 2runpod.api_key = "your_api_key"
 3pod = runpod.create_pod(
 4    name="llama3-70b-speculative",
 5    image_name="vllm/vllm-openai:latest",
 6    gpu_type_id="NVIDIA RTX A6000",
 7    gpu_count=2,
 8    docker_args=(
 9        "--model TechxGenus/Meta-Llama-3-70B-Instruct-AWQ "
10        "--quantization awq "
11        "--tensor-parallel-size 2 "
12        "--speculative-model TechxGenus/Meta-Llama-3-8B-Instruct-AWQ "
13        "--num-speculative-tokens 5 "
14        "--max-model-len 8192"
15    )
16)

The critical thing to watch: draft model acceptance rate. Monitor it via Prometheus:

vllm:spec_decode_draft_acceptance_length

If acceptance drops below 0.5 tokens per step, you have the wrong draft model. It adds overhead rather than saving it. The draft model must match the target architecture – vLLM won’t warn you about this at startup.

Deployment Mode

The last decision is serverless vs. pods, and it’s purely a traffic pattern question.

Spiky or unpredictable traffic belongs on serverless – you pay only for actual inference time. Sustained, predictable load is cheaper on pods because you’re not paying the serverless premium on every call.

Neither is universally better. The mistake is running pods for workloads that get 10 requests an hour or paying for serverless when you have a steady 200 RPS.

Why Decode is the Bottleneck

LLM generation has two phases and they’re bound by different things.

Prefill is compute-bound – processing all the input tokens in one parallel forward pass. Decode is memory-bound – generating one token at a time, loading the entire model’s KV cache from VRAM on every single step.

This means VRAM bandwidth determines inter-token latency, not FLOPs. Two GPUs with similar compute specs can have very different inference latency:

That’s a ~4x difference on long-form generation. If you’re choosing hardware for an inference workload and only looking at FLOPs, you’re optimizing the wrong thing.

The practical checklist once you understand this:

• Quantize with AWQ first (or FP8 on H100s)
• Pick SGLang for agents and JSON, vLLM for chat
• Enable speculative decoding for long-form endpoints, monitor acceptance rate
• Wire Prometheus to vllm:gpu_cache_usage_perc to know when you’re VRAM-constrained
• Match deployment mode to your actual traffic shape

The hardware question comes last, not first.