vLLM PagedAttention: Batching and KV-Cache Optimization for High-Throughput Local Inference

If you've benchmarked vLLM against Ollama or llama.cpp on the same hardware and wondered why vLLM consistently achieves 2x to 5x higher throughput, the answer is PagedAttention. It's not magic, but it's a genuinely clever memory optimization that changes how you reason about local inference at scale.

Most LLM inference runtimes treat the key-value (KV) cache as a monolithic block that must be allocated upfront. This wastes memory on padded sequences and fragments the cache whenever a batch ends. PagedAttention solves this by treating the KV-cache like virtual memory: it's fragmented into fixed-size logical pages that can be reassigned on the fly. This single shift enables batching patterns that were impossible before.

For teams building agentic systems, high-volume inference workloads, or cost-optimized e-commerce search backends on modest hardware, understanding PagedAttention is the difference between "local LLM" as a curiosity and "local LLM" as a legitimate production tier.

The KV-Cache Problem vLLM Solved

In a standard transformer, the KV-cache grows linearly with sequence length: each token you generate adds a new row to the key and value matrices. For a 7B model generating a 2K-token response, that's a substantial memory footprint. Most inference runtimes allocate enough space upfront to handle the worst case: maximum sequence length, maximum batch size, maximum number of tokens to generate. Most of that space stays empty.

Worse, when a batch request finishes early, the allocated memory for that sequence persists in the cache even though you're no longer using it. A batch of 32 requests where 30 finish in 10 tokens and 2 finish in 500 tokens means your cache is fragmented and you've wasted memory slots for the early finishers.

llama.cpp handles this with a simple strategy: allocate a single buffer, pack sequences tightly, and defragment when a sequence ends. Effective, but serial. Ollama uses a similar approach with per-process isolation. Both work, neither scales gracefully to mixed-length batches.

How PagedAttention Works

PagedAttention divides the KV-cache into fixed-size logical pages (typically 16 tokens per page). Each sequence maintains a list of page indices, not a contiguous memory block. When a sequence needs a new KV entry, the runtime assigns an available page; when it ends, pages return to the free pool.

Standard KV-Cache Allocation:
Sequence 1 (100 tokens): [████████...empty...] (padded to 512)
Sequence 2 (50 tokens):  [██...empty...empty...] (padded to 512)
Total memory used: ~1GB for 2 sequences

PagedAttention with Pages (16 tokens/page):
Sequence 1: [Page 5, 12, 33, 67]
Sequence 2: [Page 2, 8, 44]
Free pages: [1, 3, 4, 6, 7, 9, 10, ...]
Total memory used: ~80MB for 2 sequences (12.5% of standard)

This is not a theoretical advantage. On an M4 Pro with 36GB unified memory, vLLM can batch 128 concurrent requests at mixed sequence lengths where Ollama maxes out at 4. The math is straightforward: fragmentation and padding overhead drops from 60%+ to under 5%.

The second win is continuous batching. vLLM can accept new requests mid-iteration and insert them into the current batch without waiting for existing requests to complete. llama.cpp and Ollama don't support this natively; they finish a batch, then start the next one. The latency difference for high-concurrency workloads is dramatic: sub-millisecond insertion vs. 100ms+ batch boundary delays.

Practical Configuration for Local Inference

PagedAttention is enabled by default in vLLM, but you control the memory footprint via three knobs:

Max model length: The maximum sequence length you'll ever request (default: model's trained context window). Set this explicitly if you're using a 32K-context model but never query past 4K; vLLM allocates only what you need.

vllm serve mistral-7b-v0.1 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 1

GPU memory utilization: How aggressively vLLM should pack the cache into available memory (0.0 to 1.0, default 0.9). At 0.95+, you're squeezing the last few percent at the cost of latency variance. At 0.7, you get predictable latency but lower throughput.

Block size (page size): vLLM defaults to 16 tokens per page. Smaller pages (8 tokens) reduce internal fragmentation but increase page table overhead. Larger pages (32 tokens) waste more space on short sequences but reduce bookkeeping. Leave this at 16 unless profiling shows otherwise.

For a 7B model on an M4 Pro (36GB unified memory), this configuration:

vllm serve mistral-7b-v0.1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --enable-chunked-prefill

Achieves approximately:

• Batch size: 32 concurrent requests
• Throughput: 180 tokens/second across the batch
• Per-request latency: 800ms (for a 512-token generation)
• Memory: ~28GB (leaving 8GB headroom)

llama.cpp at the same quantization level handles 1 concurrent request at 120 tokens/second. The throughput delta is real.

When PagedAttention Dominates; When It Doesn't

PagedAttention shines in three scenarios:

1. High-concurrency batch inference. You have 50+ requests queued and need to maximize aggregate throughput. vLLM's continuous batching model wins decisively. Ollama and llama.cpp are single-request systems that queue sequentially.

2. Mixed-length generation. Sequences finish at different times (some 100 tokens, others 2K). vLLM's logical paging prevents fragmentation; traditional allocators waste 40%+ of the cache on padding.

3. Long context windows. Using a 32K or 100K model? PagedAttention makes this practical on modest hardware. Without it, you're forced to quantize aggressively or scale to cloud GPUs.

Where it doesn't matter as much:

Single-request, fixed-length use cases. If you're building a local LLM as a single-user development tool (one request at a time, fixed generation length), llama.cpp's simplicity and lower dependency overhead might be preferable. The latency is identical; throughput is irrelevant.

Very small models on very tight memory budgets. On a machine with 8GB total memory running a 3B model, you're memory-bound regardless of the optimization. vLLM's overhead (Python runtime, worker processes) can actually be worse than llama.cpp's minimal C++ binary.

Integration Patterns for Production

vLLM ships with an OpenAI-compatible API server, making it a drop-in replacement for OpenAI SDK code:

from openai import OpenAI

client = OpenAI(
    api_key="token-here",
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="mistral-7b-v0.1",
    messages=[{"role": "user", "content": "Classify this product..."}],
    temperature=0.7,
    max_tokens=500,
)

For production e-commerce use cases (product search, recommendation ranking, content generation), vLLM can run as a sidecar service on a dedicated GPU node or even distributed across a small cluster using tensor-parallel serving.

The key constraint: vLLM expects NVIDIA or AMD GPUs. Apple Silicon support exists but is experimental and significantly slower than on GPU hardware. For local development on an M-series Mac, use ollama or llama.cpp; for production, allocate a proper GPU.

Evaluating vLLM for Your Stack

Ask these questions:

1. Do you have more than 4 concurrent inference requests? If yes, vLLM wins on throughput. If no, overhead outweighs benefit.
2. Are you generating variable-length outputs? vLLM's paging model excels. Fixed-length batches are indifferent between runtimes.
3. Do you need an OpenAI-compatible API endpoint? vLLM provides it; llama.cpp requires extra tooling (llama-server). Perplexity doesn't.
4. Is your hardware GPU-based? vLLM shines on CUDA/AMD. On Apple Silicon or older CPUs, llama.cpp is more practical.

FAQ

Q: Can I use PagedAttention with open-source models? A: Yes. PagedAttention is architecture-agnostic. Any transformer-based model (Mistral, Llama, Qwen, etc.) benefits. It doesn't require model-specific training or fine-tuning.

Q: What's the memory overhead of vLLM vs. llama.cpp? A: vLLM's Python + worker overhead is ~500MB. For a 7B model (6GB quantized + vLLM overhead = 6.5GB), that's ~8% overhead. For a 70B model (45GB + 0.5GB), it's <2%. llama.cpp is lighter (~50MB), but the gap narrows with larger models.

Q: Does PagedAttention work with quantized models? A: Fully. GPTQ, AWQ, GGUF quantized models all benefit from PagedAttention's memory fragmentation reduction. Quantization + PagedAttention is the optimal pairing for local inference.

Q: Can I use vLLM on Apple Silicon in production? A: Technically yes, but it's not recommended. vLLM's Metal backend is slower than the same model on llama.cpp or MLX. For production on Apple Silicon, use llama.cpp or MLX. For development, vLLM works fine.

When This Applies to Your Stack

If you're building a local inference backend for e-commerce search, recommendation ranking, or content generation, and you need to handle more than 4 concurrent requests without spinning up cloud infrastructure, vLLM with PagedAttention is the right choice. The throughput gains translate directly to cost savings and latency reduction.

If you're running single-request workloads (one user, one browser tab, synchronous generation), stick with llama.cpp or Ollama. Simpler, lighter, equally fast.

The decision is architectural: are you optimizing for aggregate throughput or simplicity? PagedAttention is the answer to the first question.

Contra Collective has built production vLLM deployments for search backends, recommendation systems, and autonomous agent pipelines. If you're evaluating local inference infrastructure, we can help you navigate the trade-offs and size the hardware correctly. Internal link: contact for AI infrastructure consulting

Q: Is vLLM's batching worth the operational complexity? A: For high-concurrency workloads, absolutely. The throughput gains (2x to 5x) justify the operational overhead. For low-concurrency or prototyping, no. Start with Ollama or llama.cpp; graduate to vLLM when you hit batch size limits.

Q: How do I monitor vLLM's cache utilization in production? A: vLLM exposes Prometheus metrics at /metrics. Monitor vllm_cache_block_manager_usage (cache utilization %), vllm_request_latency_ms (per-request latency), and vllm_num_inflight_requests (current batch size).

Q: Can I mix vLLM with other inference runtimes? A: Yes. Run vLLM for high-throughput batch workloads; run llama.cpp for single-user dev tools. Use request routing to send traffic to the appropriate backend based on concurrency patterns.