vLLM vs llama.cpp: Batching Architecture and Production Readiness

The llama.cpp vs vLLM conversation usually ends with "both are fast." That's misleading. They're fast at different things, and choosing between them requires understanding the architectural gap.

llama.cpp is built around the principle: handle one request at a time, do it as fast as possible, keep the memory footprint small. vLLM is built around: accumulate requests into a batch, reorder them, apply optimizations like PagedAttention, maximize throughput even if latency varies. These are fundamentally different design philosophies.

For teams building production inference backends, this difference compounds. Get it wrong, and you're either wasting hardware capacity (choosing llama.cpp for high-volume workloads) or experiencing latency unpredictability (choosing vLLM for single-request scenarios).

Architecture: The Core Difference

llama.cpp: Request Serialization Model

llama.cpp processes requests sequentially. Request 1 comes in, gets loaded into memory, generates tokens, completes. Request 2 starts. If you have 10 concurrent requests queued, llama.cpp queues them and processes them one at a time.

This is intentional. It simplifies memory management, avoids the overhead of batching logic, and guarantees predictable latency: every request gets the same tokens/second throughput regardless of concurrency.

Timeline (llama.cpp, 10 concurrent requests):
T=0ms:  Req1 loads (200ms overhead)
T=200ms: Req1 generates at 80 tok/sec (2500 tokens = 31 sec)
T=31.2s: Req2 loads (200ms overhead)
T=31.4s: Req2 generates (2500 tokens = 31 sec)
...
T=310s: All 10 requests complete

Per-request latency: 31 seconds + queue wait
Aggregate throughput: ~10 tokens/sec (10 requests * 2500 tokens / 310 seconds)

The advantage: simplicity, predictability, and low CPU overhead. The disadvantage: terrible throughput under concurrent load.

vLLM: Continuous Batching Model

vLLM accumulates multiple requests into a single batch and generates tokens for all of them in parallel. When a request finishes early, new requests are inserted into the batch mid-iteration without restarting. This is continuous batching.

Timeline (vLLM, 10 concurrent requests, max batch size 32):
T=0ms:   Requests 1-10 loaded and batched
T=200ms: Batch generates 1 token for all 10 requests simultaneously
T=204ms: Req1 finishes (100 tokens); Req11 inserted into batch
T=400ms: Batch generates 2nd token for all requests
...
T=31s:   All requests complete

Per-request latency: 31 seconds (similar to llama.cpp)
Aggregate throughput: ~3,200 tokens/sec (32 concurrent * 100 tok/sec)

The advantage: high aggregate throughput, efficient hardware utilization, scales naturally with concurrency. The disadvantage: latency is variable (requests inserted mid-batch experience different latencies), requires careful memory management.

Feature Comparison: What This Means in Practice

Real-World Scenario: E-Commerce Product Search

You're building a search backend for a mid-market retailer: 10K concurrent users, each making ~5 search requests per session. That's 50K requests/hour, or ~14 requests/second peak.

Using llama.cpp

llama.cpp queues the 14 requests serially. With a 7B model at 100 tok/second, each 500-token response takes 5 seconds. With 14 requests queued, the last request waits 70 seconds before it even starts. Unacceptable latency.

Option: Run 14 separate llama.cpp instances (one per request stream). Cost: $2,000+/month in infrastructure (14 GPUs or equivalent CPU). Throughput: acceptable. User experience: acceptable.

Using vLLM

vLLM batches all 14 requests. They all generate in parallel, taking 5 seconds total. New requests are inserted continuously as prior ones finish. Cost: 1 GPU or $150/month. Throughput: 14 requests in 5 seconds = 2.8 requests/second sustained. User experience: excellent.

The infrastructure cost gap is the real story: llama.cpp forces you to provision per-request, vLLM forces you to provision per-batch. For concurrent workloads, vLLM wins dramatically.

When llama.cpp Wins

Single-user development tools. If you're building an IDE plugin or terminal tool where one user generates code interactively, llama.cpp's latency predictability matters. You want 80 tokens/second, every time. vLLM's variable latency (50-120 tok/sec depending on batch composition) is a surprise.

Apple Silicon development. llama.cpp's Metal backend is simpler and slightly faster than vLLM's MLX integration. If you're developing locally on an M-series Mac, llama.cpp is more practical.

Minimal dependencies. llama.cpp is a single binary. vLLM requires Python, PyTorch, CUDA or HIP. If you need a zero-dependency inference engine, llama.cpp wins.

Latency SLAs under 100ms. If your contract requires P99 latency under 100ms, vLLM's variable latency makes this harder. With llama.cpp, every request gets the same latency profile. Set your SLA to 1-2x that throughput, and you're guaranteed to hit it.

When vLLM Wins

Concurrent API workloads. Search backends, recommendation APIs, content generation services. Anything where multiple requests arrive within milliseconds of each other. vLLM's continuous batching is built for this.

Cost-sensitive infrastructure. vLLM can serve 10x the throughput of llama.cpp on the same hardware. If you're running on cloud GPUs (which cost money), vLLM's efficiency is worth the operational complexity.

GPU batching is already in your stack. If you're already using vLLM or PyTorch elsewhere, adding vLLM for inference is a natural fit. No new dependencies.

Context-aware batching matters. For scenarios where different requests have different context lengths (e.g., RAG with variable document retrieval), vLLM's PagedAttention handles this more gracefully than llama.cpp's fixed context.

Production Deployment Patterns

Pattern 1: llama.cpp with Queue Sharding

For teams that prefer llama.cpp's simplicity but need higher throughput:

# Run N llama.cpp instances (one per CPU core)
instances = [
    subprocess.Popen([
        'llama-server',
        f'--port {8000 + i}',
        '--ngl 99',  # GPU offloading
    ])
    for i in range(8)
]

# Load-balance requests round-robin
request_queue = i % len(instances)

This gives you ~8x throughput without the complexity of vLLM. Trade-off: 8x infrastructure cost compared to vLLM's single instance.

Pattern 2: vLLM with Request Deduplication

For teams using vLLM, reduce latency variance with deduplication:

from vllm import LLM, SamplingParams

llm = LLM(
    model='mistral-7b-v0.1',
    tensor_parallel_size=1,
    max_model_len=8192,
    gpu_memory_utilization=0.85,
    enable_prefix_caching=True,  # Dedup common prompts
)

# Batch similar requests to maximize cache hits
responses = llm.generate(
    prompts=deduplicated_prompts,
    sampling_params=SamplingParams(temperature=0.7, max_tokens=500),
)

Prefix caching reduces latency variance by 30-50% on workloads with repeated prompt prefixes (common in e-commerce).

Operational Considerations

Monitoring.

• llama.cpp: Monitor request queue depth and per-request latency.
• vLLM: Monitor batch size, latency percentiles (P50 vs P95 will diverge), and GPU memory fragmentation.

Scaling.

• llama.cpp: Scale horizontally (add more instances). Scales linearly until you hit network bottlenecks.
• vLLM: Scale vertically (increase batch size or model size) until you hit GPU memory, then scale horizontally. More efficient but more complex.

Debugging.

• llama.cpp: Single request fails? Look at one process. Simple.
• vLLM: Request fails in a batch of 32? Trace which request, isolate its context, reproduce. Harder.

Migration Path

If you're currently on llama.cpp and need higher throughput:

1. Benchmark your workload. Measure peak concurrent requests. If peak is <4, stay with llama.cpp. If >4, consider vLLM.
2. Pilot vLLM. Run a side-by-side test with vLLM for 1-2 weeks. Measure latency P50/P95/P99, batch size distribution, GPU memory usage.
3. If latency variance is acceptable. Migrate 10% of traffic to vLLM, monitor, expand to 100%.
4. If latency variance is unacceptable. Add request deduplication, prefix caching, or return to llama.cpp with sharding.

FAQ

Q: Can I use both llama.cpp and vLLM in the same infrastructure? A: Yes. Route high-latency-tolerance batch jobs (search reranking, content generation) to vLLM. Route low-latency interactive requests (IDE autocomplete, chat) to llama.cpp. Use a request router to handle both.

Q: Which is faster for a single request? A: llama.cpp is slightly faster (lower request load time, no batching overhead). Difference is 5-15%, negligible for most use cases.

Q: Does vLLM work on Apple Silicon? A: Partially. vLLM's MLX backend exists but is slower than llama.cpp's Metal backend. For M-series Macs, use llama.cpp or MLX directly.

Q: How much latency variance does vLLM introduce? A: Typical range: 50-120 tokens/second on a 7B model depending on batch composition. Use prefix caching and request deduplication to narrow this range.

Q: Can I run both on the same GPU? A: Not simultaneously. They both occupy GPU memory. Either run them on separate GPUs or time-multiplex (alternate between them). Neither is practical.

When This Applies to Your Stack

If you're building a backend service with concurrent requests, vLLM's architecture is worth the operational complexity. If you're building a single-user tool, llama.cpp's simplicity is worth staying with, even if you need to over-provision hardware.

The decision is not about features. It's about request concurrency patterns: are you serving one user or thousands?

Contra Collective has deployed both architectures in production: llama.cpp for agentic development tools, vLLM for search and recommendation backends. If you're evaluating local inference infrastructure for your specific workload, we can help you benchmark both and size the hardware correctly. Internal link: contact for AI infrastructure consulting

Q: Should I use a request queue (like Celery) in front of llama.cpp? A: Only if you're building a service. For development tools, a queue adds latency (good for batching, bad for perceived responsiveness). For APIs, a queue is necessary to handle request bursts.

Q: How does quantization affect the vLLM vs llama.cpp comparison? A: Both benefit equally from quantization (GPTQ, AWQ). The architectural difference (batching vs. serial) remains regardless of quantization format. Choose quantization based on accuracy needs, not runtime.

Q: Can I auto-scale vLLM? A: With difficulty. vLLM requires GPU allocation upfront. Kubernetes auto-scaling for GPU workloads is possible but non-trivial. llama.cpp scales more naturally (add processes, not infrastructure).