Local RAG on Apple Silicon: End to End Latency for Embed, Rerank, and Generate on M5 Max (2026)

Most local RAG writing benchmarks one thing: the generation model, in tokens per second, in isolation. That is the last stage of a pipeline with four stages in front of it, and on a single Mac those stages are not free and they are not parallel by default. The user does not experience your decode speed. The user experiences the wall clock from pressing enter to reading a grounded answer, and that clock starts running at the embedding call. This post treats the whole chain as one latency budget and measures where the time actually goes when embed, retrieve, rerank, and generate all run on the same M5 Max.

The reason this matters on Apple Silicon specifically is that everything shares one pool of unified memory and one GPU. On a cloud stack you scale each stage independently and the retriever, the reranker, and the generator live on different machines. On a Mac they take turns. An embedding model, a reranker, and a 32B generator all want residency at once, and if they cannot all stay warm you pay a load cost mid request. The pipeline latency is a property of the orchestration, not of any single model.

The Pipeline and How We Timed It

The chain under test is the standard one. A user question is embedded into a query vector. That vector hits a local vector index for top-k retrieval. The candidate passages are reranked by a cross encoder. The top passages are packed into a prompt and sent to the generation model, which streams an answer. We measured the wall time of each stage independently and then the fully composed request, because the composed number exposes contention that the isolated numbers hide.

Hardware was a single M5 Max with 128 GB unified memory, macOS on default power, nothing else contending. The embedding model was a local bge style encoder in MLX, the index was an in process vector store holding roughly 50,000 chunks of about 400 tokens each, the reranker was a small cross encoder, and the generator was a 32B model at 4 bit. We fixed retrieval at top-50 candidates narrowed to top-5 after rerank, and prompts landed around 3,000 tokens after passage packing. Every number below is a median of five runs, and we report time to first token for the generator rather than full completion, because first token is what the user feels as latency and full completion is a throughput question we cover elsewhere.

Where the Time Goes

Stage	Warm median	Cold median	Share of warm budget
Query embedding	40 ms	0.9 s	3%
Vector retrieval (top-50)	25 ms	25 ms	2%
Rerank (50 to 5)	310 ms	1.4 s	22%
Prompt assembly	15 ms	15 ms	1%
Generation to first token	1.0 s	4.2 s	72%
End to end (warm)	1.39 s	6.5 s	100%

Two things stand out immediately. Warm, the generator prefill dominates at roughly three quarters of the budget, and the reranker is a distant but real second. The retrieval step, the one people fixate on when they argue about vector databases, is noise at 25 ms against a 1.4 second budget. Cold, the picture inverts into a mess: every model that was not resident pays a load, and the sum of three cold loads can quadruple the total. The cold column is the real story of local RAG, because on a memory constrained box you do not get to keep all three models warm for free.

The Reranker Is the Quiet Tax

The reranker earns a closer look because it is the stage teams most often add without measuring. A cross encoder scores every query and passage pair, so reranking 50 candidates means 50 forward passes through the reranker, not one. That is why it costs 310 ms warm while embedding a single query costs 40 ms. Push the candidate count to 100 and the rerank stage scales close to linearly while retrieval barely moves. The lever here is candidate count, not reranker model choice, and most pipelines over retrieve out of habit. We went deeper on reranker model tradeoffs on this same hardware in the local reranker benchmark teardown, but the pipeline lesson is simpler: retrieve the smallest candidate set that still surfaces the right passage, because every extra candidate is a full reranker inference.

Generation Prefill Is the Real Bottleneck

The generator dominates the warm budget, but not for the reason people assume. It is not the decode speed. Time to first token is prefill, and prefill scales with prompt length. A RAG prompt is long by construction because you just packed five retrieved passages into it. At 3,000 tokens of context the prefill is doing real work before a single output token appears, and that work is the 1.0 second in the table. Retrieve more passages to improve recall and you lengthen the prompt and lengthen the prefill, which is a direct latency cost that recall focused tuning tends to ignore.

This is where prefix caching changes the math. If a large part of your prompt is stable across requests, a system preamble, tool definitions, a fixed instruction block, you can cache its KV representation and skip re-prefilling it every time. In a RAG setting the retrieved passages change per query, so they cannot be cached, but the scaffolding around them can. We measured what that recovers on Apple Silicon in the prompt caching teardown; for RAG specifically the win is proportional to how much of your prompt is fixed versus retrieved.

# Order the pipeline so the generator loads while cheaper stages run.
async def rag(question, models):
    q_vec   = await models.embed(question)          # 40 ms warm
    cands   = await index.search(q_vec, k=50)       # 25 ms
    # Kick the generator warm-up concurrently with rerank so its load,
    # if it was evicted, overlaps the reranker's 50 forward passes.
    warm    = asyncio.create_task(models.ensure_resident("gen-32b"))
    top5    = await models.rerank(question, cands)[:5]
    await warm
    prompt  = assemble(question, top5)              # stable preamble is prefix-cached
    return models.generate_stream(prompt)           # 1.0 s to first token warm

The Residency Problem

Add up the weights: a small embedder, a small reranker, and a 32B generator at 4 bit sit comfortably in 128 GB with room for KV cache. So on a well sized box the answer is to pin all three resident and never pay a cold load in the hot path. The warm budget is your real budget, and 1.4 seconds to first token for a fully local, fully private RAG answer is a good place to be.

The trouble starts when the generator grows. Swap the 32B for a 70B and you are near the memory ceiling once KV cache on a 3,000 token prompt is included, and the embedder or reranker can get evicted under pressure. Now a request that should cost 40 ms of embedding pays a 0.9 second cold load first, because the OS reclaimed those pages for the generator. The fix is a residency policy that pins the two small, always hit models and treats only the large generator as the swappable one, since it is going to be resident during generation anyway. The small models are cheap to keep and expensive to reload at exactly the wrong moment.

When This Applies to Your Stack

If your RAG runs against a hosted API, none of this is your problem, the provider keeps every stage warm across a fleet and you pay per token instead of per millisecond of load. This becomes your problem the moment privacy, cost at volume, or offline operation pushes the whole pipeline onto local hardware. A support tool that cannot send customer data to a third party, an on premise assistant over internal documents, a field device with no reliable network: these are the cases where local RAG is the requirement, and where the end to end latency budget, not the generation benchmark, decides whether it feels usable.

If your team is standing up a private RAG tier on Apple Silicon and wants the stage budget and residency policy designed before it ships rather than debugged in production, Contra Collective builds these pipelines with the latency contract specified up front. The embedding and reranker choices we cover in the local embeddings study matter, but the composition matters more, because the slowest pipeline is not the one with the slowest model, it is the one that reloads a model in the hot path.

FAQ

Which stage should I optimize first in a local RAG pipeline? Generation prefill, because it dominates the warm budget, and the cheapest way to cut it is to retrieve fewer, better passages so the prompt is shorter. After that, cap your reranker candidate count. Do not spend time optimizing vector retrieval; at 25 ms it is not your problem.

Does the vector database choice matter for latency? On a local single box index of tens of thousands of chunks, barely. Retrieval is a rounding error against generation prefill. The vector store choice matters for recall, memory footprint, and update patterns, not for the latency a user feels on a single query.

Why does reranking cost so much more than embedding? Embedding the query is one forward pass. A cross encoder reranker runs one forward pass per candidate, so reranking 50 candidates is 50 inferences. The cost scales with how many candidates you retrieve, which is why over retrieval is the common and avoidable mistake.

Can I keep all the models warm on one Mac? With a small embedder, a small reranker, and a mid size generator, yes, comfortably on 128 GB. It gets tight with a 70B generator once KV cache is included, at which point you pin the two small models and let only the generator be the one that can be evicted.

Is local RAG first token latency competitive with a hosted API? Warm, a well sized local pipeline reaches first token in well under two seconds, which is competitive for most interactive uses. The gap is cold start: a hosted API hides load behind an always warm fleet, while locally you own it, so your job is to keep the hot path warm.