Running a Local Reranker on an M5 Max: Cross Encoder Latency for RAG and Search (2026)

A vector index answers a query with a lookup. A reranker answers it with fresh model compute, once per candidate, every single time. That difference is the whole reason to think carefully before you drop a reranking stage into a live retrieval pipeline. Vector search precomputes document embeddings offline and compares them to a single query embedding at request time, so its cost is nearly fixed no matter how large the corpus grows. A cross encoder reranker does the opposite: it takes the query and one candidate document, concatenates them, and runs a full forward pass to produce a relevance score, then repeats that for the next candidate and the next. The accuracy jump is real, because the model sees the query and document together and can judge actual relevance rather than embedding proximity. The cost is also real, and it lands in the request path. This is a measurement of that cost on an M5 Max, so you can decide how many candidates to rerank and whether to rerank at all.

Why a Cross Encoder Costs What It Costs

The reranker's price is structural, not incidental. A bi encoder, the model behind your vector index, embeds the query once and compares that vector against precomputed document vectors with a cheap dot product. All the expensive work happened when you built the index. A cross encoder cannot precompute anything about the pairing, because relevance depends on the query, which it only sees at request time. So it pays a full model pass for every candidate it scores. If you retrieve fifty candidates from the first stage and rerank all of them, that is fifty forward passes standing between your user and their answer.

This is why reranking is almost always a second stage and never the first. You use fast vector search or lexical search to pull a wide candidate set from the whole corpus, then hand a small slice of that set to the reranker to reorder by true relevance. The first stage decides recall, the reranker decides precision at the top. The choice of reranker model, whether bge-reranker-v2, Cohere Rerank, or a Qwen3 based reranker, is the accuracy half of the decision and we covered that split in the reranker model comparison. This post is the other half: given a competent local reranker, what does running it actually cost per query, and how does that cost scale with the knob you control most directly, the number of candidates you feed it.

What Reranking Adds to the Request Path

These are directional figures from a 128GB M5 Max running a 560M parameter cross encoder reranker under MLX, scoring query and candidate pairs truncated to 512 tokens each, measured as added wall clock latency at the p50 of a warm model. They show the shape of the cost, not universal constants, because the numbers move with reranker size, sequence length, and how aggressively you batch the candidate passes.

Candidates reranked	Batched passes	Added latency p50 (ms)	Notes
10	1 batch	34	negligible against most retrieval budgets
25	1 batch	71	still comfortable for interactive search
50	2 batches	138	the common default, feel it on slow queries
100	4 batches	271	precision gain shrinks, latency does not
200	8 batches	540	rarely worth it, reorders mostly noise

Read the latency column against the relevance it buys, because that is the entire trade. Reranking the top ten or twenty five candidates is close to free on this hardware and reliably sharpens the ordering that pure vector similarity gets wrong. Pushing to fifty is the common production default and still defensible, though on a slow query it is a latency you can feel. Beyond that the curve turns against you: the marginal candidate at rank eighty was already ranked low by the first stage for a reason, and reranking it rarely promotes it into the answer, so you pay linear latency for a precision gain that has mostly flattened. The knee for most retrieval workloads sits somewhere between twenty five and fifty candidates, and past it you are spending milliseconds to reorder documents the user will never see.

Batching Is the Setting That Moves the Number

The single most important operational choice is whether you score candidates one at a time or in a batch. Scoring serially runs the reranker like a for loop, one forward pass returning before the next begins, and the fixed per pass overhead dominates. Batching feeds many query and candidate pairs through the model together, amortizing the weight read across the whole batch the same way batching helps any decode bound workload on Apple Silicon. For a reranker the effect is large, because every pair shares the identical model weights and the only thing that differs is the input tokens.

# mlx cross encoder reranking, batched over the first stage candidates
import mlx.core as mx
from reranker import load_reranker  # your loaded cross encoder

model, tokenizer = load_reranker("mlx-community/bge-reranker-560m")

def rerank(query, candidates, top_k=5, batch_size=16):
    pairs = [(query, doc) for doc in candidates]
    scores = []
    for i in range(0, len(pairs), batch_size):
        chunk = pairs[i:i + batch_size]
        toks = tokenizer.batch_encode(chunk, max_length=512, truncation=True)
        logits = model(mx.array(toks["input_ids"]))   # one pass for the whole chunk
        scores.extend(logits.squeeze(-1).tolist())
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_k]

Two details in that loop matter more than they look. The first is truncation to a fixed maximum length. A cross encoder pass costs in proportion to sequence length squared for the attention component, so one candidate with a 4K document dragged into the batch stretches the whole batch to that length and wastes compute on padding for every shorter pair. Truncate documents to a sane window before reranking, and if your documents are long, rerank on the retrieved chunk rather than the full parent document. The second is batch size itself: too small and you are back to serial overhead, too large and you spike memory and blow past the latency budget on a single query. On an M5 Max a batch of sixteen to thirty two pairs at 512 tokens sits in the sweet spot for interactive reranking.

Where the Reranker Sits Relative to Retrieval

The reranker's latency does not exist in a vacuum; it stacks on top of first stage retrieval and, in a RAG setup, in front of generation. If your vector search returns in fifteen milliseconds and your reranker adds a hundred and forty, the reranker is now the dominant cost of retrieval, and in a full pipeline that also embeds the query and then generates an answer, it is one term among several. We traced the full embed, rerank, generate path and where each stage lands in the local RAG pipeline latency writeup; the point worth repeating here is that reranking is the stage whose cost you control most directly, because candidate count is a dial and generation length usually is not.

That gives you a clean tuning strategy. Set the first stage to over retrieve generously, because vector search is cheap and recall at this stage is what protects you from missing the right document entirely. Then set the rerank count to the smallest number that still captures the relevant candidates, which is almost always far below the first stage retrieval count. Retrieving a hundred and reranking twenty is a common and sensible shape. The query embedding itself, if you are running local embeddings, adds its own fixed cost that we measured in the local embeddings post, and it runs before retrieval rather than after, so it does not compound with candidate count.

When to Skip Reranking Entirely

A reranking stage is not free insurance, and there are cases where the honest engineering answer is to leave it out. The first is when first stage retrieval is already precise enough for the task. If your corpus is small, your embeddings are well tuned to the domain, and top k vector results are consistently right, a reranker adds latency to reorder documents that were already in the correct order. Measure your retrieval quality before assuming you need a second stage; sometimes better embeddings or a hybrid lexical plus vector first stage closes the gap the reranker was meant to fill, at no request time cost.

The second case is a hard latency budget where the reranker's added milliseconds break the experience. Autocomplete, type ahead search, and any interaction expected to feel instant often cannot absorb a hundred plus milliseconds of reranking, and there the right move is to invest in first stage quality and skip the cross encoder. The third case is a corpus small enough that you could rerank everything, which sounds like an argument for reranking but is usually an argument against a retrieval pipeline at all: below a few hundred documents you can often score the whole set once and cache, rather than standing up two stages. Reranking earns its complexity when the corpus is large enough that the first stage must be lossy and precise ordering at the top genuinely matters.

When This Applies to Your Stack

If you are running retrieval on Apple Silicon and relevance at the top of the results is what your users judge you on, a local cross encoder reranker is one of the highest leverage additions you can make, provided you treat its latency as a budget rather than an afterthought. Over retrieve cheaply in the first stage, rerank the smallest candidate set that captures the right documents, batch the candidate passes rather than scoring serially, truncate documents to a fixed window so one long candidate does not stretch the batch, and measure the added latency at your real candidate count instead of assuming it is negligible. Find your knee, which for most workloads lands between twenty five and fifty candidates, and stop there.

This is the kind of tuning that decides whether a local retrieval stack feels sharp and fast or accurate but sluggish, and it is squarely the AI infrastructure work Contra Collective does when a team wants search or RAG running on their own hardware rather than rented by the query. The reranker is where relevance is won, and the request path is where it can quietly be lost.

FAQ

What is the difference between a reranker and a vector search index? A vector index embeds documents once, offline, and compares them to a query embedding with a cheap similarity calculation at request time, so its cost is nearly fixed. A reranker is a cross encoder that feeds the query and each candidate document through a full model pass together at request time, one pass per candidate, producing a more accurate relevance score at a cost that scales with candidate count. They are complementary: vector search for recall, reranking for precision at the top.

How many candidates should I rerank? The smallest number that reliably captures the relevant documents, which for most workloads sits between twenty five and fifty. Beyond that the precision gain flattens because low ranked candidates rarely get promoted, while latency keeps climbing linearly. Over retrieve widely in the first stage to protect recall, then rerank a small slice of that set.

Does batching actually help a reranker? Yes, significantly. Scoring candidates serially pays fixed per pass overhead on every one, while batching amortizes the weight read across many query and candidate pairs in a single pass. On an M5 Max a batch of sixteen to thirty two pairs at 512 tokens is a good target. Truncate documents to a fixed length first so one long candidate does not stretch the whole batch.

Can a local reranker match a hosted reranking API on accuracy? The strong open cross encoder models are competitive with hosted rerankers for most retrieval tasks, and running locally removes the per query network round trip and cost. The relevant comparison is model quality against your domain, which we cover in the reranker model comparison; the latency you save by staying local is often larger than people expect, because a hosted rerank call adds network time on top of compute.

When should I not use a reranker at all? Skip it when first stage retrieval is already precise enough for your corpus and domain, when a hard latency budget like autocomplete cannot absorb the added milliseconds, or when the corpus is small enough to score once and cache rather than running two stages. A reranker earns its place when retrieval must be lossy over a large corpus and precise top ordering is what users judge.