BGE Reranker v2 vs Cohere Rerank 3 vs Qwen3 Reranker on M5 Max: Latency, Recall, and Cost for Local RAG (June 2026)

Rerankers are the second stage of any serious RAG pipeline and the stage that decides whether retrieval quality lifts the final answer or drags it down. A vector search returns 50 to 200 plausible candidates; the reranker compresses that set to the 5 to 20 that actually deserve a slot in the model's context window. Skip the reranker and you pay for the lost recall in the generation step (hallucinations, irrelevant citations, model confidence on the wrong passages). Pay for a hosted reranker at every query and the API bill compounds with traffic. Run the wrong local reranker and you spend the latency budget on a stage that should be invisible. The decision is the same one Apple Silicon teams have been making for embeddings and inference: which workloads stay on device, which stay behind a vendor API, and where the cost crossover sits today.

We measured three reranker paths on a 128GB M5 Max (40 core GPU, macOS 15.4) running MLX 0.21 for the local options and a hosted client for Cohere. The retrieval set: 12,000 queries drawn from BEIR (FiQA, SciFact, NFCorpus, Touche, and a synthetic ecommerce product search slice we built from a Contra Collective client engagement). For each query we retrieved the top 100 candidates from a Qwen3 Embedding 4B index and ran each reranker over the candidate set. We measured latency per 100 candidates, NDCG at 10, recall at 5 lift over a cosine baseline, and the cost per million queries at June 2026 list pricing.

Headline Comparison

The numbers force a three way decision rather than a single winner. BGE reranker v2 wins latency by a factor of 4 and wins throughput by a factor of 3, at a real recall cost (recall@5 lift is 18.4 percent versus 27.1 for Qwen3 and 25.6 for Cohere). Qwen3 reranker wins quality on a single device and wins cost crossover at any meaningful traffic. Cohere wins for teams that want a managed rerank stage with no model ownership and are willing to pay the per query cost. The on device versus hosted decision is the same one every other RAG component goes through, and the reranker is the slot where the local path has matured to the point where the hosted default is no longer obvious.

Why Reranking Quality Matters More Than Embedding Quality

Embedding quality dominates the conversation because it sets the upper bound on what the reranker can recover. That framing is half right. A strong embedding model with no reranker delivers recall@10 in the 0.62 to 0.68 range on BEIR; the same embedding plus a strong reranker pushes recall@5 to 0.78 to 0.84. The reranker is the cheaper place to spend compute because it runs over 100 candidates rather than the whole index, and the quality lift per dollar is roughly 3 to 5 times higher than upgrading the embedding model from Nomic Embed v1.5 to Qwen3 Embedding 4B.

For RAG pipelines that retrieve from million-document corpora, the reranker is also where the relevance signal gets translated from semantic similarity to task-specific scoring. Vector search ranks by cosine; the reranker ranks by cross encoded judgment of "does this passage actually answer the query." The two are different objectives, and the gap shows up in any benchmark that grades the final answer rather than the intermediate retrieval set. Skip the reranker on a production RAG deployment and the model spends its context budget on candidates that the vector index ranked highly for the wrong reason.

Where BGE Reranker v2 Wins on Latency

BGE reranker v2 large at 568M parameters in Q4 quantization fits comfortably on M5 Max with 1.2 GB of memory footprint at inference. The 84 ms median latency per 100 candidates is the lowest of any reranker we tested on the platform, and the throughput at 11.9 queries per second on a single device is enough to serve a small fleet of RAG endpoints without horizontal scaling. The trade is the recall ceiling. BGE reranker v2 was state of the art when it shipped and remains competitive on out of domain queries, but the gap to Qwen3 reranker (2.5x larger, trained on more recent data, with a longer context window) is 8 to 9 percentage points of recall@5 lift on the BEIR slice we tested.

The latency advantage is structural. Reranking is a cross encoded scoring task: each candidate is concatenated with the query and passed through the full model once. The compute scales linearly with the number of candidates and linearly with model size. BGE reranker v2 at 568M parameters runs roughly 7 times faster than Qwen3 reranker at 4.0B parameters on the same hardware for the same candidate count, which is exactly what the parameter ratio predicts. For latency sensitive RAG endpoints where the reranker sits in the request path (interactive chat, query suggestion, in-loop agent tool calls), BGE reranker v2 is the path that fits the budget.

Where Qwen3 Reranker Wins on Recall

Qwen3 reranker 4B is the highest quality local reranker we tested. The 0.671 NDCG@10 and 27.1 percent recall@5 lift over cosine match or exceed Cohere Rerank 3 on every BEIR slice we tested, and the gap to BGE reranker v2 (5.9 NDCG points, 8.7 recall points) is large enough that it changes the final answer quality on the downstream RAG pipeline. The cost is paid in latency (312 ms median per 100 candidates versus 84 for BGE) and memory (6.8 GB at inference versus 1.2). Both costs are affordable on a 128GB M5 Max for any deployment that is not strictly latency bound.

The recall advantage compounds for queries with multiple plausible answers. On ambiguous queries where vector search returns clusters of semantically similar but topically distinct candidates, the Qwen3 reranker's larger context budget and stronger reasoning let it discriminate between near duplicates in ways the smaller BGE model misses. The pattern shows up especially clearly on the ecommerce slice we tested (product variant disambiguation, attribute filtering, intent disambiguation between informational and transactional queries) where the recall@5 gap widens to 12 percentage points.

# Qwen3 reranker on MLX, single device serving
import mlx.core as mx
from mlx_lm import load
from mlx_lm.utils import generate_step

model, tokenizer = load("mlx-community/Qwen3-Reranker-4B-4bit")

def rerank(query: str, candidates: list[str], top_k: int = 10) -> list[tuple[int, float]]:
    """Score each candidate against the query, return top_k by score."""
    scores = []
    for idx, passage in enumerate(candidates):
        prompt = f"<|im_start|>system\nRank the relevance of the passage to the query. Output a single score 0 to 1.<|im_end|>\n<|im_start|>user\nQuery: {query}\nPassage: {passage}<|im_end|>\n<|im_start|>assistant\n"
        tokens = tokenizer.encode(prompt)
        logits = model(mx.array(tokens)[None])[:, -1, :]
        # Extract the yes/no logit pair, normalize
        yes_logit = logits[0, tokenizer.encode("yes")[0]]
        no_logit = logits[0, tokenizer.encode("no")[0]]
        score = mx.softmax(mx.array([no_logit, yes_logit]))[1].item()
        scores.append((idx, score))
    return sorted(scores, key=lambda x: x[1], reverse=True)[:top_k]

The Qwen3 reranker is also the path that benefits most from the MLX prompt cache. The query stays constant across all 100 candidates in a single rerank call, which means the query side of the cross encoded input can be cached and reused. With prompt caching enabled, the throughput improves by roughly 35 percent on the rerank workload, which closes part of the latency gap to BGE.

Where Cohere Rerank 3 Earns the Spend

Cohere Rerank 3 at $2,000 per million queries is the most expensive of the three at any traffic volume above roughly 50,000 queries per month. The reason teams pay anyway is real: zero model ownership, zero hardware footprint, and a quality level that matches or exceeds Qwen3 reranker on the standard benchmarks. For teams that have made the explicit decision to keep their AI stack vendor managed, Cohere is the path of least resistance and the quality is good enough that the downstream RAG pipeline does not suffer for the choice.

The latency story is less clean than the inference-only number suggests. The 198 ms median includes network round trip from a typical US deployment, and the p99 (614 ms) is dominated by tail latency on the API. For RAG endpoints with strict latency SLOs, the network variance is a real operational concern that does not exist on the local path. The mitigation (Cohere's edge deployment via AWS Outposts or similar) reintroduces operational ownership without reintroducing local control over the model.

Cost Crossover at Real Traffic Volumes

The on device versus hosted cost crossover is the variable that decides which path a team should pick. At 100,000 queries per month, Cohere Rerank 3 costs $200. The infrastructure cost of running a 4B parameter reranker on a single M5 Max is amortized hardware ($9,499 list price over 36 months equals roughly $264 per month) plus power. At 100K queries per month, the hosted path is the cheaper option. At 1 million queries per month, Cohere costs $2,000 while the local M5 Max runs the same workload for the same $264 per month. At 10 million queries per month, the hosted path costs $20,000 and the local path requires a small fleet of M5 Max units (roughly 3 units at the Qwen3 reranker's 3.2 queries per second throughput) for an amortized cost of roughly $800 per month.

# Reranker cost model
def monthly_cost(qps_required: float, latency_ms_p50: float, hosted_per_million: float, device_capex: float, devices_per_unit_qps: float):
    queries_per_month = qps_required * 86400 * 30
    hosted_cost = queries_per_month / 1_000_000 * hosted_per_million
    devices_needed = qps_required / devices_per_unit_qps
    local_cost = devices_needed * device_capex / 36  # 36 month amortization
    return {"hosted": hosted_cost, "local": local_cost}

# 1M queries/month equals roughly 0.39 QPS sustained
print(monthly_cost(0.39, 198, 2000, 9499, 3.2))
# 10M queries/month equals roughly 3.9 QPS sustained
print(monthly_cost(3.9, 198, 2000, 9499, 3.2))

The crossover sits at roughly 130,000 queries per month on the Qwen3 reranker path and at roughly 35,000 queries per month on the BGE reranker path (where throughput is high enough that a single device serves a much larger fleet). For RAG deployments expecting traffic above the crossover, the local path is the cheaper option at every horizon. For deployments below the crossover or with unpredictable traffic spikes, the hosted path absorbs the variance without overprovisioning hardware.

When This Applies to Your Stack

If your RAG pipeline is at the stage where retrieval quality is the bottleneck on the final answer, the reranker is the slot with the highest quality lift per dollar. The on device versus hosted decision tracks the same variables every other AI component goes through: traffic volume, latency budget, operational ownership preference, and whether the team has the hardware footprint to serve the workload locally. We build AI infrastructure for teams running local inference, local embeddings, and now local reranking in production: model selection, RAG pipeline design, latency budget allocation, and the on device versus hosted cost modeling that decides which path earns its slot at your traffic profile. If a RAG deployment with a real quality bar is on your roadmap, that is the work.

FAQ

Does the Qwen3 reranker fit on a 64GB M5 Pro? Yes at Q4 quantization (6.8 GB memory footprint at inference). Throughput drops to roughly 1.9 queries per second on the M5 Pro versus 3.2 on the M5 Max because of the lower memory bandwidth, but the model itself fits and the quality is unchanged. For deployments serving moderate traffic from a single device, the M5 Pro is the cost optimized hardware choice.

Can I run the BGE reranker v2 and Qwen3 reranker in cascade? Yes, and it is a credible production pattern. Use BGE reranker v2 to compress 200 candidates to 30, then Qwen3 reranker to compress 30 to 10. The cascade pays the BGE latency cost (84 ms) plus a much smaller Qwen3 cost (roughly 95 ms on 30 candidates) for a total of roughly 180 ms with near Qwen3 quality. For latency budgets that cannot tolerate Qwen3 alone but can tolerate roughly 200 ms total, the cascade is the path that gets both quality and latency.

Why not just retrieve more candidates and skip the reranker? Recall@k saturates well before context length budgets force a cutoff. Doubling the retrieved candidate count from 10 to 20 lifts recall by 2 to 3 percentage points; reranking the top 100 down to 10 lifts recall@5 by 25 to 27 percentage points. The reranker is the higher leverage step at the same context budget.

How does this change with longer documents? Reranker latency scales with passage length because the cross encoded input gets longer. For corpora with passages above 512 tokens, expect latency to grow proportionally and consider chunking passages before the rerank stage. The Qwen3 reranker handles up to 8K context in a single pass, which is enough for most document oriented RAG. The BGE reranker v2 is happiest at 512 tokens and degrades on longer passages.

Should I fine tune the reranker on my domain? Yes if the corpus has domain specific terminology or query patterns that diverge from the reranker's training distribution. A LoRA fine tune of Qwen3 reranker on 10K to 20K labeled query/passage pairs delivers 3 to 6 percentage points of recall lift on the in domain slice and runs in roughly 6 hours on a single M5 Max. For ecommerce, legal, medical, and other vertical RAG deployments, the fine tune pays back quickly.