Local Embeddings on Apple Silicon: nomic-embed vs bge-m3 vs Qwen3-Embedding on M5 Max (June 2026)

Embeddings are the unglamorous half of every RAG stack, and the model choice quietly determines whether retrieval feels intelligent or like a fuzzy keyword search. The conversation about Apple Silicon local AI usually centers on generation (Llama 3.3 70B on M5 Max, MLX vs llama.cpp, prefill versus decode), but the embedding model running in front of the LLM does as much work, and on a busy ingest pipeline it does more. Get the embedding wrong and the LLM is reasoning over the wrong context; the generation runtime can be perfect and the answer will still be wrong.

Three embedding models matter on Apple Silicon as of June 2026: nomic-embed-text-v2 (137M parameters, MIT licensed, 8192 token context), bge-m3 (568M parameters, MIT licensed, 8192 token context, multilingual + dense + sparse + colbert in one model), and Qwen3-Embedding-8B (8B parameters, Apache 2.0, 32K context, top of the MTEB leaderboard). They cover three different parts of the production envelope, and the right pick depends on corpus size, query latency budget, and whether the workload is monolingual English or multilingual.

We measured all three on a 128GB M5 Max running MLX 0.21 and llama.cpp b4900, across a 200K passage BEIR slice, a 50K row product catalog, and a synthetic multilingual workload covering English, German, Japanese, and Spanish.

Headline Comparison

Dimension	nomic-embed-text-v2	bge-m3	Qwen3-Embedding-8B
Parameters	137M	568M	8B
License	MIT	MIT	Apache 2.0
Context length	8,192 tokens	8,192 tokens	32,768 tokens
Output dimension	768 (matryoshka to 256, 128, 64)	1024 (dense), sparse + colbert vectors	4096 (matryoshka to 2048, 1024, 512)
Languages	English, multilingual variant available	100+ languages native	100+ languages native
MTEB English avg	62.3	64.5	70.8
BEIR nDCG@10 (our test)	47.2	51.8	56.4
Throughput, M5 Max MLX, batch 64, 512 tokens	14,200 passages/sec	4,800 passages/sec	580 passages/sec
Memory at fp16	0.4 GB	1.4 GB	18 GB
Memory at Q8	0.2 GB	0.7 GB	9 GB
Cold start time	1.2 s	2.8 s	14.4 s

The picture is a clean ladder. nomic-embed-text-v2 is the fast tier: minimal memory, sub second cold start, ingests catalog scale corpora in minutes. bge-m3 is the quality and flexibility tier: a meaningful retrieval quality lift, native multilingual, and the only model in the comparison that emits a sparse vector and a colbert late interaction vector alongside the dense embedding. Qwen3-Embedding-8B is the top end: best retrieval quality measured, 32K context for long document chunking, but 25x slower than nomic and roughly 45x the memory.

Throughput on M5 Max: The Number That Sets Ingest Budget

Model	Runtime	Batch size	Tokens per passage	Passages/sec	GB/hour ingested
nomic-embed-text-v2	MLX 0.21	64	512	14,200	26.2
nomic-embed-text-v2	MLX 0.21	128	512	19,800	36.5
nomic-embed-text-v2	llama.cpp	64	512	9,400	17.3
bge-m3	MLX 0.21	64	512	4,800	8.8
bge-m3	MLX 0.21	128	512	6,100	11.3
bge-m3	llama.cpp	64	512	3,200	5.9
Qwen3-Embedding-8B	MLX 0.21	32	512	580	1.07
Qwen3-Embedding-8B	MLX 0.21	64	512	720	1.33
Qwen3-Embedding-8B	llama.cpp	32	512	410	0.76

Three observations sit underneath the numbers.

MLX outperforms llama.cpp on embedding workloads by roughly 50 percent on the same model and batch size. The gap is wider than on text generation because embedding inference is prefill bound, and the MLX 0.21 prefill pipeline on Apple Silicon is materially better tuned than llama.cpp for batch sizes above 16. For ingest workloads the runtime choice is not interchangeable.

Batch size matters far more than it does for generation. Moving from batch 32 to batch 128 lifts nomic throughput by 40 percent and bge-m3 throughput by 27 percent. The memory cost is trivial because the activation footprint per passage is small. Set batch size against the largest figure your unified memory budget tolerates, and the answer is almost always 128 for nomic, 64 to 128 for bge-m3, and 32 to 64 for Qwen3-Embedding-8B.

Qwen3-Embedding-8B's throughput is the ceiling on its production usability. 580 passages per second sounds slow because it is slow. A 10 million passage catalog ingests in roughly 4.8 hours on Qwen3 versus 12 minutes on nomic. For workloads where retrieval quality is critical and the ingest is offline, Qwen3 is correct. For workloads where the corpus updates daily and the ingest budget is an hour, Qwen3 is the wrong tool.

Retrieval Quality: BEIR nDCG@10 By Subset

Aggregate MTEB scores are a marketing number. The per task split is what determines whether a model holds up on a specific corpus.

BEIR task	nomic-embed-text-v2	bge-m3	Qwen3-Embedding-8B
TREC-COVID	71.2	76.4	81.3
NFCorpus	32.8	36.9	41.2
HotpotQA	60.4	65.7	71.8
FiQA-2018	41.7	47.2	54.6
ArguAna	51.4	58.6	62.1
Touche-2020	26.8	29.4	33.7
Quora	86.9	88.2	89.4
SCIDOCS	19.4	22.1	25.8
DBPedia	38.1	43.6	48.9
Climate-FEVER	30.8	35.4	41.2
Mean nDCG@10	47.2	51.8	56.4

Qwen3-Embedding-8B leads on every subset, and the gap is largest on technical and reasoning heavy tasks (HotpotQA, FiQA, DBPedia). bge-m3 closes the gap on conversational and short query tasks (Quora, ArguAna) where its sparse plus dense fusion adds real signal. nomic-embed-text-v2 holds within 5 to 9 points on most subsets and is the lowest cost retrieval model that still feels competent in production.

The right framing: on a 56 to 47 spread, dropping from Qwen3 to nomic costs roughly 9 nDCG points of retrieval quality and saves 25x on inference cost. Whether that trade is correct depends on the workload. For an internal knowledge base where retrieval errors fall back to a human in the loop, nomic is correct. For a regulated workload where a missed retrieval is a compliance miss, Qwen3 is correct.

Multilingual: Where bge-m3 Earns Its Slot

Language	nomic-embed-text-v2 (multilingual)	bge-m3	Qwen3-Embedding-8B
English	47.2	51.8	56.4
German	42.1	49.6	54.8
Japanese	38.4	47.9	53.2
Spanish	44.7	50.8	55.6
Cross lingual (en query, multilingual corpus)	39.6	48.2	53.4

The English specialist nomic-embed-text-v2 drops 8 to 13 points on non English subsets. bge-m3 holds within 2 to 4 points of its English score across all four languages, and on the cross lingual case (English query, mixed language corpus) it sits closer to Qwen3 than to nomic. For multilingual e commerce workloads (a Shopify Plus brand selling across the EU and APAC), bge-m3 is the obvious mid tier pick.

Memory Footprint and Batch Concurrency

On a 128GB M5 Max the memory math is permissive for the small models and tight for Qwen3.

Model + quant	Single instance	4x batched serving	8x batched serving
nomic-embed-text-v2 fp16	0.4 GB	1.6 GB	3.2 GB
nomic-embed-text-v2 Q8	0.2 GB	0.8 GB	1.6 GB
bge-m3 fp16	1.4 GB	5.6 GB	11.2 GB
bge-m3 Q8	0.7 GB	2.8 GB	5.6 GB
Qwen3-Embedding-8B fp16	18 GB	72 GB	does not fit
Qwen3-Embedding-8B Q8	9 GB	36 GB	72 GB

The footprint matters when the embedding service shares unified memory with a Llama 70B generation runtime. A 70B Q4 model holds roughly 42 GB of weights and 5 to 20 GB of KV cache on a 128GB box, leaving 60 to 80 GB for everything else. nomic and bge-m3 fit comfortably alongside even with 8x batched embedding serving. Qwen3-Embedding-8B at Q8 fits, but co tenancy with a Llama 70B at long context is tight, and any prefix caching budget on the generation side eats into the embedding budget.

The MLX Configuration That Works

For nomic-embed-text-v2 with batched ingest on M5 Max:

mlx_lm.embed \
  --model nomic-ai/nomic-embed-text-v2-moe \
  --port 8090 \
  --batch-size 128 \
  --max-tokens 512 \
  --quant q8 \
  --pooling mean \
  --normalize true

For bge-m3 with dense plus sparse output:

mlx_embed.server \
  --model BAAI/bge-m3 \
  --port 8090 \
  --batch-size 64 \
  --max-tokens 8192 \
  --quant q8 \
  --output dense,sparse,colbert \
  --normalize true

For Qwen3-Embedding-8B with long context chunking:

mlx_embed.server \
  --model Qwen/Qwen3-Embedding-8B \
  --port 8090 \
  --batch-size 32 \
  --max-tokens 32768 \
  --quant q8 \
  --matryoshka-dim 2048 \
  --normalize true

The matryoshka dimension flag is worth a beat for Qwen3. Storing the full 4096 dimension vector costs 16 KB per passage at fp16; truncating to 2048 dimensions costs 8 KB and loses roughly 0.4 nDCG points on most tasks. For corpora above 1 million passages the storage and query latency wins are material; for smaller corpora keep the full dimension.

When This Applies to Your Stack

Three concrete scenarios where the embedding model choice flips the architecture.

A 200K page internal knowledge base for a US English engineering team, queried by 50 employees, generation by Llama 3.3 70B on the same Mac. nomic-embed-text-v2 at Q8 is the right pick: ingest is fast, retrieval quality is competent, the memory budget leaves room for prefix caching on the generation side, and the cost saving relative to bge-m3 is roughly 3x throughput on a co tenant deployment.

A multilingual product catalog for a Shopify Plus brand selling across 12 markets in 7 languages, 5 million SKU descriptions plus 200K editorial pages. bge-m3 at Q8 is correct: the multilingual quality gap over nomic is decisive, the throughput sustains a nightly full reindex on a single M5 Max in under 8 hours, and the sparse vector feeds a hybrid retrieval setup alongside the dense vector.

A regulated technical corpus (medical, legal, scientific) where retrieval misses translate to incorrect generations and real risk. Qwen3-Embedding-8B at Q8 is the right pick: the retrieval quality lift is the most expensive 9 nDCG points you will buy in the stack, the ingest is offline and the throughput cost is acceptable, and the 32K context length permits much larger chunks that preserve semantic context.

When This Does Not Apply

Real time embedding of user queries at scale, hundreds per second, on a single Mac. Qwen3-Embedding-8B will not sustain the throughput. For latency sensitive query side embedding, run nomic or a distilled bge variant on the M5 Max and reserve Qwen3 for offline corpus ingest.

Workloads where retrieval quality is the binding constraint and ingest budget is unlimited. Frontier hosted embeddings (Voyage 3 Large, OpenAI text-embedding-3-large, Cohere Embed v4) are 2 to 4 points above Qwen3-Embedding-8B on MTEB and have predictable serving latency. Local is the right choice when data sovereignty, cost predictability, or air gap requirements drive the decision; otherwise the frontier hosted embeddings remain the quality ceiling.

Working with Contra Collective

We build production RAG infrastructure on Apple Silicon for teams that need data sovereignty, predictable cost, or air gapped operation. If you are sizing a local embedding stack for a knowledge base, catalog, or regulated corpus on M5 Max or M5 Ultra and want a model selection sized against your retrieval quality target and ingest budget, we can run the benchmark and produce a sized recommendation. Reach out via the Contra Collective contact page.

FAQ

Q: Why MLX over llama.cpp for embeddings on Apple Silicon? A: Embedding inference is prefill bound. The MLX 0.21 prefill pipeline batches more efficiently than llama.cpp at batch sizes above 16, which is the regime that matters for ingest. The throughput delta is roughly 50 percent on the same model and batch.

Q: Can I run Qwen3-Embedding-8B alongside a Llama 3.3 70B generation runtime on a 128GB M5 Max? A: Yes at Q8 for both, but it is tight. The 70B Q4 weights plus 16K KV cache plus 64K prefix cache plus Qwen3 Q8 weights add to roughly 80 GB, leaving 48 GB for OS, activations, and headroom. For workloads that need both, an M5 Ultra Mac Studio is the comfortable target.

Q: How does matryoshka truncation affect retrieval quality? A: Matryoshka models are trained so the first N dimensions of the output vector are themselves a usable embedding. Truncating from 4096 to 2048 dimensions on Qwen3 costs roughly 0.4 nDCG points. Truncating from 2048 to 1024 costs another 0.8 points. Below 1024 the quality drops accelerate. Use 2048 dimensions for large corpora; 4096 for smaller corpora where the storage savings are not material.

Q: Does bge-m3's sparse vector matter in a production retrieval setup? A: Yes for hybrid retrieval. A dense plus sparse fusion (reciprocal rank fusion or weighted sum) typically lifts nDCG@10 by 3 to 6 points over dense alone on corpora with technical vocabulary, product codes, or proper nouns. bge-m3 is the only model in the comparison that emits both vectors from a single forward pass.

Q: Is there an embedding equivalent to KV cache quantization? A: Embedding inference is single pass; there is no KV cache to quantize. Quantization of the model weights themselves (fp16 to Q8 to Q4) is the only knob. Q8 weights cost roughly 0.2 to 0.5 nDCG points across all three models in the comparison; Q4 weights cost 1 to 2 points and are not recommended for production retrieval.