Posts tagged with #mlx from the Contra Collective team.
Two very different architectures win at local speech to text on Apple Silicon. Whisper large-v3 is an autoregressive encoder decoder that is multilingual and robust. Parakeet is a FastConformer transducer that is far faster and streams. Here is how each behaves on a 64GB M5 Max under MLX, and which one belongs behind your feature.
A Mixture of Experts model activates a fraction of its parameters per token, but every expert still has to sit in memory. Here is why total parameters set the residency floor on an M5 Max, why active parameters set decode speed, and how to size the machine so a MoE model fits instead of swapping.
What actually happens when a local LLM asks for more unified memory than the M5 Max can wire down: the wired limit, the swap cliff, and how tokens per second collapses once the model, KV cache, and your app compete for the same pool.
How to run more than one model on a single M5 Max, with a unified memory budget for a router plus a generation model, real model swap latency numbers, and the point where a 128GB Mac forces you to choose between resident models and headroom for context.
How much quality you actually lose quantizing a local LLM to 4-bit and 8-bit on an M5 Max, measured with perplexity, KL divergence, and task benchmarks rather than vibes, plus where the cliff sits and which layers to keep at higher precision.
How to fine tune 8B to 14B models locally on an M5 Max with mlx-lm LoRA and QLoRA: rank choices, peak memory, training throughput, adapter serving, and when a local run beats a rented cloud GPU in 2026.
Where the latency actually goes in a fully local RAG pipeline on M5 Max: embed, retrieve, rerank, and generate as one budget. Stage by stage timings, the cost of loading three models on one box, and the orchestration choices that decide whether local RAG feels instant or slow in 2026.
Model load time and first token latency for local LLM inference on M5 Max, measured for MLX and llama.cpp. Weight load, mmap behavior, warm versus cold first token, and how model swap latency decides multi model serving on Apple Silicon in 2026.
M5 Max, M5 Ultra, and M4 Max measured for tokens per watt and energy cost per million tokens under MLX. Wall power draw, sustained throughput, quantization, and the electricity line item that decides which Mac wins the local inference unit economics in 2026.
Whisper Large v3 Turbo, NVIDIA Parakeet TDT, and Distil Whisper compared on a 128GB M5 Max under mlx-whisper. Word error rate, real time factor, latency, peak memory, and cost per audio hour on a real long form transcription workload in 2026.
KV cache offload from unified memory to NVMe SSD on a 128GB M5 Max lets a Llama 3.3 70B server hold 512K context across many concurrent sessions. Throughput, prefill latency, decode penalty, and the queue depth that keeps the SSD path viable for production inference in 2026.
Tool calling reliability on Apple Silicon tested across Qwen3 32B and Llama 3.3 70B under mlx-lm. JSON schema adherence rates, recovery behavior on malformed calls, and the local model that earns a seat in a real agent loop.
Qwen3 235B MoE and Llama 3.3 70B Dense compared on a 512GB M5 Ultra Mac Studio under mlx-lm. Tokens per second, peak memory, sustained throughput, and cost per million tokens for a local production inference workload in 2026.
MLX 0.21 FP8 quantization compared against Q6 and Q4 on a 128GB M5 Max for production inference. Throughput, peak memory, quality regression on eval suites, and the quantization choice that actually belongs in a production serving path.
BGE reranker v2, Cohere Rerank 3 (hosted), and Qwen3 reranker compared on M5 Max for local RAG. Latency per 100 candidates, NDCG, recall, and cost per million queries that decide whether reranking belongs on device or behind an API.
Fine-tuning Llama 3.3 70B with LoRA on a 256GB M5 Ultra Mac Studio. Throughput per step, unified memory budget, convergence behavior, and the MLX vs llama.cpp choice that decides whether local fine-tuning fits your workflow.
Qwen3-Coder 30B on M5 Max running a real agentic loop. Prefill latency, decode throughput, tool-call cycle time, and the MLX vs llama.cpp tradeoffs that decide whether a local coding agent earns its slot.
Draft model selection decides whether speculative decoding doubles throughput or wastes cycles on Apple Silicon. Llama 3.2 1B, Qwen3 0.5B, and a custom distilled draft tested on M5 Max for acceptance rate, token speedup, and engineering cost.
Constrained decoding on Apple Silicon compared. Outlines, llama.cpp GBNF grammars, and MLX logit bias measured on throughput, schema fidelity, and tool call accuracy for local agents on M5 Max.
Local embedding models on M5 Max. nomic-embed-text-v2, bge-m3, and Qwen3-Embedding-8B compared on throughput, retrieval quality, memory footprint, and the production tradeoffs that matter for RAG on Apple Silicon.
MLX prompt caching on M5 Max. Prefix cache hit rates, prefill savings, memory footprint, and the eviction policy that matters for agent, RAG, and code review workloads on Apple Silicon.
MLX KV cache quantization on M5 Max. Q8 vs Q4 vs FP16 throughput, context length, and quality trade offs on Llama 3.3 70B and Qwen 3.6 27B under sustained load.
M5 Ultra batched inference under MLX, llama.cpp, and vLLM Metal. Throughput per user at batch 1, 4, 8, 16, and 32 on a Mac Studio for production serving.
Decode throughput on M5 Pro and M5 Max measured at 32K, 128K, 512K, and 1M context with llama.cpp and MLX. Where Apple Silicon long context inference breaks.
Tokens per second is half the story. The other half is what those tokens cost in electricity, hardware amortization, and rack space. Most Apple Silicon local LLM benchmarks publish a decode number on Llama 3.1 8B and stop. That makes for a clean headline and a useless TCO model. We measured wall power on M4 Pro, M5 Pro, M5 Max, and M5 Ultra under sustained inference for Llama 3.3 8B and Llama 3.3 70B, then converted everything to Joules per generated token and dollars per million tokens. The numbers settle the "should we self host on Apple Silicon" conversation in a way that throughput alone cannot.
Every Apple Silicon local inference benchmark on the internet runs for thirty seconds, quotes a peak token rate, and calls it a day. That number is useful for marketing copy and useless for capacity planning. If you are deploying MLX or llama.cpp behind a real backend, the question is not what the SoC does in the first thirty seconds. The question is what it does over four hours of continuous decode under realistic batch conditions, after the package power has saturated and the thermal envelope has clamped down.
Text-only local LLM inference on Apple Silicon stopped being interesting about a year ago. The runtimes converged, quantization formats stabilized, and the throughput numbers across llama.cpp, MLX, and Ollama settled into a predictable ranking. Vision language models are a different story. The runtime support is uneven, the quantization story is half-finished, and the benchmarks people quote in blog posts are almost always wrong because they measure pure decode throughput and ignore the image preprocessing path that dominates real latency.
MLX 0.22 landed in late May 2026 with a refactored flash attention implementation that closes most of the gap to what llama.cpp pulls out of Metal Flash Attention 2. For engineering teams running local LLMs on M5 Ultra hardware, this is the version that changes the runtime calculus.
Most teams running local LLMs on Apple Silicon hit the same wall: unified memory caps out at 512GB on a single M5 Ultra, and anything beyond Llama 4 Scout or a quantized Maverick stops fitting. The cloud option is real, but for teams who deliberately chose local inference for privacy, cost, or latency reasons, the answer for 2026 is distributed MLX.