Posts tagged with #m5-max from the Contra Collective team.
An importance matrix reweights quantization error toward the weights a calibration corpus actually exercises. On an M5 Max running GGUF through llama.cpp, we measured where imatrix calibration helps, where it does nothing, and why the bit width you picked decides whether it is worth the extra step.
Two very different architectures win at local speech to text on Apple Silicon. Whisper large-v3 is an autoregressive encoder decoder that is multilingual and robust. Parakeet is a FastConformer transducer that is far faster and streams. Here is how each behaves on a 64GB M5 Max under MLX, and which one belongs behind your feature.
A Mixture of Experts model activates a fraction of its parameters per token, but every expert still has to sit in memory. Here is why total parameters set the residency floor on an M5 Max, why active parameters set decode speed, and how to size the machine so a MoE model fits instead of swapping.
What actually happens when a local LLM asks for more unified memory than the M5 Max can wire down: the wired limit, the swap cliff, and how tokens per second collapses once the model, KV cache, and your app compete for the same pool.
How to run more than one model on a single M5 Max, with a unified memory budget for a router plus a generation model, real model swap latency numbers, and the point where a 128GB Mac forces you to choose between resident models and headroom for context.
How much quality you actually lose quantizing a local LLM to 4-bit and 8-bit on an M5 Max, measured with perplexity, KL divergence, and task benchmarks rather than vibes, plus where the cliff sits and which layers to keep at higher precision.
Why llama.cpp and MLX run local LLMs on the M5 Max GPU and ignore the Apple Neural Engine. The ANE is a fixed shape accelerator built for CoreML, and autoregressive decode is the one workload it handles badly. What the ANE is good for, and when it might matter.
How to fine tune 8B to 14B models locally on an M5 Max with mlx-lm LoRA and QLoRA: rank choices, peak memory, training throughput, adapter serving, and when a local run beats a rented cloud GPU in 2026.
Flux.1, SDXL, and Stable Diffusion 3.5 compared for local image generation on an M5 Max: seconds per image, step counts, quantization, peak memory, and prompt adherence. Which diffusion model fits a real backend on Apple Silicon in 2026.
Kokoro, Piper, and XTTS v2 compared for local text to speech on an M5 Max: real time factor, latency to first audio, voice quality, cloning support, and memory. Which local TTS engine fits a real backend in 2026, measured on Apple Silicon.
M5 Max, RTX 5090, and DGX Spark compared for local LLM inference: memory ceiling, decode throughput, prefill on long prompts, power draw, and the amortized cost per million tokens that decides which box to buy for a self hosted model in 2026.
Where the latency actually goes in a fully local RAG pipeline on M5 Max: embed, retrieve, rerank, and generate as one budget. Stage by stage timings, the cost of loading three models on one box, and the orchestration choices that decide whether local RAG feels instant or slow in 2026.
Model load time and first token latency for local LLM inference on M5 Max, measured for MLX and llama.cpp. Weight load, mmap behavior, warm versus cold first token, and how model swap latency decides multi model serving on Apple Silicon in 2026.
M5 Max, M5 Ultra, and M4 Max measured for tokens per watt and energy cost per million tokens under MLX. Wall power draw, sustained throughput, quantization, and the electricity line item that decides which Mac wins the local inference unit economics in 2026.
Whisper Large v3 Turbo, NVIDIA Parakeet TDT, and Distil Whisper compared on a 128GB M5 Max under mlx-whisper. Word error rate, real time factor, latency, peak memory, and cost per audio hour on a real long form transcription workload in 2026.
KV cache offload from unified memory to NVMe SSD on a 128GB M5 Max lets a Llama 3.3 70B server hold 512K context across many concurrent sessions. Throughput, prefill latency, decode penalty, and the queue depth that keeps the SSD path viable for production inference in 2026.
Tool calling reliability on Apple Silicon tested across Qwen3 32B and Llama 3.3 70B under mlx-lm. JSON schema adherence rates, recovery behavior on malformed calls, and the local model that earns a seat in a real agent loop.
MLX 0.21 FP8 quantization compared against Q6 and Q4 on a 128GB M5 Max for production inference. Throughput, peak memory, quality regression on eval suites, and the quantization choice that actually belongs in a production serving path.
BGE reranker v2, Cohere Rerank 3 (hosted), and Qwen3 reranker compared on M5 Max for local RAG. Latency per 100 candidates, NDCG, recall, and cost per million queries that decide whether reranking belongs on device or behind an API.
Qwen3-Coder 30B on M5 Max running a real agentic loop. Prefill latency, decode throughput, tool-call cycle time, and the MLX vs llama.cpp tradeoffs that decide whether a local coding agent earns its slot.
Draft model selection decides whether speculative decoding doubles throughput or wastes cycles on Apple Silicon. Llama 3.2 1B, Qwen3 0.5B, and a custom distilled draft tested on M5 Max for acceptance rate, token speedup, and engineering cost.
Local embedding models on M5 Max. nomic-embed-text-v2, bge-m3, and Qwen3-Embedding-8B compared on throughput, retrieval quality, memory footprint, and the production tradeoffs that matter for RAG on Apple Silicon.
MLX prompt caching on M5 Max. Prefix cache hit rates, prefill savings, memory footprint, and the eviction policy that matters for agent, RAG, and code review workloads on Apple Silicon.
MLX KV cache quantization on M5 Max. Q8 vs Q4 vs FP16 throughput, context length, and quality trade offs on Llama 3.3 70B and Qwen 3.6 27B under sustained load.
Decode throughput on M5 Pro and M5 Max measured at 32K, 128K, 512K, and 1M context with llama.cpp and MLX. Where Apple Silicon long context inference breaks.
Tokens per second is half the story. The other half is what those tokens cost in electricity, hardware amortization, and rack space. Most Apple Silicon local LLM benchmarks publish a decode number on Llama 3.1 8B and stop. That makes for a clean headline and a useless TCO model. We measured wall power on M4 Pro, M5 Pro, M5 Max, and M5 Ultra under sustained inference for Llama 3.3 8B and Llama 3.3 70B, then converted everything to Joules per generated token and dollars per million tokens. The numbers settle the "should we self host on Apple Silicon" conversation in a way that throughput alone cannot.
Most Apple Silicon local inference benchmarks publish a single decode number for an 8B parameter model and stop. That works for a demo and falls apart the moment you try to put MLX or llama.cpp behind a real backend. The interesting question is what prefill latency, decode throughput, and batch size scaling actually look like on a 70B class model under the two runtimes most teams pick between in mid 2026: mlx-lm and llama.cpp with the Metal backend.
Every Apple Silicon local inference benchmark on the internet runs for thirty seconds, quotes a peak token rate, and calls it a day. That number is useful for marketing copy and useless for capacity planning. If you are deploying MLX or llama.cpp behind a real backend, the question is not what the SoC does in the first thirty seconds. The question is what it does over four hours of continuous decode under realistic batch conditions, after the package power has saturated and the thermal envelope has clamped down.