Posts tagged with #local-inference from the Contra Collective team.
Why llama.cpp and MLX run local LLMs on the M5 Max GPU and ignore the Apple Neural Engine. The ANE is a fixed shape accelerator built for CoreML, and autoregressive decode is the one workload it handles badly. What the ANE is good for, and when it might matter.
How to fine tune 8B to 14B models locally on an M5 Max with mlx-lm LoRA and QLoRA: rank choices, peak memory, training throughput, adapter serving, and when a local run beats a rented cloud GPU in 2026.
Flux.1, SDXL, and Stable Diffusion 3.5 compared for local image generation on an M5 Max: seconds per image, step counts, quantization, peak memory, and prompt adherence. Which diffusion model fits a real backend on Apple Silicon in 2026.
Kokoro, Piper, and XTTS v2 compared for local text to speech on an M5 Max: real time factor, latency to first audio, voice quality, cloning support, and memory. Which local TTS engine fits a real backend in 2026, measured on Apple Silicon.
M5 Max, RTX 5090, and DGX Spark compared for local LLM inference: memory ceiling, decode throughput, prefill on long prompts, power draw, and the amortized cost per million tokens that decides which box to buy for a self hosted model in 2026.
Where the latency actually goes in a fully local RAG pipeline on M5 Max: embed, retrieve, rerank, and generate as one budget. Stage by stage timings, the cost of loading three models on one box, and the orchestration choices that decide whether local RAG feels instant or slow in 2026.
Qwen3-Coder 30B on M5 Max running a real agentic loop. Prefill latency, decode throughput, tool-call cycle time, and the MLX vs llama.cpp tradeoffs that decide whether a local coding agent earns its slot.
Local embedding models on M5 Max. nomic-embed-text-v2, bge-m3, and Qwen3-Embedding-8B compared on throughput, retrieval quality, memory footprint, and the production tradeoffs that matter for RAG on Apple Silicon.
MLX KV cache quantization on M5 Max. Q8 vs Q4 vs FP16 throughput, context length, and quality trade offs on Llama 3.3 70B and Qwen 3.6 27B under sustained load.
M5 Ultra batched inference under MLX, llama.cpp, and vLLM Metal. Throughput per user at batch 1, 4, 8, 16, and 32 on a Mac Studio for production serving.
Tokens per second is half the story. The other half is what those tokens cost in electricity, hardware amortization, and rack space. Most Apple Silicon local LLM benchmarks publish a decode number on Llama 3.1 8B and stop. That makes for a clean headline and a useless TCO model. We measured wall power on M4 Pro, M5 Pro, M5 Max, and M5 Ultra under sustained inference for Llama 3.3 8B and Llama 3.3 70B, then converted everything to Joules per generated token and dollars per million tokens. The numbers settle the "should we self host on Apple Silicon" conversation in a way that throughput alone cannot.