GPU vs Apple Neural Engine for Local LLM Inference on M5 Max: Why the Runtimes Skip the ANE (2026)

Open Activity Monitor while an 8B model streams tokens on your M5 Max and you will see the GPU pinned and the Neural Engine flat at zero. This surprises people. Apple spends keynote minutes on the Neural Engine, quotes its trillions of operations per second, and positions it as the machine learning engine of the chip. So why does every serious local LLM runtime, llama.cpp, MLX, Ollama, LM Studio underneath them, put the whole model on the GPU and never touch the ANE? The short answer is that the Neural Engine is very good at exactly the workload a language model decode step is not. This post explains the hardware, the reason the runtimes made this call, and the few places where the Neural Engine still has a role.

What the Neural Engine Actually Is

The Apple Neural Engine is a fixed function accelerator. It is a grid of multiply accumulate units wired for one job: pushing tensors through a precompiled graph of operations at very high efficiency per watt. You do not program it directly. You hand CoreML a model, CoreML compiles the graph, and the runtime decides which operations land on the ANE, which fall back to the GPU, and which run on the CPU. The programming surface is the CoreML graph, not a general compute API.

That design is a deliberate trade. The ANE gives up flexibility to win on power. For the workloads it was built for, image classification, face detection, the vision and audio models that run constantly on an iPhone, it delivers inference at a fraction of the energy the GPU would draw. The constraints that make it efficient are the same constraints that make it a poor fit for a transformer decode loop: it wants static tensor shapes, a fixed graph known at compile time, and a limited menu of supported operations at specific precisions.

Why Autoregressive Decode Fights the Hardware

A language model generating text does one awkward thing over and over. It produces a single token, appends it to the sequence, grows the key value cache by one position, and runs the whole forward pass again to produce the next token. Two properties of that loop collide with the ANE design.

The first is dynamic shape. Every decode step the sequence length changes, so the tensors flowing through attention change shape on every iteration. The Neural Engine is happiest with shapes fixed at compile time. You can pad to a maximum length and recompile for buckets of sizes, but you pay for the padding on every step and you fight the tooling the entire way. The GPU, driven by Metal, simply dispatches a kernel with the current dimensions and moves on.

The second is that decode is memory bandwidth bound, not compute bound. Generating one token at batch size one is a sequence of large matrix by vector products where the model weights are read from memory once and used barely at all before the next weight block is needed. The bottleneck is how fast you can stream weights out of unified memory, not how many multiply accumulate units you can light up. The ANE's advantage is compute density per watt. When the workload is bandwidth bound, that advantage does not convert into tokens per second, because both the GPU and the ANE are drinking from the same unified memory pool at the same bandwidth ceiling. We put hard numbers on that bandwidth wall in the M5 Max versus RTX 5090 versus DGX Spark inference comparison.

The Tooling Reality: Metal Won Because It Fits

Even if the hardware match were closer, the software would still point at the GPU. MLX was written by Apple's own machine learning research group as an array framework that compiles to Metal. It exposes the flexible, general compute the transformer needs: arbitrary kernels, dynamic shapes, the quantization formats the local model ecosystem actually ships in. llama.cpp reaches the same GPU through its Metal backend for the same reason. Neither has a path to the Neural Engine because CoreML, the only door to the ANE, does not give a runtime the low level control an LLM kernel author wants. We compared those two GPU backends head to head in the llama.cpp Metal versus MLX backend breakdown.

Property	Apple Neural Engine	GPU (Metal)
Programming surface	CoreML graph only	Metal kernels, MLX, direct control
Tensor shapes	Prefers static, compile time	Fully dynamic
Quantization support	CoreML palettization, narrow	GGUF, MLX 4-bit and 8-bit, flexible
Best workload	Fixed graph vision and audio	Large flexible matmul, LLM decode
Power efficiency	Excellent on its workloads	Higher draw, higher throughput
KV cache growth	Awkward, needs recompile or padding	Native, resized per step
LLM runtime support	Effectively none	llama.cpp, MLX, Ollama, LM Studio

So Is the Neural Engine Useless for LLMs?

No, and this is where the nuance matters. The Neural Engine is bad at the decode loop specifically. It is not bad at every part of an inference pipeline. Three cases are worth knowing.

Prefill is more compute bound than decode. When you feed a long prompt, the model processes many tokens in parallel before it emits the first output token, and that phase looks more like the dense batched matmul the ANE handles well. A hybrid runtime could in principle push prefill toward the Neural Engine and keep decode on the GPU. Nobody ships this in the mainstream local stack today because the engineering cost of straddling two accelerators, with the CoreML conversion tax on one side, outweighs the win, but it is not a hardware impossibility.

Small fixed models are the ANE's home turf. Embedding models, rerankers, classifiers, speech models, anything with a fixed shape and a modest parameter count runs beautifully through CoreML on the Neural Engine at very low power. If your product runs a small always on model alongside an occasional large one, the right architecture may put the small model on the ANE and the large model on the GPU. Vision language models are a partial case here too, since the vision encoder can be a fixed graph even when the language decoder is not, a split we touched on in the local vision LLM guide.

Battery constrained inference is the scenario where the efficiency gap could flip the decision. On a plugged in Mac Studio nobody cares that the GPU draws more watts. On a laptop running an on device assistant all day, the power difference between a fixed model on the ANE and the same model on the GPU is real. This is exactly why Apple's own on device features route through CoreML and the Neural Engine rather than MLX on the GPU.

When This Applies to Your Stack

If you are standing up local inference on Apple Silicon and wondering whether you are leaving performance on the table by not using the Neural Engine, the answer for a general purpose LLM is no. Put the model on the GPU with MLX or llama.cpp, size it to your unified memory, and stop thinking about the ANE. The runtimes made the correct call and there is no faster path to tokens per second hiding in the CoreML toolchain for a large autoregressive model in 2026.

Where the calculus changes is a mixed workload on battery, or a pipeline built from several small fixed models plus one large one. There, splitting the graph across accelerators, small and fixed on the Neural Engine, large and flexible on the GPU, can cut power draw meaningfully without hurting the experience. That is a real architecture decision with real tooling cost, and it is worth modeling before you commit.

If your team is designing a local inference layer and needs the accelerator split, the memory budget, and the power envelope worked out against your actual models rather than a spec sheet, Contra Collective builds and benchmarks Apple Silicon inference systems that ship into production. The Neural Engine is a precise tool, and knowing when not to reach for it is half of using the chip well.

FAQ

Can I force llama.cpp or MLX to use the Neural Engine? No. Neither framework has an ANE backend. The only route to the Neural Engine is CoreML, and these runtimes target Metal on the GPU by design. Forcing the issue means converting your model to CoreML and accepting the dynamic shape and quantization limits, which for a large LLM costs more than it returns.

Does the Neural Engine make local LLMs more power efficient? Not for the decode loop. Token generation is memory bandwidth bound, and the ANE and GPU share the same unified memory bandwidth, so the ANE's efficiency advantage does not convert to faster or cheaper tokens. The efficiency win is real only for fixed shape models like embedding, vision, and speech models.

Why does Apple use the ANE for its own AI features then? Because those features are mostly small fixed models running constantly on battery powered devices, which is precisely the ANE's strength. A compiled CoreML model for a specific on device task is a different workload from a general purpose 8B or 14B chat model with a growing context.

Would a hybrid prefill on ANE, decode on GPU design help? In theory, because prefill is more compute bound and closer to the ANE's comfort zone. In practice no mainstream runtime ships it, because coordinating two accelerators and paying the CoreML conversion cost outweighs the prefill speedup for typical prompt lengths. It is an engineering trade, not a hardware wall.

Should I buy more Neural Engine cores for local LLM work? No. Optimize for GPU cores and unified memory bandwidth, which are what actually bound local LLM throughput. Neural Engine capacity matters for on device vision and audio pipelines, not for running larger or faster language models.