Local Agent Tool-Call Loop Latency on an M5 Max: Where the Seconds Go Across a Multi-Step Run

A one shot prompt and an agent loop are not the same performance problem, and treating them the same is why so many local agents feel slow for reasons their builders cannot explain. A single generation is a prefill of the prompt followed by a decode of the answer, and the tokens per second figure you benchmarked predicts it well. An agent is a loop that runs the model several times over a conversation that grows at every step, and the number that predicts its latency is not decode speed at all. It is how fast the machine can prefill an ever longer context, over and over, on a GPU that has nothing else to give. This post takes a concrete agent run on an M5 Max apart, measures each part, and shows why the fix is caching, not a smaller model.

The Anatomy of One Agent Step

Every iteration of a tool calling agent has the same four parts, and each one costs something different.

First, the model reads the current conversation. That is a prefill over the system prompt, the user's task, and everything the loop has accumulated so far: prior reasoning, prior tool calls, and prior tool results. On the first step that is short. By the fifth step it can be thousands of tokens, and prefill time scales with that length.

Second, the model decodes a tool call. This is usually short, a function name and a small JSON argument object, so decode here is cheap, often under a hundred tokens.

Third, your code runs the tool. This cost lives entirely outside the model: a database read, an HTTP call, a file operation. It can be a millisecond or it can be a network round trip that dwarfs everything else.

Fourth, the tool result is appended and the loop repeats. The result becomes new context that the next step has to prefill, which is exactly why the conversation grows and the per step prefill climbs.

The trap is that people benchmark part two, decode, because that is what a one shot benchmark measures, and then they are surprised when the agent is slow. On a local M5 Max serving its own traffic, the dominant cost across a multi step run is part one, the repeated prefill of the growing context.

Where the Seconds Actually Go

To make this concrete, here is a five step agent run on an M5 Max, an 8B class model quantized to Q4, driven through a local server. The task is a realistic tool using flow: the model calls a search tool, reads results, calls a fetch tool, reads a document, then answers. The numbers are from a controlled test harness, meant to show the shape of the cost, not a guaranteed figure for your setup.

Step	Cumulative context (tokens)	Prefill, no cache	Prefill, prefix cache	Decode (tokens)	Tool exec
1: emit search call	420	0.9s	0.9s	38	0.20s
2: read results, emit fetch	1,180	2.4s	0.5s	44	0.60s
3: read doc, reason	3,050	6.1s	1.1s	120	0.00s
4: emit follow up call	3,600	7.2s	0.4s	40	0.35s
5: final answer	4,250	8.4s	0.5s	260	0.00s

Read the two prefill columns against each other, because that comparison is the whole point. Without caching, prefill climbs every step as the conversation grows, and by step five the model spends more than eight seconds simply reading the context before it decodes a single token of the answer. Sum the no cache prefill column and it is roughly twenty five seconds of pure prefill across the run, against maybe three seconds of actual decode. The model is not slow. The loop is re-reading everything it already read, five times over.

The prefix cache column is the same run with one change: the unchanged prefix of the conversation is kept in the KV cache between steps, so each step only prefills the tokens that are genuinely new, the latest tool result. Prefill per step drops to fractions of a second, because appending a tool result and continuing is cheap, while reprocessing four thousand tokens from scratch is not. The total prefill for the run falls from about twenty five seconds to roughly three, and the wall clock time of the agent collapses accordingly.

Why Prefill Dominates and Decode Does Not

The intuition that trips people up is that decode feels like the expensive part because it is the visible part, the tokens streaming out. But an agent's decode is mostly tiny tool calls, forty tokens here, forty there, with one longer final answer. Prefill, by contrast, processes the entire growing context on every step, and prefill is compute bound in a way that scales with sequence length. The prefill of a four thousand token context is not four thousand small independent operations; it is an attention computation over the whole sequence, and on a single M5 Max that work lands on the same GPU that would otherwise be serving anyone else.

This is the same prefill scaling that governs long context generation generally, covered in more depth in long context prefill latency scaling. The difference in an agent is that you pay it repeatedly, once per step, and the context you pay for is one you already processed on the previous step. That redundancy is what caching removes.

Making the Loop Cache Correctly

Prefix caching only helps if each step's prompt is a strict extension of the previous step's prompt. The moment you edit earlier turns, reorder history, or inject a summary in the middle, the cached prefix is invalidated from the edit point onward and you pay full prefill again from there. The discipline is to append and only append.

# Keep the KV cache across steps; append tool results, never rewrite history.
cache = make_prompt_cache(model)          # persists between iterations
messages = [system, user_task]

for step in range(max_steps):
    prompt = render(messages)             # strict superset of last iteration
    # Only the newly appended tokens are prefilled; the prefix is reused.
    reply = generate(model, tokenizer, prompt, prompt_cache=cache)

    call = parse_tool_call(reply)
    if call is None:
        break                             # model produced the final answer
    result = run_tool(call)               # cost lives outside the model
    messages.append(assistant(reply))     # append, do not edit prior turns
    messages.append(tool(result))         # next step prefills only this

Two failure modes hide here. The first is silently invalidating the cache by templating the conversation differently between steps, for example re-rendering the system prompt with a changing timestamp, which shifts every token after it and defeats the cache for the entire prompt. Pin anything volatile out of the cached prefix. The second is letting the context grow without bound: prefix caching makes each step cheap relative to reprocessing, but the KV cache itself still consumes unified memory that scales with context length, so a runaway loop that never terminates will eventually pressure memory even with caching on. A hard cap on steps and on total context is the backstop.

The Tool Latency You Control

The one part of the loop that has nothing to do with the model is tool execution, and it is worth measuring separately because it is often where a slow agent's time actually goes once prefill is cached. A tool that hits a local database returns in milliseconds. A tool that fetches a web page or calls an external API can take longer than the entire model portion of the step, and no amount of inference tuning touches it. Once you have caching in place, profile the tool calls: if a single external fetch is two seconds, that is now your bottleneck, and the fix is on the tool side, parallelizing independent calls or caching their results, not on the model side. Separating model time from tool time in your traces is what tells you which half of the loop to optimize.

When This Applies to Your Stack

This matters the moment your local deployment stops answering single questions and starts running loops, which is what any real agent does. If you are profiling a slow agent on an M5 Max, measure prefill per step and decode per step separately before you touch the model, because the reflex to swap in a smaller model addresses decode, and decode was never the problem. Turn on prefix caching, keep the conversation append only so the cache stays valid, cap steps and total context so memory stays bounded, and then measure tool execution on its own to see what is left. The same discipline that keeps a single local model responsive under concurrency, discussed in admission control and backpressure, is what keeps an agent loop from turning a four thousand token conversation into a thirty second wait, and it composes with the retrieval side covered in the local RAG latency budget.

Building agents that run on your own hardware rather than a metered API, where every redundant prefill is your electricity and your latency, is the local inference work Contra Collective does. The model choice is the easy decision; the loop architecture is what decides whether the agent is usable.

FAQ

Why is step five of my agent so much slower than step one? Because step five prefills a much longer conversation. Each step appends the previous tool result to the context, so the prompt the model reads grows every iteration, and prefill time scales with that length. Without caching, step five reprocesses everything from steps one through four before it decodes anything, which is why the later steps dominate the wall clock time.

Is a smaller model the fix for a slow local agent? Usually not. A smaller model decodes faster, but an agent's decode is mostly short tool calls, so decode was never the bottleneck. The dominant cost is repeated prefill of the growing context, and a smaller model prefills a long context only marginally faster. Prefix caching addresses the actual problem; model swapping addresses the wrong one.

What does prefix caching actually save in an agent loop? It reuses the KV cache for the unchanged part of the conversation between steps, so each step only prefills the newly appended tokens rather than the whole history. In the run measured above, that cut total prefill for a five step agent from roughly twenty five seconds to about three, because appending a tool result and continuing is cheap while reprocessing the full context every step is not.

What invalidates the prefix cache? Any change to the earlier part of the prompt. If you edit prior turns, reorder history, insert a summary in the middle, or re-render the system prompt with a changing value like a timestamp, the cache is invalid from the change point onward and you pay full prefill from there. Keeping the conversation strictly append only is what preserves the cache.

Does caching remove the need to cap context length? No. Caching makes each step cheap relative to reprocessing, but the KV cache still occupies unified memory proportional to the context length. A loop that never terminates will keep growing the cache and eventually pressure memory, so a hard cap on steps and total tokens is still the backstop that keeps a runaway agent from exhausting the machine.

This post reports results from a controlled local test harness, not a production benchmark, and the figures will vary with your model, quantization, tools, and context sizes; measure your own agent before sizing capacity around these numbers.