Token Logprobs for Confidence and Abstention: Reading Uncertainty from Local LLMs on an M5 Max

Every application built on a local model runs into the same uncomfortable fact. The model writes a wrong answer with the same calm fluency it writes a correct one, and the prose gives you nothing to distinguish the two. A support assistant states a refund policy that does not exist. A RAG system answers from a passage that never mentioned the topic. In each case the text is well formed and confident, and confidence in the wording is not confidence in the fact. What you need is a numeric signal for when to trust the output, when to route it to a larger model, and when to say nothing at all. That signal is already being generated on every token, most local stacks just throw it away.

What a logprob actually is

When a model decodes a token it first computes a score for every entry in its vocabulary, passes those scores through a softmax, and gets a probability distribution over what the next token could be. Sampling picks one token from that distribution. The log probability of the chosen token, the logprob, is the natural log of how much mass the model put on it at that step. A logprob near zero means the model was almost certain, since the probability was close to one. A large negative logprob means the model spread its mass across many candidates and the token it landed on was only one of several plausible options.

Two derived quantities matter more than the per token value. The sum of logprobs across a generation is the log probability of the whole sequence, and its length normalized version, the mean token logprob, is the usual starting point for a confidence score. Exponentiate the negative mean and you get perplexity, the same quantity used to judge the eviction and quantization tradeoffs elsewhere in local inference. The forward pass computed all of this already, so reading it out adds no measurable latency. The work is entirely in deciding which feature to trust.

Why the raw average lies

The intuitive move is to average the token logprobs and treat a high average as high confidence. It is a weak signal, and understanding why is the whole point. A generation is mostly structural tokens, punctuation, articles, connective words, formatting, and those tokens are trivially predictable, so their logprobs sit close to zero and drag the average up regardless of whether the load bearing content is correct. A wrong answer wrapped in fluent boilerplate can carry a perfectly healthy mean logprob because ninety percent of its tokens were never in doubt. The average measures fluency far more than it measures truth.

Sampling temperature further muddies the raw number. Temperature rescales the distribution before sampling, so the same underlying uncertainty produces different logprobs at different temperature settings, and a confidence threshold tuned at one temperature is meaningless at another. If you are going to read logprobs for confidence, compute them from the model's actual distribution at a fixed low temperature, and keep that setting consistent, a discipline that connects directly to the sampler settings that govern the rest of your decode. The feature you want is not the average over everything. It is the behavior of the least confident tokens, the ones where the model actually hesitated.

The signals, measured

Here is a controlled comparison on an M5 Max running an 8B class instruct model quantized to Q4 through a local server, at temperature 0.2 with logprobs captured on every generated token. The harness runs a mix of factual questions where the ground truth answer is known, splits the generations into correct and incorrect, and asks a simple question of each candidate confidence feature: how well does it separate the two groups. The separation is reported as the gap between the median feature value on correct versus incorrect answers, normalized so larger means a cleaner split. These are harness figures meant to show the shape of each signal, not a guaranteed number for your model or prompt mix.

Confidence feature	Separates correct from wrong	Cost to compute	Notes
Mean token logprob	weak	free	dominated by predictable structural tokens
Minimum token logprob	moderate	free	catches the single most uncertain step
Mean logprob of content tokens only	strong	free	drop stopwords and punctuation first
Token entropy at each step	strongest	small extra	needs the full distribution, not just the sampled logprob
Answer span logprob (final answer only)	strong	free if you can locate the span	isolate the tokens that carry the claim

The pattern is consistent. Averaging over the whole generation is the worst usable signal because it is diluted by tokens that were never uncertain. The minimum token logprob does better because a single very low confidence token often sits exactly where the model made up a name, a number, or a fact. Restricting the average to content bearing tokens, dropping the punctuation and function words, sharpens it considerably. The strongest signal is per step entropy, the spread of the whole distribution rather than the probability of the one token that got sampled, because entropy sees a step where the model was torn between several answers even when the sampled token's own probability looks acceptable. Entropy costs slightly more since you have to read the top of the distribution rather than a single value, but on an M5 Max that cost is negligible next to the decode itself.

The extraction and abstention loop

The working pattern captures logprobs during decode, computes a content weighted uncertainty score, and gates the answer on it. The gate has three outcomes rather than two: accept, escalate, or abstain.

import math

# Fixed low temperature so scores are comparable across calls.
TEMPERATURE = 0.2
ACCEPT_ABOVE = -0.35   # mean content logprob; tune on your own labeled set
ABSTAIN_BELOW = -1.10  # below this, do not answer at all

def content_confidence(tokens):
    # tokens: list of (text, logprob) for the generated answer span.
    vals = [lp for (t, lp) in tokens if is_content(t)]  # drop punctuation/stopwords
    if not vals:
        return -math.inf
    return sum(vals) / len(vals)

def gate(answer_tokens):
    score = content_confidence(answer_tokens)
    if score >= ACCEPT_ABOVE:
        return ("accept", score)        # trust the local answer
    if score <= ABSTAIN_BELOW:
        return ("abstain", score)       # refuse rather than guess
    return ("escalate", score)          # send to a larger model or a human

Two details decide whether this holds up. The first is span isolation. The confidence you care about is the confidence in the claim, not in the surrounding explanation, so when your output format lets you locate the answer span, score that span alone. Structured output helps here, because a schema that puts the answer in a known field makes the load bearing tokens trivial to find, which is one more reason to reach for the constrained decoding patterns even when you do not strictly need them for parsing. The second is that both thresholds have to be set on your own labeled data. The numbers in the code are placeholders; the correct values depend on your model, your quantization, and the difficulty of your questions, and the only way to find them is to run a few hundred questions with known answers, plot the confidence scores for the correct and incorrect groups, and pick the cut points where the groups separate.

What logprobs cannot tell you

Logprobs measure the model's certainty, and certainty is not correctness. The failure mode that survives every confidence gate is the confident hallucination, where the model has internalized something false and reports it with high probability at every token. No logprob feature catches that, because the model genuinely is not uncertain; it is wrong and sure. Confidence gating raises the floor by catching the answers the model itself was unsure about, and those are a large share of errors in practice, but it does not catch the errors the model is serene about. For those you need grounding, a retrieval step that forces the answer to cite a source, and a check that the cited source actually supports the claim, which is a different tool from uncertainty and composes with it rather than replacing it. The same caution applies to calibration drift: a threshold tuned on one distribution of questions will mis fire on a different distribution, so treat the cut points as something you re measure when your traffic changes, not a constant you set once.

When this applies to your stack

Reach for logprob based confidence the moment a wrong answer from your local model costs more than no answer, which describes almost any assistant that states policies, answers from documents, or feeds a downstream automated step. Capture logprobs during decode, score the content tokens or the answer span rather than the whole generation, and gate on a three way accept, escalate, abstain decision rather than forcing every answer through. Set your thresholds on a few hundred labeled examples and re check them when your question mix shifts. Pair the gate with retrieval grounding so you catch both the errors the model was unsure about and, through source checking, some of the errors it was sure about. The result is a local model that knows when to keep quiet, which is the difference between an assistant a team can rely on and one they learn to double check on everything, at which point it has saved them nothing.

Building the confidence gate, calibrating its thresholds against your real traffic, and wiring the escalation path to a larger model or a human when the local model abstains, is the local inference work Contra Collective takes on when a prototype has to be trusted with answers that matter. The model gives you the uncertainty for free. Reading it correctly is the engineering.

FAQ

What is a token logprob? It is the natural log of the probability the model assigned to the token it actually generated, at the moment it generated it. A value near zero means the model was almost certain; a large negative value means it was spread across many options. The forward pass computes the full distribution to sample from, so the logprob is available at no extra cost, you just have to ask the server to return it.

Why not just average all the logprobs for a confidence score? Because most tokens in any generation are structural, punctuation, articles, formatting, and those are trivially predictable, so their high logprobs inflate the average regardless of whether the actual claim is correct. A fluent wrong answer can show a healthy mean logprob. Score the content tokens or the answer span instead, and consider per step entropy, which separates correct from incorrect answers more cleanly.

Do logprobs catch hallucinations? Only the uncertain ones. When a model made something up at a step where it was torn between options, the low logprob or high entropy flags it. But a confident hallucination, where the model has internalized something false and reports it with high probability, produces normal looking logprobs and slips through. Confidence gating needs to be paired with retrieval grounding and source checking to catch those.

Does reading logprobs slow down inference on an M5 Max? No measurably. The sampled token's logprob is already computed. Reading entropy costs slightly more because you inspect the top of the distribution rather than a single value, but that cost is negligible next to the decode step itself. The expensive part is not extraction, it is calibrating the thresholds on labeled data.

How do I choose the accept and abstain thresholds? Empirically. Run a few hundred questions with known answers through your exact model and quantization, compute the confidence score for each, and separate them into correct and incorrect groups. Plot both distributions and set the accept cut where correct answers cluster and the abstain cut where incorrect ones do. Re measure when your question mix changes, because the thresholds are specific to your traffic, not universal constants.

This post reports results from a controlled local test harness, not a production benchmark, and the figures will vary with your model, quantization, temperature, and question distribution; calibrate your own thresholds on labeled data before trusting a confidence gate.