Logprobs model confidence is the most direct signal an LLM gives you about its own certainty, yet most teams either ignore it or misuse it. This guide walks through extracting logprobs from major providers, converting them into calibrated probabilities, and building sequence-level confidence metrics you can actually trust in production.
Step 1: Understand what logprobs actually are
Logprobs (log probabilities) are the natural logarithm of the probability the model assigns to each token at each generation step. If a model outputs token t with probability p, the logprob is ln(p). Since probabilities range from 0 to 1, logprobs are always negative (or zero for certainty).
Why log space? Numerical stability. Multiplying dozens of small probabilities underflows to zero; summing logprobs stays tractable. The model’s internal softmax outputs logits — unnormalized scores — and logprobs are those logits after log-softmax normalization.
Key distinction: logprobs reflect the model’s internal belief given its training and the prompt context. They are not calibrated confidence in the statistical sense. A model can be 99% confident and completely wrong. Treat logprobs as a consistency signal, not ground truth.
Step 2: Enable logprobs in your API request
Most OpenAI-compatible APIs accept a logprobs boolean and optional top_logprobs integer. Set logprobs=true to get the chosen token’s logprob; set top_logprobs=N to also receive the top N alternative tokens at each position with their logprobs.
import os
from openai import OpenAI
client = OpenAI(
api_key=os.getenv("N4N_API_KEY"),
base_url="https://api.n4n.ai/v1" # OpenAI-compatible endpoint
)
response = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[
{"role": "user", "content": "What is the capital of France?"}
],
max_tokens=50,
temperature=0.0, # deterministic for reproducible logprobs
logprobs=True,
top_logprobs=5
)Temperature matters. At temperature=0, the model picks the argmax token and logprobs reflect the raw softmax output. At higher temperatures, the sampling distribution flattens and logprobs no longer match the greedy path. For confidence measurement, use temperature=0 or very low values.
Step 3: Parse the response structure
The response includes a choices[0].logprobs object with a content array. Each element corresponds to one generated token and contains:
token: the generated token stringlogprob: log probability of that tokenbytes: UTF-8 byte representation (useful for multibyte tokens)top_logprobs: array of alternative tokens with their logprobs (if requested)
def extract_token_logprobs(response):
"""Extract (token, logprob) pairs from the first choice."""
if not response.choices[0].logprobs or not response.choices[0].logprobs.content:
return []
return [
(entry.token, entry.logprob)
for entry in response.choices[0].logprobs.content
]
token_logprobs = extract_token_logprobs(response)
for token, lp in token_logprobs[:10]:
print(f"{repr(token):>12} logprob={lp:.4f} prob={math.exp(lp):.4f}")Output example:
'P' logprob=-0.0001 prob=0.9999
'a' logprob=-0.0002 prob=0.9998
'r' logprob=-0.0003 prob=0.9997
'i' logprob=-0.0004 prob=0.9996
's' logprob=-0.0005 prob=0.9995Note: the first few tokens of a high-confidence completion often have logprobs extremely close to zero (probability ~1.0). This is normal for deterministic continuations.
Step 4: Convert logprobs to probabilities and confidence scores
Convert each token’s logprob to linear probability with exp(logprob). This gives you per-token confidence. But single-token confidence is noisy — short tokens like “the” or “a” naturally have high probability. You need aggregation.
import math
from dataclasses import dataclass
from typing import List
@dataclass
class TokenConfidence:
token: str
logprob: float
probability: float
@property
def confidence_pct(self) -> float:
return self.probability * 100
def compute_token_confidences(token_logprobs: List[tuple]) -> List[TokenConfidence]:
results = []
for token, lp in token_logprobs:
prob = math.exp(lp)
results.append(TokenConfidence(token, lp, prob))
return results
confidences = compute_token_confidences(token_logprobs)
for tc in confidences[:15]:
print(f"{repr(tc.token):>12} {tc.confidence_pct:6.2f}%")Step 5: Aggregate token-level confidence into sequence-level metrics
A single number for the whole completion is more useful than a per-token list. Three common aggregations serve different purposes:
Mean token probability — arithmetic mean of exp(logprob) across all tokens. Intuitive but skewed by high-probability function words.
Geometric mean (perplexity-based) — exp(mean(logprobs)). This is the inverse of perplexity. Penalizes any low-probability token heavily. Better for detecting hallucination or uncertainty.
Minimum token probability — the weakest link. Useful as a conservative gate: if any token falls below a threshold, flag the whole generation.
from statistics import mean
from typing import Optional
@dataclass
class SequenceConfidence:
mean_probability: float
geometric_mean_probability: float
min_probability: float
token_count: int
sum_logprobs: float
@property
def mean_confidence_pct(self) -> float:
return self.mean_probability * 100
@property
def geometric_confidence_pct(self) -> float:
return self.geometric_mean_probability * 100
@property
def min_confidence_pct(self) -> float:
return self.min_probability * 100
def aggregate_confidence(confidences: List[TokenConfidence]) -> SequenceConfidence:
if not confidences:
return SequenceConfidence(0, 0, 0, 0, 0)
probs = [c.probability for c in confidences]
logprobs = [c.logprob for c in confidences]
mean_prob = mean(probs)
geo_mean_prob = math.exp(mean(logprobs))
min_prob = min(probs)
return SequenceConfidence(
mean_probability=mean_prob,
geometric_mean_probability=geo_mean_prob,
min_probability=min_prob,
token_count=len(confidences),
sum_logprobs=sum(logprobs)
)
seq_conf = aggregate_confidence(confidences)
print(f"Mean confidence: {seq_conf.mean_confidence_pct:.2f}%")
print(f"Geometric mean conf: {seq_conf.geometric_confidence_pct:.2f}%")
print(f"Min token confidence: {seq_conf.min_confidence_pct:.2f}%")
print(f"Token count: {seq_conf.token_count}")Which to use? For a general-purpose confidence score, geometric mean is the strongest single metric — it correlates with perplexity and penalizes uncertainty anywhere in the sequence. For gating (e.g., “only auto-accept if confident”), use minimum token probability with a threshold like 0.1 or 0.05. Mean probability is fine for dashboards but overstates confidence on typical text.
Step 6: Handle edge cases and provider differences
Real-world usage requires handling several complications:
Missing logprobs: Some models or providers don’t support logprobs, or return null for certain tokens (e.g., special tokens, whitespace-only tokens). Guard against None.
Byte-level tokens: Multibyte UTF-8 characters may appear as multiple tokens or with bytes field populated. The token field may show a replacement character. Use bytes for accurate display if needed.
Provider-specific fields: OpenAI returns top_logprobs as a list of objects with token and logprob. Anthropic’s /messages endpoint returns logprobs in a different structure. If you’re routing through a gateway like n4n.ai that normalizes to OpenAI format, you get consistent structure across 240+ models — but verify the fields exist before accessing.
def safe_extract_logprobs(response) -> List[tuple]:
"""Extract logprobs defensively across provider variations."""
try:
choice = response.choices[0]
if not choice.logprobs or not choice.logprobs.content:
return []
results = []
for entry in choice.logprobs.content:
# Some providers may omit logprob for special tokens
if entry.logprob is None:
continue
token = entry.token or (entry.bytes.decode('utf-8', errors='replace') if entry.bytes else '')
results.append((token, entry.logprob))
return results
except (AttributeError, IndexError, TypeError):
return []Streaming responses: With stream=True, logprobs arrive in each chunk. Accumulate them the same way, but note that some providers only send logprobs on the final chunk or omit them during streaming. Test your specific model.
Context length effects: Logprobs for early tokens in a long completion are conditioned on less context than later tokens. This is inherent to autoregressive generation — don’t overinterpret position-based variation.
Step 7: Verify your implementation works
Build a test harness with known inputs to validate your pipeline end-to-end.
def test_confidence_pipeline():
"""Verify logprob extraction and aggregation with deterministic prompts."""
# High-confidence completion (factual recall)
response_high = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Complete: The capital of France is"}],
max_tokens=10,
temperature=0.0,
logprobs=True,
top_logprobs=3
)
# Low-confidence completion (ambiguous continuation)
response_low = client.chat.completions.create(
model="meta-llama/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Write a creative story about a"}],
max_tokens=20,
temperature=0.8, # higher temp = lower confidence
logprobs=True,
top_logprobs=3
)
for label, resp in [("High confidence", response_high), ("Low confidence", response_low)]:
token_lps = safe_extract_logprobs(resp)
confidences = compute_token_confidences(token_lps)
agg = aggregate_confidence(confidences)
print(f"\n{label}:")
print(f" Tokens: {agg.token_count}")
print(f" Mean: {agg.mean_confidence_pct:.1f}%")
print(f" Geo: {agg.geometric_confidence_pct:.1f}%")
print(f" Min: {agg.min_confidence_pct:.1f}%")
# Sanity checks
assert agg.token_count > 0, "Should have tokens"
assert 0 <= agg.mean_probability <= 1, "Probability in range"
assert agg.geometric_mean_probability <= agg.mean_probability, "Geo mean <= arithmetic mean"
test_confidence_pipeline()Expected pattern: the factual completion shows geometric mean > 90%, minimum > 50%. The creative prompt at temperature 0.8 shows geometric mean < 50%, minimum often < 10%. If your numbers don’t follow this pattern, check:
temperature=0for the high-confidence testlogprobs=Trueandtop_logprobsset in both requests- Model actually supports logprobs (some don’t)
Step 8: Put it in production — thresholds and fallbacks
Confidence scores are only useful if you act on them. Define clear thresholds for your use case:
CONFIDENCE_THRESHOLDS = {
"auto_accept": 0.85, # geometric mean >= 85% -> no human review
"flag_review": 0.50, # geometric mean < 50% -> escalate
"reject": 0.10, # min token prob < 10% -> hard stop
}
def route_by_confidence(seq_conf: SequenceConfidence) -> str:
if seq_conf.min_probability < CONFIDENCE_THRESHOLDS["reject"]:
return "REJECT"
if seq_conf.geometric_mean_probability >= CONFIDENCE_THRESHOLDS["auto_accept"]:
return "AUTO_ACCEPT"
if seq_conf.geometric_mean_probability < CONFIDENCE_THRESHOLDS["flag_review"]:
return "HUMAN_REVIEW"
return "CONDITIONAL_ACCEPT"Log the confidence metrics alongside every generation. Over time, correlate them with downstream quality signals (user feedback, error rates, hallucination detection) to calibrate your thresholds. What “85% geometric mean” means for your model, your prompts, and your task will differ from generic benchmarks.
Step 9: Advanced — calibrate with temperature scaling
Raw logprobs are often overconfident. If you have labeled data (correct/incorrect generations), fit a temperature scaling parameter T on a validation set:
def calibrate_logprobs(logprobs: List[float], temperature: float) -> List[float]:
"""Apply temperature scaling to logprobs."""
return [lp / temperature for lp in logprobs]
# Find T that minimizes NLL on held-out calibration set
# Typical values: 1.0 (no calibration) to 2.5 (heavy calibration)This is the same technique used to calibrate classification heads. For generative tasks, apply it per-token before aggregation. Only worth it if you have sufficient evaluation data and confidence decisions are high-stakes.
You now have a complete pipeline: request logprobs, parse defensively, convert to probabilities, aggregate with geometric mean, set thresholds, and verify with test cases. The geometric mean of token probabilities is your workhorse metric — it’s mathematically grounded, interpretable, grounded (inverse perplexity), and sensitive to uncertainty anywhere in the generation. Start there, log everything, and tune thresholds against real outcomes.