When you call a language model API and request logprobs: true, the response includes the natural logarithm of the probability assigned to each token the model considered at each generation step. These values let you measure model confidence, build constrained decoders, and evaluate output quality without running a separate judge model. Understanding what are logprobs and how to interpret them is foundational for any engineer moving beyond simple chat completion into production LLM systems.
How logprobs work in the API response
Most OpenAI-compatible APIs return logprobs as an array parallel to the generated tokens. Each entry contains the log probability of the sampled token plus, optionally, the top-k alternatives the model considered at that position. The values are natural logarithms, so they’re negative numbers closer to zero for higher probability.
{
"choices": [{
"logprobs": {
"content": [
{
"token": "The",
"logprob": -0.023,
"bytes": [84, 104, 101],
"top_logprobs": [
{"token": "The", "logprob": -0.023, "bytes": [84, 104, 101]},
{"token": "A", "logprob": -1.847, "bytes": [65]},
{"token": "This", "logprob": -2.104, "bytes": [84, 104, 105, 115]}
]
},
{
"token": " answer",
"logprob": -0.156,
"bytes": [32, 97, 110, 115, 119, 101, 114],
"top_logprobs": [
{"token": " answer", "logprob": -0.156, "bytes": [32, 97, 110, 115, 119, 101, 114]},
{"token": " response", "logprob": -1.234, "bytes": [32, 114, 101, 115, 112, 111, 110, 115, 101]},
{"token": " result", "logprob": -1.567, "bytes": [32, 114, 101, 115, 117, 108, 116]}
]
}
]
}
}]
}The logprob field is ln(P(token | context)). To recover the probability, exponentiate: prob = math.exp(logprob). The top_logprobs array shows the model’s next-best guesses, which is where most of the engineering value lives.
Why logprobs matter for production systems
Confidence scoring without a judge model
You can compute a sequence-level confidence score by averaging token logprobs (or summing them for joint log-likelihood). This lets you flag low-confidence generations for human review or automatic retry.
def sequence_confidence(logprobs_content: list[dict]) -> float:
"""Average token logprob as a confidence proxy."""
token_logprobs = [entry["logprob"] for entry in logprobs_content]
return sum(token_logprobs) / len(token_logprobs)
def joint_log_likelihood(logprobs_content: list[dict]) -> float:
"""Sum of token logprobs = log P(sequence | context)."""
return sum(entry["logprob"] for entry in logprobs_content)Averaging normalizes for length; summing gives you the true joint probability. Choose based on whether you’re comparing same-length completions (sum) or variable-length ones (average).
Constrained generation and structured output
Logprobs enable building your own constrained decoders. Instead of relying on the provider’s JSON mode or grammar enforcement, you can mask invalid tokens at each step by reading top_logprobs, zeroing out disallowed continuations, renormalizing, and sampling from the result. This is how you get guaranteed-valid JSON, SQL, or regex-matched output on models that don’t natively support constrained decoding.
def constrained_next_token(top_logprobs: list[dict], allowed_tokens: set[str]) -> str:
"""Pick highest-prob allowed token from top-k."""
for entry in top_logprobs:
if entry["token"] in allowed_tokens:
return entry["token"]
# Fallback: allow any token if none match (shouldn't happen with sufficient top_k)
return top_logprobs[0]["token"]Token-level eval and hallucination detection
Sharp drops in token logprob often correlate with hallucination boundaries — the model becomes uncertain exactly where it starts fabricating. You can build token-level heatmaps for human reviewers or automatic flagging.
def find_low_confidence_spans(logprobs_content: list[dict], threshold: float = -2.0) -> list[tuple[int, int]]:
"""Return (start_idx, end_idx) of contiguous low-confidence tokens."""
spans = []
start = None
for i, entry in enumerate(logprobs_content):
if entry["logprob"] < threshold:
if start is None:
start = i
elif start is not None:
spans.append((start, i - 1))
start = None
if start is not None:
spans.append((start, len(logprobs_content) - 1))
return spansCost-aware routing
If you’re routing requests across multiple models — say, a cheap model for easy queries and an expensive one for hard ones — logprobs from the cheap model’s first few tokens can serve as a difficulty signal. Low average logprob on the prefix suggests the cheap model is uncertain; escalate.
Concrete example: building a confidence-calibrated classifier
Suppose you’re classifying support tickets into categories using an LLM. You want to auto-resolve high-confidence predictions and route the rest to humans. Here’s a minimal pipeline:
import openai
import math
from dataclasses import dataclass
from typing import Literal
CATEGORIES = ["billing", "technical", "account", "other"]
CATEGORY_TOKENS = {c: f" {c}" for c in CATEGORIES} # space-prefixed for tokenization
@dataclass
class ClassificationResult:
category: str
confidence: float
calibrated_confidence: float
should_auto_resolve: bool
def classify_ticket(ticket_text: str, auto_resolve_threshold: float = 0.9) -> ClassificationResult:
prompt = f"""Classify this support ticket into one category: {', '.join(CATEGORIES)}.
Ticket: {ticket_text}
Category:"""
response = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
max_tokens=1,
temperature=0,
logprobs=True,
top_logprobs=len(CATEGORIES),
)
# The generated token should be one of our category tokens
content = response.choices[0].logprobs.content[0]
top_logprobs = content.top_logprobs
# Extract logprobs for our categories
cat_logprobs = {}
for entry in top_logprobs:
token = entry.token.strip()
if token in CATEGORIES:
cat_logprobs[token] = entry.logprob
# Convert to probabilities and normalize
logprobs_array = [cat_logprobs.get(c, -20.0) for c in CATEGORIES]
max_lp = max(logprobs_array)
probs = [math.exp(lp - max_lp) for lp in logprobs_array]
total = sum(probs)
probs = [p / total for p in probs]
best_idx = probs.index(max(probs))
best_category = CATEGORIES[best_idx]
confidence = probs[best_idx]
# Simple temperature calibration (Platt scaling would be better in production)
calibrated = confidence ** 0.8 # heuristic: models tend to be overconfident
return ClassificationResult(
category=best_category,
confidence=confidence,
calibrated_confidence=calibrated,
should_auto_resolve=calibrated >= auto_resolve_threshold,
)This pattern — single-token classification with top_logprobs covering your label set — is the cleanest way to get calibrated probabilities from an LLM. It avoids the variance of free-text generation and gives you a proper distribution over your label space.
Common misconceptions
Logprobs are not calibrated probabilities
A logprob of -0.1 (P ≈ 0.9) does not mean the model is correct 90% of the time. LLMs are notoriously overconfident, especially on out-of-distribution inputs. Treat logprobs as relative confidence within a single generation, not absolute calibration. If you need calibrated probabilities, run a calibration set through your pipeline and fit Platt scaling or isotonic regression.
Summing logprobs across tokens assumes independence
The joint log-likelihood sum(logprob_i) equals log P(tokens | context) only because the chain rule decomposes it that way: P(t1, t2, ...) = P(t1) * P(t2 | t1) * .... This is mathematically correct. But comparing joint log-likelihoods across different sequence lengths is meaningless — longer sequences naturally have lower joint probability. Always normalize by length (average) or use perplexity: exp(-average_logprob).
Top-k logprobs don’t sum to 1
The top_logprobs array shows only the k most likely tokens. The remaining probability mass (the “long tail”) is omitted. If you renormalize just the top-k, you’re implicitly assuming the tail has zero mass. For high-entropy positions (e.g., creative writing), the tail can be substantial. For low-entropy positions (e.g., classification, code), top-5 or top-10 usually captures >99% of the mass.
Logprobs reflect the sampling temperature
If you request temperature=0.7 and logprobs=true, the returned logprobs are from the temperature-scaled distribution, not the raw model logits. The API applies temperature before computing the logprobs you see. This means logprobs at temperature > 0 are “softer” (closer to uniform) than the model’s true beliefs. For confidence scoring, always use temperature=0 (or the lowest allowed) to get the sharpest distribution.
Bytes field is not always one byte per character
The bytes array shows the raw UTF-8 bytes of the token. A single token can be multiple bytes (emoji, non-Latin scripts) or a single byte (ASCII). Don’t assume len(bytes) == len(token) or that bytes maps 1:1 to characters. Use it for exact byte-level reconstruction if you’re building a tokenizer-aware diff tool; otherwise ignore it.
Practical tips for working with logprobs
Request enough top_logprobs. For classification with 20 categories, request top_logprobs=20. The default is often 5. If your label isn’t in the top-k, you get no signal for it.
Handle missing logprobs gracefully. Some providers don’t return logprobs for all models, or they omit top_logprobs when logprobs=true but top_logprobs isn’t specified. Write defensive code:
def safe_get_top_logprobs(choice) -> list[dict]:
if not choice.logprobs or not choice.logprobs.content:
return []
first = choice.logprobs.content[0]
return getattr(first, "top_logprobs", []) or []Cache logprob computations. If you’re running the same prompts repeatedly (e.g., few-shot classification), the logprobs for the prompt prefix are identical across calls. Some APIs let you return prompt logprobs (echo=true in older OpenAI APIs, prompt_logprobs in vLLM). Use them to avoid recomputing.
Watch for tokenizer artifacts. A “word” like “unhappiness” might be one token or three (un, happi, ness). Logprobs are per-token, not per-word. If you need word-level confidence, you must aggregate across subword tokens — typically by averaging or taking the minimum.
When not to use logprobs
Don’t reach for logprobs when:
- You need calibrated probabilities for high-stakes decisions (medical, legal, financial). Use a dedicated classifier with proper calibration.
- You’re comparing across different models or different prompts. Logprob scales shift with vocabulary size, context length, and training objective.
- The API you’re using doesn’t expose them. Some providers (Anthropic, Google) have limited or no logprob support. Design your pipeline to degrade gracefully.
Summary
Logprobs are the model’s internal confidence signal exposed at the token level. They’re free metadata on every generation — no extra inference cost — and they unlock confidence scoring, constrained decoding, hallucination detection, and cost-aware routing. The key is remembering they’re relative, uncalibrated, and temperature-dependent. Use them as engineering signals, not ground truth.