When a model produces garbage, hallucinates, or picks the wrong token, logprobs debugging model output is the fastest way to see what the model actually considered. Most engineers reach for prompt engineering or temperature tweaks first, but the log probability distribution tells you exactly why the model made each decision. This guide walks through extracting, interpreting, and acting on logprobs from OpenAI-compatible APIs.
Step 1: Enable logprobs in your request
Most OpenAI-compatible endpoints accept a logprobs boolean and an optional top_logprobs integer (typically 1–20) that returns the highest-probability alternatives at each position. Start with a minimal request to confirm the endpoint honors the parameter.
import os
import json
from openai import OpenAI
client = OpenAI(
api_key=os.getenv("N4N_API_KEY"),
base_url="https://api.n4n.ai/v1"
)
response = client.chat.completions.create(
model="meta-llama/llama-3.1-8b-instruct",
messages=[
{"role": "user", "content": "Complete: The capital of France is"}
],
max_tokens=10,
temperature=0.0,
logprobs=True,
top_logprobs=5
)
print(json.dumps(response.model_dump(), indent=2))Run this and verify the response contains a choices[0].logprobs object with content — an array of token objects, each carrying token, logprob, bytes, and a top_logprobs list of alternatives. If the field is missing, your provider or model may not support it; check the provider’s documentation or try a different model.
Step 2: Inspect the raw token distribution
The logprob field is the natural logarithm of the token’s probability. Convert to probability with exp(logprob) or compare logprobs directly — higher (less negative) means more likely. A typical token entry looks like this:
{
"token": " Paris",
"logprob": -0.0023,
"bytes": [32, 80, 97, 114, 105, 115],
"top_logprobs": [
{"token": " Paris", "logprob": -0.0023, "bytes": [32, 80, 97, 114, 105, 115]},
{"token": " Lyon", "logprob": -6.2, "bytes": [32, 76, 121, 111, 110]},
{"token": " Marseille", "logprob": -8.1, "bytes": [32, 77, 97, 114, 115, 101, 105, 108, 108, 101]}
]
}The model assigned ~99.8% probability to “ Paris“ (exp(-0.0023) ≈ 0.998). The alternatives are orders of magnitude less likely. This is a confident, correct completion.
Now try a prompt where the model might hallucinate:
response = client.chat.completions.create(
model="meta-llama/llama-3.1-8b-instruct",
messages=[
{"role": "user", "content": "Complete: The capital of Australia is"}
],
max_tokens=10,
temperature=0.0,
logprobs=True,
top_logprobs=5
)If the model outputs “ Sydney“ with logprob -0.1 but “ Canberra“ appears in top_logprobs at -2.5, the model knew the right answer but assigned it lower probability. That’s a knowledge retrieval issue, not a reasoning failure — different fix.
Step 3: Detect low-confidence spans programmatically
Write a helper that flags tokens where the top choice’s probability drops below a threshold, or where the gap to the second-best choice is small. Both signal uncertainty.
import math
def find_uncertain_tokens(logprobs_content, prob_threshold=0.5, gap_threshold=0.1):
"""
Returns list of (index, token, prob, gap) for uncertain tokens.
prob_threshold: flag if top token probability < this
gap_threshold: flag if (top_prob - second_prob) < this
"""
uncertain = []
for i, entry in enumerate(logprobs_content):
top = entry
top_prob = math.exp(top.logprob)
second_prob = math.exp(top.top_logprobs[1].logprob) if len(top.top_logprobs) > 1 else 0.0
gap = top_prob - second_prob
if top_prob < prob_threshold or gap < gap_threshold:
uncertain.append((i, top.token, top_prob, gap))
return uncertain
# Usage
uncertain = find_uncertain_tokens(response.choices[0].logprobs.content)
for idx, tok, prob, gap in uncertain:
print(f"Position {idx}: '{tok}' prob={prob:.3f} gap={gap:.3f}")Run this on a hallucinated answer. You’ll often see a cascade: the first wrong token has moderate probability (0.3–0.6), then subsequent tokens lock in with high probability because the model is now completing its own mistake. The first low-confidence token is the intervention point.
Step 4: Compare temperatures with logprobs
Temperature reshapes the distribution. At temperature 0, the model picks argmax. At higher temperatures, probability mass spreads. Logprobs let you quantify this without guessing.
def get_token_distribution(prompt, model, temperatures=[0.0, 0.3, 0.7, 1.0]):
results = {}
for temp in temperatures:
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=5,
temperature=temp,
logprobs=True,
top_logprobs=10
)
first_token = resp.choices[0].logprobs.content[0]
dist = {alt.token: math.exp(alt.logprob) for alt in first_token.top_logprobs}
results[temp] = dist
return results
dists = get_token_distribution(
"Complete: The inventor of the light bulb was",
"meta-llama/llama-3.1-8b-instruct"
)
for temp, dist in dists.items():
print(f"\nTemperature {temp}:")
for tok, prob in sorted(dist.items(), key=lambda x: -x[1])[:5]:
print(f" {tok!r:15} {prob:.4f}")You’ll see the distribution flatten as temperature rises. If “ Edison“ drops from 0.95 to 0.6 and “ Tesla“ rises from 0.02 to 0.15, you now have data to decide whether temperature 0.3 is a sweet spot for your use case — or whether the prompt itself needs disambiguation.
Step 5: Diagnose structural failures with position-level entropy
Entropy at each position measures how “spread out” the distribution is. High entropy means the model is genuinely uncertain; low entropy with a wrong answer means the model is confidently wrong. Compute per-position entropy from the top-k logprobs:
def position_entropy(logprobs_content):
entropies = []
for entry in logprobs_content:
probs = [math.exp(alt.logprob) for alt in entry.top_logprobs]
# Normalize in case top_logprobs doesn't sum to 1
total = sum(probs)
probs = [p / total for p in probs]
h = -sum(p * math.log(p) for p in probs if p > 0)
entropies.append((entry.token, h))
return entropies
entropies = position_entropy(response.choices[0].logprobs.content)
for tok, h in entropies:
print(f"{tok!r:10} entropy={h:.3f}")Interpretation guide:
- Entropy < 0.1: Near-deterministic. Model is committed.
- Entropy 0.1–1.0: Moderate uncertainty. Model is choosing among a few plausible options.
- Entropy > 1.0: High uncertainty. Model is guessing or the context is ambiguous.
Plot entropy across a generation. A sudden spike often marks where the model “goes off the rails” — useful for truncation strategies or for inserting a verification step in a chain-of-thought pipeline.
Step 6: Use logprobs to validate structured output
When you need JSON, function calls, or constrained formats, logprobs reveal whether the model understands the syntax or is just guessing brackets. Request logprobs on a structured completion:
response = client.chat.completions.create(
model="meta-llama/llama-3.1-8b-instruct",
messages=[
{"role": "system", "content": "Output only valid JSON with keys: name, age, city"},
{"role": "user", "content": "John, 30, New York"}
],
max_tokens=50,
temperature=0.0,
logprobs=True,
top_logprobs=5
)
for entry in response.choices[0].logprobs.content:
top = entry.top_logprobs[0]
print(f"{entry.token!r:8} top={top.token!r:8} p={math.exp(top.logprob):.3f}")Look for structural tokens ({, ", :, ,, }) with probability near 1.0. If the model assigns 0.4 to { and 0.3 to [, it’s confused about the format — add a few-shot example or switch to a model with better instruction following. If structural tokens are near 1.0 but field values are wrong, the problem is extraction, not formatting.
Step 7: Detect tokenization artifacts
Sometimes “unexpected output” is actually a tokenization quirk. Logprobs expose the bytes field, which shows the raw UTF-8 bytes for each token. Use this to spot:
- Whitespace sensitivity:
" Paris"(leading space) vs"Paris"— different tokens, different probabilities - Case splits:
"Paris"vs" Paris"vs"PARIS"— may fragment across tokens - Non-ASCII: Accented characters often split into multiple tokens
def show_tokenization(logprobs_content):
for entry in logprobs_content:
bytes_repr = bytes(entry.bytes).decode('utf-8', errors='replace')
print(f"token={entry.token!r:12} bytes={bytes_repr!r:12} logprob={entry.logprob:.4f}")
show_tokenization(response.choices[0].logprobs.content)If you see the model assigning mass to " Paris" and "Paris" separately, your prompt may inconsistently include leading spaces. Normalize whitespace in your template or use a tokenizer-aware prompt builder.
Step 8: Build a logprob-based retry/fallback policy
Now that you can measure confidence, automate recovery. A simple policy: if the first token’s probability < 0.7 or entropy > 0.8, retry with lower temperature or a clarified prompt.
def should_retry(logprobs_content, first_token_prob_thresh=0.7, entropy_thresh=0.8):
if not logprobs_content:
return True
first = logprobs_content[0]
first_prob = math.exp(first.logprob)
# Compute entropy of first position
probs = [math.exp(alt.logprob) for alt in first.top_logprobs]
total = sum(probs)
probs = [p / total for p in probs]
entropy = -sum(p * math.log(p) for p in probs if p > 0)
return first_prob < first_token_prob_thresh or entropy > entropy_thresh
def complete_with_fallback(prompt, model, max_retries=2):
for attempt in range(max_retries + 1):
temp = 0.0 if attempt == 0 else 0.3 * attempt
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=100,
temperature=temp,
logprobs=True,
top_logprobs=10
)
if not should_retry(resp.choices[0].logprobs.content):
return resp.choices[0].message.content
# Optionally: modify prompt for next attempt
prompt += "\n\nThink carefully before answering."
return resp.choices[0].message.content # Last attemptThis pattern — measure, decide, retry — turns logprobs from a debugging tool into a runtime control signal.
Step 9: Log logprobs for production observability
Don’t just use logprobs interactively. Persist them alongside requests and responses. A minimal log entry:
{
"request_id": "req_abc123",
"model": "meta-llama/llama-3.1-8b-instruct",
"prompt_hash": "sha256:...",
"completion_tokens": 42,
"first_token_prob": 0.998,
"mean_entropy": 0.12,
"max_entropy": 0.45,
"low_confidence_positions": [15, 16, 17],
"timestamp": "2024-01-15T10:23:45Z"
}Aggregate mean_entropy and low_confidence_positions across traffic. Spikes correlate with:
- Prompt template regressions
- Model version changes (even minor ones)
- Input distribution shift (new user intents, OOD data)
- Provider-side degradation
If you route through a gateway that exposes per-token usage and provider health signals, you can correlate logprob degradation with provider latency or error rates — distinguishing model confusion from infrastructure issues.
Step 10: Verify your debugging workflow
Create a test suite of known-good and known-bad prompts. For each, assert expected logprob patterns:
import pytest
TEST_CASES = [
{
"name": "capital_france",
"prompt": "Complete: The capital of France is",
"expected_first_token": " Paris",
"min_first_prob": 0.95,
"max_entropy": 0.1
},
{
"name": "ambiguous_completion",
"prompt": "Complete: The best programming language is",
"max_first_prob": 0.4, # Should be uncertain
"min_entropy": 1.0
},
{
"name": "json_format",
"prompt": 'Output JSON: {"name": "John"}',
"structural_tokens": ["{", "\"", ":", "}", "\""],
"min_structural_prob": 0.99
}
]
@pytest.mark.parametrize("case", TEST_CASES)
def test_logprob_expectations(case):
resp = client.chat.completions.create(
model="meta-llama/llama-3.1-8b-instruct",
messages=[{"role": "user", "content": case["prompt"]}],
max_tokens=20,
temperature=0.0,
logprobs=True,
top_logprobs=10
)
content = resp.choices[0].logprobs.content
first = content[0]
first_prob = math.exp(first.logprob)
if "expected_first_token" in case:
assert first.token == case["expected_first_token"]
if "min_first_prob" in case:
assert first_prob >= case["min_first_prob"]
if "max_first_prob" in case:
assert first_prob <= case["max_first_prob"]
if "max_entropy" in case or "min_entropy" in case:
probs = [math.exp(alt.logprob) for alt in first.top_logprobs]
total = sum(probs)
probs = [p / total for p in probs]
entropy = -sum(p * math.log(p) for p in probs if p > 0)
if "max_entropy" in case:
assert entropy <= case["max_entropy"]
if "min_entropy" in case:
assert entropy >= case["min_entropy"]
if "structural_tokens" in case:
for i, expected_tok in enumerate(case["structural_tokens"]):
assert content[i].token == expected_tok
assert math.exp(content[i].logprob) >= case["min_structural_prob"]Run this in CI. When a model upgrade or prompt change breaks an assertion, you’ll know exactly which token distribution shifted — no guesswork.
Logprobs turn “the model did something weird” into “the model assigned 0.12 probability to the correct token at position 3 because the prompt omitted a disambiguating context.” That specificity is what lets you fix the root cause instead of layering prompt patches. Start instrumenting every completion path with the helpers above; the first time you catch a silent degradation before users report it, the investment pays off.