The frequency_penalty parameter openai exposes is one of the most misunderstood knobs in the API. Most engineers treat it as a “repetition reducer” and crank it up when they see duplicate phrases, but the actual mechanism operates at the token level with linear scaling that behaves differently than intuition suggests. Understanding the exact math lets you tune it precisely instead of guessing.
The math behind the penalty
OpenAI applies frequency penalty during logit computation, before the softmax that produces probabilities. For each token in the vocabulary, the adjusted logit is:
adjusted_logit[token] = raw_logit[token] - frequency_penalty * count[token]Where count[token] is how many times that token has already appeared in the completion so far (not the prompt). The penalty scales linearly with frequency — a token seen twice gets double the penalty of a token seen once.
This is distinct from presence_penalty, which applies a flat cost once a token appears at least once. Frequency penalty compounds; presence penalty saturates.
# Simplified representation of what happens internally
def apply_frequency_penalty(logits, token_counts, penalty):
"""
logits: shape (vocab_size,) - raw model outputs
token_counts: dict[token_id, int] - occurrences in completion so far
penalty: float - the frequency_penalty value
"""
for token_id, count in token_counts.items():
if count > 0:
logits[token_id] -= penalty * count
return logitsThe parameter accepts values from -2.0 to 2.0. Positive values discourage repetition; negative values encourage it (useful for structured formats where you want repeated delimiters).
When frequency_penalty helps — and when it hurts
Good fits
Suppressing verbatim loops in long completions. If you’re generating a 2,000-token report and the model starts cycling through “in conclusion, in conclusion, in conclusion,” a modest penalty (0.3–0.6) breaks the cycle without distorting normal prose.
Discouraging overuse of common function names in code. When generating a Python module, the model may repeat def, return, or self more than natural. A penalty around 0.2–0.4 keeps syntax valid while reducing token bloat.
Reducing hallucinated list repetition. Prompts like “list 50 unique ideas” often produce duplicates. Frequency penalty helps, but combine it with a well-structured prompt that explicitly numbers items.
Poor fits
Creative writing where repetition is intentional. Poetry, rhetoric, and dialogue often rely on anaphora or refrain. Penalty flattens these deliberately repeated structures.
Short completions (< 200 tokens). The penalty barely activates because tokens don’t repeat enough to matter. You’re adding complexity for no gain.
JSON or rigid schema output. The model must repeat braces, brackets, commas, and field names. Penalty fights the format. Use presence_penalty at most, or neither.
Practical tuning guide
Start with the default (0.0). Only add penalty after you observe a specific repetition problem in production logs.
Step 1: Measure the baseline
Log completions with logprobs=true and top_logprobs=5. Compute the empirical token frequency distribution for a sample of 50–100 completions.
from collections import Counter
import tiktoken
def token_frequency_distribution(completions, model="gpt-4o"):
enc = tiktoken.encoding_for_model(model)
all_tokens = []
for text in completions:
all_tokens.extend(enc.encode(text))
return Counter(all_tokens)
# Example output snippet:
# Counter({264: 1847, 286: 1523, 345: 1201, ...}) # token_id: countIdentify tokens that appear disproportionately. The tokenizer merges common words into single tokens, so “the” (token 264) dominating is expected. Look for content tokens repeating abnormally.
Step 2: Apply minimal effective penalty
Test in this order: 0.1, 0.3, 0.5, 0.8. Stop when the target repetition disappears without degrading coherence.
# Quick curl test loop
for p in 0.1 0.3 0.5 0.8; do
echo "=== penalty=$p ==="
curl -s https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"gpt-4o\",
\"messages\": [{\"role\": \"user\", \"content\": \"Write a 500-word essay on distributed systems.\"}],
\"frequency_penalty\": $p,
\"max_tokens\": 800
}" | jq -r '.choices[0].message.content'
doneEvaluate outputs for:
- Target repetition: gone?
- Fluency: unnatural word choices, missing conjunctions, stilted flow?
- Semantic drift: does the model avoid the right word because it used a synonym earlier?
Step 3: Validate with automated checks
Add a regression test that flags regressions.
import pytest
def test_frequency_penalty_does_not_degrade_quality():
prompt = "Explain CAP theorem in three paragraphs."
baseline = generate(prompt, frequency_penalty=0.0)
penalized = generate(prompt, frequency_penalty=0.5)
# Heuristic: penalized should not be significantly shorter
# (model avoiding words it already used)
assert len(penalized) >= len(baseline) * 0.85
# Heuristic: no new grammatical errors (crude check)
assert penalized.count(".") >= baseline.count(".") * 0.7
# Target: specific token repetition reduced
baseline_counts = token_frequency_distribution([baseline])
penalized_counts = token_frequency_distribution([penalized])
# Content tokens (skip top 50 most common)
content_tokens_baseline = {t: c for t, c in baseline_counts.items() if t not in COMMON_TOKENS}
content_tokens_penalized = {t: c for t, c in penalized_counts.items() if t not in COMMON_TOKENS}
max_repetition_baseline = max(content_tokens_baseline.values())
max_repetition_penalized = max(content_tokens_penalized.values())
assert max_repetition_penalized < max_repetition_baseline * 0.7Common pitfalls
Pitfall 1: Confusing token-level with word-level repetition
The penalty operates on tokens, not words. “unhappiness” might be one token (or two: “un” + “happiness”). “New York” is two tokens. A penalty that stops “the” from repeating won’t stop “New York” from repeating if the model treats it as two separate tokens.
# Demonstration
enc = tiktoken.encoding_for_model("gpt-4o")
print(enc.encode("New York")) # [2356, 2357] -- two tokens
print(enc.encode("unhappiness")) # [4521] -- one token (model-dependent)
print(enc.encode("the the the")) # [264, 264, 264] -- same token thriceIf you need word-level deduplication, post-process or constrain via grammar/logit bias.
Pitfall 2: Over-penalizing high-frequency function words
At penalty ≥ 1.0, the model starts avoiding “the”, “and”, “of”, “to” — producing telegraphic, unnatural output. The fix isn’t lowering penalty globally; it’s using logit_bias to exempt specific tokens.
# Exempt the top 20 function tokens from penalty
FUNCTION_TOKENS = [264, 286, 287, 290, 307, 314, 316, 328, 336, 345,
356, 362, 366, 372, 382, 391, 402, 411, 417, 423]
payload = {
"model": "gpt-4o",
"messages": [...],
"frequency_penalty": 0.8,
"logit_bias": {str(t): 100 for t in FUNCTION_TOKENS}, # large positive bias ≈ exemption
}Note: logit_bias adds to logits after frequency penalty, so a sufficiently large bias cancels the penalty for those tokens. 100 is effectively infinite for practical purposes.
Pitfall 3: Assuming penalty persists across API calls
Frequency penalty only applies within a single completion. It does not carry over between separate API requests, even with the same conversation history. Each chat/completions call starts with a fresh token count.
If you’re building a multi-turn chat and want cross-turn repetition control, you must implement it yourself — either by feeding previous assistant messages as few-shot examples with instructions, or by post-processing.
Pitfall 4: Negative penalty for “more repetition” backfires
Setting frequency_penalty: -0.5 encourages repetition, but the model often latches onto low-information tokens (commas, “and”, “the”) rather than the structural tokens you want repeated. For JSON arrays or numbered lists, prefer explicit formatting instructions or response_format: { "type": "json_object" }.
Interaction with other sampling parameters
With temperature
High temperature (1.0+) + high frequency penalty (0.8+) = chaotic output. The model is already exploring low-probability tokens; the penalty pushes it further into the tail, producing nonsense words.
Rule: If temperature > 0.7, keep frequency_penalty ≤ 0.3.
With top_p
Top-p (nucleus sampling) truncates the vocabulary before penalty applies. If the penalty pushes a token’s probability mass outside the top-p window, it’s dropped entirely. This can cause abrupt vocabulary shifts mid-completion.
Rule: With top_p < 0.9, test penalty values more carefully — the interaction is non-linear.
With presence_penalty
They stack. A token seen 3 times with frequency_penalty=0.5 and presence_penalty=0.5 gets a total logit reduction of 0.5*3 + 0.5 = 2.0. This is rarely what you want. Pick one:
- frequency_penalty for “don’t say the same thing over and over”
- presence_penalty for “cover diverse topics/vocabulary”
Using both is a sign you haven’t diagnosed the actual repetition mode.
Testing strategy for production
Don’t tune in the playground. Build a small evaluation harness.
# eval_frequency_penalty.py
import json
from dataclasses import dataclass
from typing import List
@dataclass
class TestCase:
prompt: str
max_tokens: int
expected_min_length: int
forbidden_phrases: List[str] # phrases that indicate failure mode
TEST_CASES = [
TestCase(
prompt="Write a 300-word product description for a coffee maker.",
max_tokens=400,
expected_min_length=250,
forbidden_phrases=["coffee maker coffee maker", "brewing brewing", "features features"]
),
TestCase(
prompt="Generate a Python class for a binary search tree with insert, delete, search.",
max_tokens=600,
expected_min_length=300,
forbidden_phrases=["def def", "return return", "self. self."]
),
]
def evaluate(penalty: float) -> dict:
results = []
for tc in TEST_CASES:
out = generate(tc.prompt, frequency_penalty=penalty, max_tokens=tc.max_tokens)
# Length check
length_ok = len(out) >= tc.expected_min_length
# Forbidden phrase check
clean = out.lower()
repetition_found = any(p in clean for p in tc.forbidden_phrases)
results.append({
"prompt": tc.prompt[:50],
"length_ok": length_ok,
"repetition_found": repetition_found,
"output_preview": out[:200]
})
return {
"penalty": penalty,
"pass_rate": sum(1 for r in results if r["length_ok"] and not r["repetition_found"]) / len(results),
"details": results
}
if __name__ == "__main__":
for p in [0.0, 0.1, 0.2, 0.3, 0.5, 0.8]:
print(json.dumps(evaluate(p), indent=2))Run this against your actual prompts. The penalty that maximizes pass_rate is your production value.
One operational note
If you route through a gateway that sits in front of multiple providers (n4n.ai is one example), verify that the gateway forwards frequency_penalty unchanged to the upstream model. Some proxies clamp or ignore penalty parameters for non-OpenAI models that don’t support them natively. Check the gateway’s parameter mapping docs — or send a test request with logprobs=true and confirm the penalty effect appears in the returned logprobs.
Summary checklist
- Start at 0.0. Only increase after observing a specific repetition pattern in logs.
- Test 0.1 → 0.3 → 0.5 → 0.8. Stop at the first value that fixes the problem.
- Keep temperature ≤ 0.7 when using penalty > 0.3.
- Exempt function words via
logit_biasif penalty ≥ 0.5. - Don’t combine with presence_penalty unless you’ve measured the interaction.
- Validate with automated evals on your actual prompt templates, not generic benchmarks.
- Remember: penalty resets every API call. It does not persist across conversation turns.
The frequency_penalty parameter openai provides is a scalpel, not a sledgehammer. Used precisely, it eliminates the most annoying failure mode in long-form generation. Used blindly, it produces stilted, vocabulary-starved output that fails downstream quality checks. Measure, tune, lock it in.