Frequency penalty vs presence penalty, explained

Both frequency penalty and presence penalty suppress repetition, but they operate on different signals. Frequency penalty scales with how often a token has already appeared; presence penalty applies a flat cost once a token appears at all. Most engineers reach for one or the other without understanding the distinction, then wonder why their output still loops or why unique terms get suppressed. Here’s the mechanics, the trade-offs, and the decision framework.

How each penalty works

The OpenAI API (and every compatible endpoint) implements both penalties as additive logit biases applied before sampling. Let count(token) be the number of times a token has appeared in the generation so far (including the prompt). The adjusted logit for token t is:

logit'(t) = logit(t) - frequency_penalty * count(t) - presence_penalty * I(count(t) > 0)

Where I is the indicator function. Frequency penalty grows linearly with each occurrence. Presence penalty is a one-time step function: zero for unseen tokens, constant for any token that has appeared at least once.

This distinction matters because tokenization fragments words. “ing” appearing three times across “building”, “running”, “thinking” counts as three occurrences for frequency penalty, but only one for presence penalty (per unique token ID).

Frequency penalty: proportional suppression

Frequency penalty targets tokens that dominate the generation. It’s the right tool when a model fixates on a specific phrase, word, or subword and repeats it excessively. The penalty accumulates, so the fifth occurrence of “the” hurts more than the second.

Typical values range from 0.1 to 1.0. At 1.0, each prior occurrence subtracts 1.0 from the logit — roughly equivalent to dividing the probability by e (~2.718) per occurrence. At 2.0, you’re dividing by e² (~7.4) per occurrence, which aggressively flattens high-frequency tokens.

# Frequency penalty effect on logits (simplified)
import math

base_logit = 2.0  # ~88% probability before penalty
freq_penalty = 0.5

for count in range(1, 6):
    adjusted = base_logit - freq_penalty * count
    prob = 1 / (1 + math.exp(-adjusted))
    print(f"Count {count}: logit={adjusted:.2f}, prob={prob:.3f}")

# Output:
# Count 1: logit=1.50, prob=0.818
# Count 2: logit=1.00, prob=0.731
# Count 3: logit=0.50, prob=0.622
# Count 4: logit=0.00, prob=0.500
# Count 5: logit=-0.50, prob=0.378

The decay is smooth and predictable. This makes frequency penalty safer for long generations where you want gradual discouragement rather than a hard cutoff.

Presence penalty: binary discouragement

Presence penalty asks a yes/no question: has this token appeared? If yes, subtract a constant. It doesn’t care if the token appeared once or fifty times. This makes it effective at forcing vocabulary diversity — the model must reach for new tokens rather than reusing known ones.

Typical values also range 0.1 to 1.0, but the effect feels sharper because it applies immediately on the second occurrence. At 1.0, any token that has appeared once gets its logit reduced by 1.0, cutting its probability by ~63% regardless of how many times it appeared before.

# Presence penalty effect (binary)
base_logit = 2.0
pres_penalty = 0.8

for count in [0, 1, 2, 5, 10]:
    adjusted = base_logit - (pres_penalty if count > 0 else 0)
    prob = 1 / (1 + math.exp(-adjusted))
    print(f"Count {count}: logit={adjusted:.2f}, prob={prob:.3f}")

# Output:
# Count 0: logit=2.00, prob=0.881
# Count 1: logit=1.20, prob=0.768
# Count 2: logit=1.20, prob=0.768
# Count 5: logit=1.20, prob=0.768
# Count 10: logit=1.20, prob=0.768

Notice the plateau. Once a token is “seen,” its probability stays suppressed at the same level. This creates a stronger pressure toward novelty but can also suppress necessary function words (“the”, “and”, “to”) that must repeat for grammatical coherence.

Interaction with top-p and temperature

Both penalties apply before top-p (nucleus) sampling and temperature scaling. The order of operations in most implementations:

1. Compute raw logits from the model
2. Apply frequency and presence penalties
3. Apply temperature scaling (divide logits by temperature)
4. Apply top-p / top-k filtering
5. Sample from the resulting distribution

This means penalties can push tokens below the top-p threshold entirely, removing them from the candidate set. A token with 5% probability might drop to 1% after penalties, then get filtered out by top-p=0.9. This interaction is why high penalties combined with low top-p can cause degenerate output — the model runs out of eligible tokens.

# Pseudocode: typical sampling pipeline order
def sample_next_token(logits, temperature=1.0, top_p=0.9, 
                      freq_penalty=0.0, pres_penalty=0.0, 
                      token_counts=None):
    # 1. Apply penalties
    if token_counts is not None:
        for token_id, count in token_counts.items():
            if count > 0:
                logits[token_id] -= freq_penalty * count
                logits[token_id] -= pres_penalty
    
    # 2. Temperature
    logits = logits / temperature
    
    # 3. Top-p (nucleus) sampling
    probs = softmax(logits)
    sorted_probs, sorted_indices = torch.sort(probs, descending=True)
    cumsum_probs = torch.cumsum(sorted_probs, dim=-1)
    mask = cumsum_probs <= top_p
    mask[0] = True  # always keep at least one token
    filtered_probs = sorted_probs * mask
    filtered_probs = filtered_probs / filtered_probs.sum()
    
    # 4. Sample
    next_token = torch.multinomial(filtered_probs, 1)
    return sorted_indices[next_token]

Comparison table

When frequency penalty wins

Long-form generation with natural language. Articles, stories, emails — any output where grammatical function words must repeat. Frequency penalty lets “the” appear dozens of times with gradually diminishing probability, which matches natural language statistics.

Code generation. You need return, if, else, def, self to repeat. Presence penalty at >0.3 often breaks syntax by forcing the model to avoid these keywords. Frequency penalty at 0.1–0.3 reduces variable name repetition without breaking structure.

Reducing specific loops. If you observe the model cycling through “as mentioned previously, as mentioned previously, as mentioned previously,” frequency penalty directly targets that pattern. Presence penalty would also suppress “as”, “mentioned”, “previously” individually, which is overkill.

# Practical starting points for common tasks
TASK_PRESETS = {
    "chat": {"frequency_penalty": 0.0, "presence_penalty": 0.0},
    "article_writing": {"frequency_penalty": 0.3, "presence_penalty": 0.0},
    "creative_writing": {"frequency_penalty": 0.5, "presence_penalty": 0.3},
    "code_generation": {"frequency_penalty": 0.1, "presence_penalty": 0.0},
    "summarization": {"frequency_penalty": 0.2, "presence_penalty": 0.0},
    "brainstorming": {"frequency_penalty": 0.3, "presence_penalty": 0.5},
    "data_extraction": {"frequency_penalty": 0.0, "presence_penalty": 0.0},
}

When presence penalty wins

Brainstorming and ideation. You want distinct ideas, not variations on the same theme. Presence penalty forces the model to reach for new concepts. At 0.5–0.8, it produces noticeably more diverse bullet points, names, or angles.

Constrained vocabulary tasks. Generating tags, keywords, or categories where each output should be unique. Presence penalty naturally enforces a “no repeats” soft constraint.

Short completions where diversity > coherence. Autocomplete suggestions, title generation, or any task where the output is a set rather than a sequence.

Combating mode collapse in fine-tuned models. Some RLHF’d models collapse to a narrow set of preferred phrases (“I apologize for the confusion,” “Here’s what you need to know”). Presence penalty disrupts this more effectively than frequency penalty because the collapsed phrases often use different tokens each time but the same high-level tokens.

The danger zone: both high

Setting both penalties above 0.5 simultaneously is rarely useful. You get the worst of both: function words suppressed by presence penalty, content words suppressed by frequency penalty, and the model forced into low-probability tokens that produce incoherent output.

# What happens at extreme penalties (illustrative)
# Prompt: "The quick brown fox"
# frequency_penalty=1.5, presence_penalty=1.0, temperature=0.7

# Generation tends toward:
# "The quick brown fox jumps over lazy dog sleeping quietly beneath
#  moonlight shimmering across peaceful meadow..."
# 
# Grammatically valid but semantically drifting — the model
# keeps grabbing unused tokens to avoid penalties.

If you need strong repetition control, prefer frequency penalty up to 1.0–1.5 with presence penalty at 0.0–0.1. The linear scaling handles long contexts better than the binary hammer.

Tokenization artifacts

Both penalties operate on token IDs, not words. This creates edge cases:

• Subword fragmentation: “unbelievable” → “un”, “believ”, “able”. Frequency penalty counts each subword separately. Presence penalty treats each subword as a unique token.
• Case sensitivity: “The” and “the” are different tokens. Neither penalty connects them.
• Whitespace variants: “ hello“ (leading space) vs “hello” are different tokens.

For word-level repetition control, you need post-processing or a custom logit processor that aggregates by normalized word form. The built-in penalties are token-level approximations.

# Custom word-level frequency penalty (conceptual)
def word_level_frequency_penalty(logits, generated_text, penalty=0.5):
    """Apply penalty based on word forms, not token IDs."""
    from collections import Counter
    import re
    
    words = re.findall(r'\b\w+\b', generated_text.lower())
    word_counts = Counter(words)
    
    # Map token IDs to word forms (requires tokenizer)
    # This is pseudocode — actual implementation needs tokenizer access
    for token_id in range(logits.shape[-1]):
        word_form = tokenizer.decode([token_id]).strip().lower()
        if word_form in word_counts:
            logits[token_id] -= penalty * word_counts[word_form]
    
    return logits

Most engineers don’t need this. The token-level penalties work well enough for 95% of cases. But if you’re building a system where word-level repetition is a critical failure mode (legal documents, financial reports), the approximation gap matters.

Debugging penalty effects

When output feels wrong, isolate the penalty:

1. Set both to 0.0 — establish baseline quality
2. Add frequency penalty only — observe loop reduction
3. Add presence penalty only — observe diversity increase
4. Combine — check for over-suppression

Log the top-5 token probabilities at each step for a few generations. You’ll see exactly which tokens get pushed down and whether they’re the ones you intended to target.

# Quick test script pattern
for freq in 0.0 0.3 0.6 1.0; do
  for pres in 0.0 0.3 0.6; do
    echo "=== freq=$freq pres=$pres ==="
    python -c "
import openai
client = openai.OpenAI()
resp = client.completions.create(
    model='gpt-3.5-turbo-instruct',
    prompt='The quick brown fox',
    max_tokens=50,
    frequency_penalty=$freq,
    presence_penalty=$pres,
    temperature=0.7,
    logprobs=5
)
print(resp.choices[0].text)
"
  done
done

Which to choose

Default: neither. Most tasks need no penalty. Start at 0.0 for both. The model’s training distribution already encodes natural repetition statistics.

Frequency penalty 0.1–0.5 when:

• Generating >500 tokens of prose or code
• You observe specific phrase loops in logs
• Variable/function name repetition degrades code quality
• You need predictable, gradual suppression

Presence penalty 0.3–0.6 when:

• Brainstorming lists, names, ideas, angles
• Generating tags, keywords, categories
• Short completions where uniqueness > flow
• Fighting mode collapse in a fine-tuned model

Both low (0.1–0.2 each) when:

• Creative writing where you want variety without incoherence
• Long conversations where the model falls into verbal tics
• You’ve exhausted single-penalty tuning and need marginal improvement

Avoid presence penalty >0.3 for:

• Code generation (breaks syntax)
• Long-form factual writing (suppresses necessary entities)
• Any task where grammatical coherence matters

Avoid frequency penalty >1.5 unless:

• You’re deliberately degrading output for adversarial testing
• You have a custom logit processor that compensates

The penalties are blunt instruments. They don’t understand semantics, only token statistics, only token counts. Use them sparingly, measure the effect, and prefer prompt engineering or few-shot examples when the repetition problem is structural rather than statistical.