Sampling parameters code generation quality more than most engineers realize. The difference between a working function and a hallucinated import often comes down to three numbers: temperature, top-p, and repetition penalty. Most teams ship with defaults (0.7, 0.95, 1.0) and wonder why their generated tests are flaky or their refactors drift. This post breaks down each parameter’s mechanics, shows how they interact, and gives you concrete starting points for the code tasks you actually run.
Temperature controls the entropy floor
Temperature scales the logits before softmax. At 0.0 you get argmax — deterministic, greedy decoding. As temperature rises, the distribution flattens, giving lower-probability tokens a fighting chance. For code, this is the single most impactful knob.
# OpenAI-compatible call
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Write a Python LRU cache"}],
temperature=0.0, # deterministic
max_tokens=500,
)At 0.0–0.2: You get the modal completion. Same prompt, same output, every time. This is what you want for:
- Boilerplate generation (Django models, SQLAlchemy schemas, Protobuf stubs)
- Syntax-heavy transforms (convert this JSON to Pydantic, rewrite this callback to async/await)
- CI gate tasks where non-determinism breaks caching
At 0.3–0.5: You introduce controlled variation. The model still prefers high-probability tokens but can explore alternative valid patterns — different loop structures, different stdlib choices. Good for:
- Generating multiple test cases for the same function
- Exploring alternative implementations when you’ll review and pick one
At 0.7+: The model starts sampling from the long tail. You get creative variable names, unusual control flow, and — critically — hallucinated APIs that look plausible. I’ve seen 0.8 invent requests.async_get() and pandas.DataFrame.upsert() with convincing docstrings.
Rule of thumb: If you’re generating code that must compile and pass tests on the first try, stay ≤ 0.2. If you’re brainstorming approaches and have a human in the loop, 0.4–0.6 is fine. Never ship 0.7+ to automation.
Top-p (nucleus sampling) truncates the long tail
Top-p accumulates probability mass from the most likely tokens downward until it hits the threshold p, then renormalizes and samples from that nucleus. It’s adaptive: on a sharp distribution (the next token is obviously )), the nucleus is tiny. On a flat distribution (what variable name here?), it expands.
# Nucleus sampling at 0.95 — standard default
response = client.chat.completions.create(
model="claude-3-5-sonnet-20241022",
messages=[{"role": "user", "content": "Implement binary search"}],
temperature=0.3,
top_p=0.95,
max_tokens=800,
)Top-p = 1.0 disables nucleus sampling entirely — you sample from the full vocabulary (subject to temperature). This is rarely what you want for code.
Top-p = 0.9–0.95 is the sweet spot for most code tasks. It chops off the absolute garbage tokens (random Unicode, obvious syntax errors) while preserving legitimate alternatives. The model can still choose between for i in range(n) and for i, _ in enumerate(items) because both sit comfortably in the top 95% mass.
Top-p = 0.5–0.8 forces the model into a tighter set of high-confidence tokens. Use this when:
- You’re generating code in a constrained DSL (Terraform, Kubernetes YAML, GraphQL schemas) where deviation is costly
- You’re doing few-shot prompting with very similar examples and want the model to stick to the pattern
Top-p < 0.3 starts producing repetitive, stilted output. The model loses the ability to express necessary variation — loop variables, error messages, docstring phrasing — and you get copy-paste artifacts.
Top-k is a blunt instrument; prefer top-p
Top-k keeps only the k most likely tokens, full stop. It doesn’t adapt to distribution shape. On a sharp peak, top-k=50 might still only see 2 tokens with real mass. On a flat spot, it forces selection from 50 tokens even if the top 5 cover 90% probability.
# Top-k example — rarely needed for code
response = client.chat.completions.create(
model="codellama-34b",
messages=[{"role": "user", "content": "Quick sort in Rust"}],
temperature=0.2,
top_k=20, # hard cutoff
top_p=1.0, # disable nucleus
max_tokens=600,
)When top-k matters: Some older models (CodeLlama, StarCoder, early GPT-3.5) expose top-k but not top-p, or have buggy top-p implementations. If you’re stuck with such a model, top-k=40–50 is a reasonable default.
When to combine: Setting both top-k and top-p applies both filters (intersection). This is almost never useful for code — it just adds a second way to accidentally truncate valid tokens. Pick one. Top-p is the better default.
Repetition penalties break loops, not logic
Repetition penalty (sometimes called frequency_penalty / presence_penalty in OpenAI’s API) down-weights tokens that have already appeared in the context. The mechanism varies by provider, but the effect is the same: the model becomes reluctant to repeat n-grams.
# OpenAI-style penalties
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Generate 20 unit tests for parse_date"}],
temperature=0.4,
top_p=0.95,
frequency_penalty=0.5, # linear penalty per occurrence
presence_penalty=0.3, # binary penalty: seen or not seen
max_tokens=2000,
)Frequency penalty scales with count. If self. appears 50 times, its logit gets hit 50× the penalty value. Good for:
- Long generations where the model falls into
self.self.self.loops - Preventing variable name reuse in generated test suites (
test_case_1,test_case_2…)
Presence penalty is binary. Once a token appears, it gets a fixed penalty regardless of count. Better for:
- Forcing vocabulary diversity in docstrings and comments
- Preventing the model from re-using the same error message template
Values: 0.0–1.0 is the practical range. Above 1.0 the model starts avoiding necessary repetition — language keywords, indentation, common prefixes like async def. You’ll get syntactically broken code as the model contorts to avoid “the” or “return”.
Critical interaction: Repetition penalties apply to the entire context, including your prompt. If your few-shot examples use snake_case heavily and you set presence_penalty=0.8, the model will avoid snake_case in the completion — even if that’s the project convention. Test with your actual prompt templates.
The interaction matrix: what actually happens when you combine them
Parameters don’t act independently. Here’s what I’ve observed across GPT-4o, Claude 3.5 Sonnet, and CodeLlama-34B:
| Combination | Behavior | Use case |
|---|---|---|
| temp=0.0, top-p=1.0 | Pure greedy. Deterministic, sometimes gets stuck in local optima (repeating a line). | Golden-path generation, schema derivation |
| temp=0.2, top-p=0.95 | Default recommendation. Enough entropy to escape micro-loops, tight enough to stay syntactically valid. | Most code generation tasks |
| temp=0.5, top-p=0.95 | Explores alternatives but top-p reins in the tail. Good diversity, low hallucination. | Generating 5–10 variants for human review |
| temp=0.7, top-p=0.9 | High entropy + loose nucleus. Hallucination rate spikes. | Avoid for code |
| temp=0.3, top-p=0.5 | Tight nucleus. Model feels “constrained” — picks obvious tokens even when context suggests otherwise. | Constrained DSLs, strict pattern adherence |
| temp=0.2, top-p=0.95, freq_pen=0.3 | Breaks repetition loops without distorting vocabulary. | Long file generation (>1500 tokens) |
The trap: Raising temperature and lowering top-p (e.g., temp=0.8, top-p=0.5) doesn’t “balance out.” You get a flat distribution then a hard truncation — the model samples uniformly from a weird subset. Output feels random but not creative.
Per-task starting configurations
These are not universal constants. They’re starting points that work across the major models I’ve tested. Treat them as priors — run 20–50 generations, inspect failure modes, then adjust.
Boilerplate & scaffolding (models, migrations, config)
{
"temperature": 0.0,
"top_p": 1.0,
"frequency_penalty": 0.0,
"presence_penalty": 0.0
}Determinism > creativity. You want the same Pydantic model every run so your diffs are clean.
Algorithm implementation (sorting, graph traversal, parsing)
{
"temperature": 0.15,
"top_p": 0.95,
"frequency_penalty": 0.1,
"presence_penalty": 0.0
}Slight entropy lets the model pick between valid approaches (recursive vs iterative, heap vs sort) without hallucinating APIs.
Test generation
{
"temperature": 0.4,
"top_p": 0.95,
"frequency_penalty": 0.4,
"presence_penalty": 0.2
}You want variation here — different edge cases, different assertion styles. Penalties prevent test_foo_1 through test_foo_20 all looking identical.
Refactoring / style migration (callbacks → async, class → functional)
{
"temperature": 0.1,
"top_p": 0.9,
"frequency_penalty": 0.2,
"presence_penalty": 0.1
}Low temperature preserves intent. Slight penalties prevent the model from copying the old pattern’s variable names into the new structure.
Prototyping / exploratory coding (REPL-style, one-offs)
{
"temperature": 0.5,
"top_p": 0.95,
"frequency_penalty": 0.2,
"presence_penalty": 0.1
}Human in the loop. You’ll discard 80% anyway; maximize useful diversity.
Constrained DSL (Terraform, K8s, SQL, GraphQL)
{
"temperature": 0.05,
"top_p": 0.7,
"frequency_penalty": 0.1,
"presence_penalty": 0.0
}Tight nucleus forces adherence to valid resource types and property names. Temperature near zero prevents creative YAML keys.
How to evaluate: don’t guess, measure
Sampling parameters are hyperparameters. Treat them like you’d treat learning rate — run a sweep, measure pass rates.
import json
from concurrent.futures import ThreadPoolExecutor
PARAMS_GRID = [
{"temperature": t, "top_p": p, "frequency_penalty": f}
for t in [0.0, 0.1, 0.2, 0.3, 0.4]
for p in [0.9, 0.95, 1.0]
for f in [0.0, 0.2, 0.4]
]
def evaluate_config(params, prompts, n_trials=5):
results = []
for prompt in prompts:
for _ in range(n_trials):
code = generate(prompt, **params)
passed = run_tests(code) # your test harness
results.append({"params": params, "passed": passed})
return results
# Run in parallel, aggregate pass@k
with ThreadPoolExecutor(max_workers=8) as ex:
futures = [ex.submit(evaluate_config, p, TEST_PROMPTS) for p in PARAMS_GRID]
all_results = [f.result() for f in futures]Metrics that matter for code:
- Pass@1: First-try compilation + test pass rate. This is your automation metric.
- Pass@5: Best of 5. This is your human-review metric.
- Syntax error rate: Separate from logic failures. High syntax errors = temperature too high or top-p too low.
- Hallucination rate: Imports that don’t exist, methods that don’t exist. Track separately — it’s the most costly failure mode in production.
Run this once per model upgrade. Providers change sampling behavior silently (I’ve seen top-p=0.95 behave differently between GPT-4o snapshots two weeks apart).
The n4n.ai gateway exposes these parameters directly
When you route through n4n.ai’s OpenAI-compatible endpoint, sampling parameters pass through to the underlying provider unchanged — whether that’s Anthropic, Google, Meta, or a hosted open model. The gateway adds per-token usage metering and automatic fallback when a provider degrades, but it doesn’t rewrite your sampling config. This means the grids above work identically across 240+ models without translation layers.
Decisive takeaway
Default to temperature=0.2, top_p=0.95, frequency_penalty=0.1. This configuration sits in the Pareto frontier for almost every code generation task: low enough entropy to stay syntactically valid, high enough to escape micro-loops, with a light penalty to break repetition in long outputs.
Only deviate when you have a measured reason:
- Determinism required → temperature 0.0, top_p 1.0, penalties 0
- Maximum diversity for human review → temperature 0.5, keep top_p 0.95
- Constrained DSL → drop top_p to 0.7–0.8, keep temperature near 0
- Long generations (>2k tokens) → raise frequency_penalty to 0.3–0.5
Stop copying the chat defaults (0.7/0.95/0.0). They’re optimized for prose, not Python. Your CI pipeline will thank you.