Running a clean prompt length llm benchmark is harder than it looks: tokenizers differ, provider APIs pad or truncate silently, and caching masks real latency. If you don’t isolate prompt size as the only variable, your numbers measure infrastructure luck, not model behavior.
Step 1: Define your tokenization baseline
You cannot control prompt length without a precise token counter. Model APIs accept strings, but they bill, schedule, and attend over tokens. Character length is meaningless across model families—the same sentence can be 12 tokens in one tokenizer and 19 in another.
Use the tokenizer that matches your target model. For OpenAI-compatible models, tiktoken is the reference implementation. For Anthropic or open-weight models, use their published tokenizers or the provider’s /tokenize endpoint.
import tiktoken
def count_tokens(text: str, model: str = "gpt-4o") -> int:
enc = tiktoken.encoding_for_model(model)
return len(enc.encode(text))
# Example: same text, different models
text = "Initialize the reactor coolant loop before sampling."
print(count_tokens(text, "gpt-4o")) # 11
print(count_tokens(text, "text-embedding-3-small")) # may differLog the exact token count per request from your local counter and the usage.prompt_tokens field returned by the API. If they diverge by more than 1 token per 1000, your independent variable is already drifting.
Step 2: Build fixed-length prompt templates
Generate prompts at exact token budgets. Truncate or pad with neutral filler that does not change the task semantics. Avoid repeating the exact same filler substring thousands of times—some providers apply compression or prefix caching on repetitive spans, which corrupts the measurement.
def make_prompt(target_tokens: int, base_task: str, model: str = "gpt-4o") -> str:
enc = tiktoken.encoding_for_model(model)
base_ids = enc.encode(base_task)
filler = enc.encode(" The quick brown fox jumps over the lazy dog.")
ids = base_ids[:]
while len(ids) < target_tokens:
# rotate filler to avoid trivial repetition patterns
ids.extend(filler)
ids = ids[:target_tokens]
return enc.decode(ids)
prompt_512 = make_prompt(512, "Summarize the following system log:")
assert count_tokens(prompt_512, "gpt-4o") == 512This yields a string that decodes to exactly target_tokens tokens. Always assert the count before sending. If you benchmark multiple models, regenerate per model tokenizer.
Step 3: Disable caching and control cache hints
Provider-side prompt caching can cut latency by 50–90% on repeated prefixes, which destroys your prompt length llm benchmark. Set cache_control to ephemeral or disable it per request. When using an OpenAI-compatible gateway, pass the relevant headers so the upstream provider sees your intent.
curl https://api.openai.com/v1/chat/completions \
-H "Authorization: Bearer $KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "..."}],
"cache_control": {"type": "ephemeral"}
}'If you route through a gateway that honors client routing directives and forwards provider cache-control hints, ensure your client sends the no-cache intent. n4n.ai forwards those hints without rewriting them, so the provider still sees your directive and will not serve a cached prefix.
Also disable client-side HTTP caching and reuse connections carefully—a warm TCP connection is fine, but a warm application cache is not.
Step 4: Instrument requests with precise timers
Measure time-to-first-token (TTFT) and total generation time separately. Use streaming to capture TTFT at the network boundary, not just when your loop processes the chunk.
import time, openai
client = openai.OpenAI()
def timed_completion(prompt: str, model: str):
start = time.perf_counter_ns()
first_token_ns = None
stream = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
stream=True,
cache_control={"type": "ephemeral"} # if supported
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
if first_token_ns is None:
first_token_ns = time.perf_counter_ns()
end_ns = time.perf_counter_ns()
ttft = (first_token_ns - start) / 1e9
total = (end_ns - start) / 1e9
return ttft, totalRun each length three times. Discard the first run as cold start (model worker spin-up, DNS, TLS). Record the median of the remaining two. Never average across different prompt lengths—keep the matrix explicit.
Step 5: Sweep lengths systematically
Pick a geometric progression that spans the model’s context window without hitting truncation: 128, 256, 512, 1024, 2048, 4096, 8192. For each, generate the fixed prompt, send, and store metrics with the exact token count.
import pandas as pd
def run_sweep(model: str, lengths=(128, 256, 512, 1024, 2048, 4096, 8192)):
rows = []
for n in lengths:
p = make_prompt(n, "Summarize the following system log:", model)
ttft, total = timed_completion(p, model)
rows.append({
"model": model,
"prompt_tokens": n,
"ttft_s": ttft,
"total_s": total,
"generated_tokens": 32 # fixed output length for fairness
})
return pd.DataFrame(rows)
df = run_sweep("gpt-4o")Fix the output token count (max_tokens) across all rows. A controlled prompt length llm benchmark varies only the input; generation cost should be constant. Plot TTFT vs prompt_tokens. For dense transformers, TTFT should rise roughly linearly with prefix size.
Step 6: Normalize across models and providers
Comparing a 70B open-weight model on one host to a frontier API requires identical input tokens and identical measurement methodology. Use one OpenAI-compatible endpoint that addresses 240+ models so your client code stays constant. A gateway such as n4n.ai exposes one OpenAI-compatible endpoint that addresses 240+ models with per-token usage metering, which simplifies comparing models without rewriting client code.
Per-token metering lets you confirm the billed tokens equal your counted tokens—a sanity check that your padding logic is correct. When a provider is degraded, automatic fallback preserves your sweep instead of dropping data points. Keep your script idempotent: if a request fails, retry with the same exact prompt bytes, not a regenerated one.
Step 7: Verify success
Your benchmark is controlled only if these conditions hold:
- API
usage.prompt_tokensequals your localcount_tokenswithin 0.1% for every row. - No response carries a cache hit flag (
usage.cache_read_input_tokensmust be 0 where the field exists). - TTFT at 128 tokens is strictly lower than at 8192 tokens in every model run.
- Coefficient of variation across repeats is under 10% after warmup discard.
Write a verification assert:
def verify(df: pd.DataFrame):
assert (df["prompt_tokens"] == df["api_prompt_tokens"]).all()
assert (df["cache_hit_tokens"] == 0).all()
for model, g in df.groupby("model"):
assert g["ttft_s"].is_monotonic_increasing
return TrueIf any assertion fails, inspect padding logic, cache headers, or SDK-injected system prompts. A controlled prompt length llm benchmark is reproducible: another engineer can run your script and get the same curve shape.
Gotchas that silently break control
Tokenizer drift
A model swap from gpt-4o to claude-3-5-sonnet changes token counts for the same text. Always re-count per model and store the encoder name alongside results.
Hidden system prompts
Some SDKs or gateways inject system messages. Subtract those tokens from your budget or include them in the fixed template so every request carries the same overhead.
Batch APIs
Batch endpoints report latency in minutes or hours, not milliseconds. Never mix batch and online paths in one prompt length llm benchmark.
Streaming buffer artifacts
If your client buffers stdout or uses a slow JSON parser, TTFT measurement includes client overhead. Use perf_counter_ns at the socket read level or a minimal streaming consumer.
Control the variable, measure everything else, and your latency curves will reflect the model—not the maze of infrastructure around it.