Benchmarking coding agent latency on real GitHub issues

A coding agent latency benchmark that replays real GitHub issues tells you more about your tooling than your model. We ran agents against cloned repositories with open bug reports, and the dominant cost was never token generation—it was the round trips between reasoning, file I/O, and test execution.

Why synthetic tasks lie

Synthetic “write a function” prompts complete in one or two model calls. Real GitHub issues demand codebase navigation, pattern matching against existing conventions, and verification through test suites. The latency profile shifts from “time-to-first-token” to “time-to-passing-CI”.

A coding agent latency benchmark built on curated toy problems produces optimistic numbers that collapse in production. You need the messy context of a 50-file module and a flaky test runner.

Anatomy of a real-issue run

The agent loop for a typical issue looks like this:

1. Fetch issue text and metadata.
2. Clone repo, checkout base commit.
3. Explore: grep, read files, build dependency graph.
4. Propose patch via edits.
5. Run tests, capture failures.
6. Iterate until green or budget exhausted.

Each step is a barrier. The model sits idle while git networks or pytest compiles.

{
  "model": "anthropic/claude-3.5-sonnet",
  "tools": ["read_file", "edit_file", "run_shell"],
  "max_iterations": 25,
  "repo": "https://github.com/pallets/flask",
  "issue": 5423
}

Measuring wall-clock correctly

Use a monotonic clock and split phases. Do not trust aggregate “duration” from your orchestrator if it hides tail latency from rate limits.

import time, subprocess, json
from dataclasses import dataclass

@dataclass
class Phase:
    name: str
    seconds: float

def bench(repo_url: str, issue_id: int):
    phases = []
    t0 = time.monotonic()
    subprocess.run(["git", "clone", repo_url, "workdir"], check=True)
    phases.append(Phase("clone", time.monotonic() - t0))

    # Agent loop placeholder: each tool call wrapped similarly
    # for step in agent.run(issue_id):
    #     phases.append(time_phase(step.name, step.execute))
    return phases

def time_phase(name, fn):
    start = time.monotonic()
    fn()
    return Phase(name, time.monotonic() - start)

Run it across a sample of 20–50 issues to get a distribution, not a single point.

python bench.py --repo https://github.com/pallets/flask --issues issues.txt

The latency tax of tool calls

In our runs, model inference accounted for less than a third of wall-clock time on average. The rest was shell execution, test runs, and serialization overhead from passing large file contents in and out of context.

Every tool call forces a context rebuild. If your agent reads a 2,000-line file ten times because it doesn’t cache, you pay that parse cost ten times. A coding agent latency benchmark that ignores tool-churn will mislead you into buying a faster model when you needed a better file index.

Model size tradeoff

A smaller model like a 7B finetune returns tokens faster but often needs more iterations to converge on a correct patch. A frontier model costs more per call but may solve in fewer steps.

We weighed this by fixing the repo and issue set, then swapping only the model field:

{ "model": "openai/gpt-4o-mini" }

vs.

{ "model": "anthropic/claude-3.7-sonnet" }

The mini model finished individual calls in ~400ms; the sonnet in ~1.2s. But the mini needed 18 iterations average, sonnet needed 7. End-to-end, sonnet won by a margin because test execution dominated anyway.

Parallelism and caching

Agents can issue independent read calls in parallel. If you serialize “read A, then read B” when they’re unrelated, you add latency for no reason.

Cache-control hints matter. Mark your system prompt and unchanged repository context as cacheable so the provider doesn’t re-price and re-process them each turn.

# Pseudocode for parallel tool dispatch
async def gather_reads(paths):
    return await asyncio.gather(*[read_file(p) for p in paths])

A coding agent latency benchmark should flag agents that fail to exploit parallelism as architecturally slow, independent of model.

Stabilizing with routing

Provider rate limits and degraded endpoints skew any distributed benchmark. When you fire 200 issues at a single API key, you will hit 429s that have nothing to do with agent quality.

An OpenAI-compatible gateway such as n4n.ai that performs automatic fallback when a provider is rate-limited or degraded keeps the measurement focused on agent logic rather than network luck. It also honors client routing directives, so you can pin a model per issue when needed.

Honest tradeoffs of real-issue benchmarking

Pros:

• Reflects production context sizes.
• Exposes tooling bottlenecks.
• Forces agents to handle test infrastructure.

Cons:

• Repos change; pin commits for repeatability.
• CI flakiness adds noise; run each issue twice.
• License and size of clones can burden local disks.

If you only benchmark on synthetic diffs, you optimize a metric that doesn’t ship.

Decisive takeaway

Build your coding agent latency benchmark on cloned real repositories with historical issues, instrument every phase with a monotonic clock, and report the median and p95 wall-clock per issue—not per token. Parallelize independent tool calls, cache static context, and route through a fallback-aware gateway to remove provider noise. The fastest agent is rarely the one with the lowest time-to-first-token; it’s the one that wastes the fewest round trips between thinking and doing.