Benchmarking multi-file code generation latency at scale

Multi-file code generation latency is the metric that separates a demo from a deployable agentic coding tool. Most published LLM speed tests measure a single completion from a fixed prompt, but a real refactor touches a dependency graph of files where each write depends on the shape of the others. Optimize the wrong layer and you ship something that looks fast in a notebook and collapses under a monorepo.

Why single-file benchmarks lie

A single-file benchmark reports time-to-first-token and total generation time for one isolated prompt. That number is useless when your agent must edit auth.py, then auth_test.py, then middleware.py that imports both. The calls are not independent. The second file waits for the first to define a function signature; the third waits for the second to stabilize its behavior.

Worse, the model context for each file includes the others. You pay a hidden tax of reading, diffing, and packing neighboring files into the prompt. That tax is CPU and I/O bound, but it sits directly on the critical path.

Single-file numbers also ignore queueing. A provider that handles one request in 800 ms may take 8 seconds when your orchestrator fires ten parallel calls and hits a concurrency limit. The latency you ship is the latency of the system, not the model card.

Modeling the orchestration graph

Treat the task as a directed acyclic graph (DAG). Nodes are generation jobs. Edges are data dependencies: “file B needs the exported interface from file A.” The wall-clock cost is the length of the longest path through that DAG, not the sum of all node times.

# Critical path estimate from a simple dependency map
def critical_path(graph):
    # graph: dict[node] -> (cost, [deps])
    memo = {}
    def visit(n):
        if n in memo: return memo[n]
        cost, deps = graph[n]
        if not deps:
            memo[n] = cost
        else:
            memo[n] = cost + max(visit(d) for d in deps)
        return memo[n]
    return max(visit(n) for n in graph)

A concrete refactor graph

Suppose you migrate a Flask service to FastAPI. Files: models.py (independent), schemas.py (depends on models), routes.py (depends on schemas), tests.py (depends on routes), config.py (independent). The critical path is models → schemas → routes → tests. config.py can generate in parallel but does nothing to shorten the path that gates completion.

This is why throwing more workers at the problem rarely helps multi-file code generation latency. You parallelize the leaves, not the spine.

Dependency classes

Hard dependencies are non-negotiable. If user.ts imports types.ts, the type file must exist with correct symbols before the user file generates.

Soft dependencies are stylistic or architectural consistency. You can generate files in parallel and reconcile later with a lint pass, but that trades latency for risk of mismatch and a likely second generation round-trip.

Context assembly cost

Each node requires a prompt built from the file’s neighbors, retrieval results, and prior diffs. Transformer attention scales roughly quadratically with sequence length, so stuffing 20 files of context into every call balloons generation time even if the edit is one line. This cost is repeated at every node unless you cache the shared prefix.

Measurement methodology that survives production

You cannot manage what you do not trace. Wrap every orchestration step in a span. Capture not just the model call but the schedule decision, the context fetch, and the write verification.

from opentelemetry import trace
tracer = trace.get_tracer("codegen")

def generate_file(path, dep_paths):
    with tracer.start_as_current_span("generate_file") as span:
        span.set_attribute("file.path", path)
        with tracer.start_as_current_span("context_assembly"):
            ctx = load_and_pack(dep_paths)
        with tracer.start_as_current_span("model_call") as mspan:
            mspan.set_attribute("model", "gpt-4o-mini")
            resp = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": ctx}]
            )
        with tracer.start_as_current_span("write_verify"):
            apply_diff(path, resp.choices[0].message.content)

Export these spans as JSON to your observability backend. A representative slice:

{
  "span": "model_call",
  "file.path": "src/auth.py",
  "ttft_ms": 420,
  "total_ms": 1800,
  "model": "gpt-4o-mini"
}

Aggregate p95 critical path per repository size bucket. That distribution, not the average single call, is your real multi-file code generation latency. Break the span into ttft and decode so you can see whether you are waiting on the provider or on your own context builder.

Provider variability and fallback strategies

Model endpoints throttle. A provider that serves 50 ms at p50 can queue your request for 30 seconds during a spike. If your orchestrator blindly retries against the same endpoint, the critical path stretches.

An inference gateway such as n4n.ai that exposes one OpenAI-compatible endpoint across 240+ models and applies automatic fallback when a provider is degraded can mask isolated outages. Its honoring of client routing directives and provider cache-control hints lets you pin a stable model and reuse cached prefixes—directly cutting multi-file code generation latency when many files share a common base context. The fallback still adds retry overhead, so treat it as a safety net, not a latency optimizer.

Per-token metering is useful here only indirectly: it tells you which model class caused a cost spike that often correlates with a slow fallback path. Use it to tune routing rules, not to measure speed.

Tradeoffs: parallelism vs. coherence

The obvious lever is parallel generation. Spawn an agent per file, join at the end.

async function generateAll(files: string[]) {
  // bounded concurrency to avoid rate limits
  const pool = new ConcurrencyPool(4);
  await Promise.all(files.map(f => pool.run(() => generate(f))));
}

This shrinks wall clock when dependencies are soft. But hard dependencies force serialization. Over-parallelizing triggers rate limits and cache misses, increasing tail latency. The right move is to compute the DAG depth and parallelize only across independent layers.

Speculative generation

You can guess an interface and generate dependents before the dependency finishes, then patch mismatches. This trades extra token spend and a possible second pass for lower critical-path length. In our Flask example, speculatively generating tests.py against an assumed routes.py signature can save a round-trip, but a wrong guess costs a full regenerate. For tight latency budgets on well-understood code, it pays off. For novel architectures, it burns time.

Where to actually spend engineering time

Caching shared context is the highest-leverage fix. If ten files all import the same core/types.ts, pack that once, cache the prefix with provider cache-control, and reference it. That turns quadratic attention cost into linear for the repeated portion.

Second, instrument the critical path. You will often find that context assembly exceeds model time for small edits. Move file reads off the request thread; prefetch neighbors while the model streams.

Third, bound concurrency to the provider’s real limit. A ConcurrencyPool sized from observed 429 responses beats a fixed guess. Pair it with fallback so a single provider’s degradation doesn’t stall the DAG.

Decisive takeaway

Stop quoting single-file tokens-per-second. Measure multi-file code generation latency as the p95 length of your orchestration DAG’s critical path, trace every span, cache shared prefixes, and parallelize only across dependency layers. Do that and agentic coding tools stay responsive at monorepo scale; ignore it and you ship a benchmark winner that times out in CI.