How autonomous coding agents handle test failures

Most autonomous coding tools treat a failing test as a prompt appendix: copy the stack trace, ask the model to fix it, repeat. That approach collapses on anything larger than a toy repo because coding agents test failures are not a single signal—they are a heterogeneous stream of compile errors, assertion mismatches, environment quirks, and flaky runs. The agents that survive contact with real codebases classify failures first, isolate them, and only then attempt a targeted patch.

The naive loop: grep, patch, rerun

A first-generation agent runs the test command, captures stdout/stderr, and forwards the raw text to the model with a directive like “fix the code.” The skeleton looks like this:

def naive_fix_cycle(repo_path: str, model_client):
    for _ in range(5):
        result = run_subprocess("pytest", cwd=repo_path)
        if result.returncode == 0:
            return "green"
        prompt = f"Tests failed:\n{result.stderr}\nFix the code."
        patch = model_client.complete(prompt)
        apply_patch(patch, repo_path)
    return "gave_up"

This works for a missing import. It falls apart when the failure is a timeout in a CI container or a mismatched fixture path. The model sees the symptom, not the cause, and the loop consumes context on every iteration. At 2,000 tokens of traceback per run and five attempts, you have burned 10k tokens just to discover the test database wasn’t migrated.

Why blind retries burn tokens and context

The core problem is information density. A single pytest run can emit thousands of lines across parallel workers. Stuffing that into a context window wastes tokens and obscures the relevant frame. Worse, coding agents test failures often include noise: deprecation warnings, unrelated collection errors, and flaky network calls.

Consider a typical captured block:

tests/test_api.py:42: AssertionError: assert 503 == 200
<... 80 lines of requests internals ...>
WARNING:root:retry attempt 3

If the agent re-injects the full blob each turn, it trains the model to pattern-match on line numbers rather than understand the contract. The loop also has no termination condition beyond a fixed retry count, so it may overwrite a working file with a hallucinated import to silence a lint error. The red/green signal is binary, but the underlying faults are not.

Failure taxonomy: turning red into structured input

Robust agents convert raw test output into a typed object. At minimum, separate build failures from test assertions, and deterministic from non-deterministic ones. A useful schema:

{
  "failure_type": "assertion | compile | runtime | flaky | env",
  "test_id": "tests/test_api.py::test_status",
  "deterministic": true,
  "trace_excerpt": "assert 503 == 200",
  "suggested_scope": ["src/api/client.py"]
}

With this, the agent routes the fix attempt. A compile failure triggers a syntax-only edit pass; a flaky failure triggers a rerun with --count=3 before any code change. This reduces coding agents test failures from a monolith to a queue of work items.

Claude Code does something adjacent by running tests in a sandboxed shell and using file-scoped edits; it does not expose a formal taxonomy but relies on the model to infer it. Devin maintains an internal task list that implicitly categorizes blockers. Cursor, being inline, pushes the taxonomy burden onto the developer via the diff view.

How Claude Code, Devin, and Cursor differ

Claude Code

Claude Code operates as a terminal agent with direct filesystem access. When a test fails, it reads the truncated output, edits the suspected file, and reruns only the failing node:

pytest tests/test_api.py::test_status -q

Its strength is tight feedback latency; its weakness is limited cross-file reasoning when the failure stems from an interface change in another module. It assumes the local environment matches CI.

Devin

Devin runs remotely with a longer horizon. It will open a GitHub issue, write a reproduction script, and iterate over hours. For coding agents test failures that require schema migrations or multi-service mocks, Devin’s persistence wins. But its autonomy means a wrong fix can be committed to a branch before a human notices.

Cursor

Cursor embeds suggestions in the editor. It does not autonomously run the whole suite; instead, it reacts to the test runner panel. This keeps the developer in the loop—good for fragile legacy code, bad for throughput. The agent never sees the full failure queue, only the current cursor context.

The pattern is clear: autonomy scales with the agent’s ability to classify and isolate failures, not with raw model size.

A reference implementation

Below is a minimal but disciplined loop that classifies, isolates, and patches. It uses a generic model client interface you can swap for any OpenAI-compatible endpoint.

import json
import subprocess
from pathlib import Path

def run_test(test_id: str | None) -> subprocess.CompletedProcess:
    cmd = ["pytest", "-q"]
    if test_id:
        cmd.append(test_id)
    return subprocess.run(cmd, capture_output=True, text=True)

def classify(output: str) -> dict:
    if "could not compile" in output:
        return {"failure_type": "compile", "deterministic": True}
    if "AssertionError" in output:
        return {"failure_type": "assertion", "deterministic": True}
    if "warning" in output.lower() or "flake8" in output:
        return {"failure_type": "env", "deterministic": False}
    return {"failure_type": "runtime", "deterministic": True}

def extract_failing_test(output: str) -> str | None:
    for line in output.splitlines():
        if "::" in line and ".py" in line:
            return line.strip().split()[0]
    return None

def fix_cycle(repo: Path, model_client, max_attempts=3):
    test_id = None
    for attempt in range(max_attempts):
        res = run_test(test_id)
        if res.returncode == 0:
            return "green"
        meta = classify(res.stderr)
        if meta["failure_type"] == "env" and not meta["deterministic"]:
            if run_test(test_id).returncode == 0:
                return "green_flaky"
        prompt = {
            "task": "fix",
            "failure": meta,
            "excerpt": res.stderr[:500],
        }
        patch = model_client.complete(json.dumps(prompt))
        apply_patch(patch, repo)
        test_id = extract_failing_test(res.stderr)
    return "escalate_to_human"

The key differences from the naive version: output is classified, env/flaky failures are confirmed before patching, and the prompt carries a structured excerpt rather than 2,000 lines. When this loop calls the model through a gateway such as n4n.ai, automatic fallback to a secondary provider keeps the cycle alive if the primary model hits a rate limit—a real risk when running hundreds of fix attempts in a night.

Model routing and resilience

Autonomous fix loops issue many small LLM calls. If your primary model 429s, the agent stalls. A gateway that honors client routing directives and forwards cache-control hints lets you pin cheap models for classification and reserve frontier models for the actual patch. That split cuts cost without sacrificing quality on coding agents test failures that are mostly mechanical. Per-token metering lets you cap spend per pull request.

Tradeoffs: autonomy vs control

Full autonomy is tempting but dangerous. An agent that can git push after a green run may have masked a failure by deleting the test. Keep a human gate for merges. On the other hand, human-in-the-loop on every failure kills the velocity that justified the agent.

What a flaky failure actually costs

Auto-skipping a flaky test makes the suite green but hides rot. A better policy: quarantine the test, file a ticket, and continue. Devin does this implicitly; you should make it explicit. The cost of a quarantined test is a known unknown; the cost of a deleted assertion is silent production breakage.

Cost is another axis. Classification calls are cheap; patch calls are expensive. If you run the loop on every commit, meter per-token usage and set a budget per PR. Otherwise a single circular import can drain your monthly limit.

Takeaway

Treat test failures as structured, typed events, not raw text. Classify, isolate, and confirm flakiness before editing. Use a resilient model route so the loop never dies on a 429. Keep a human approval step for anything that touches main. Agents that follow this discipline turn coding agents test failures from a token firehose into a manageable queue—and that is the difference between a demo and a deployable tool.