When an LLM agent silently returns a wrong tool call or loops on a retry, you need visibility into every step. To trace agent failures LangSmith gives you a structured span for each model invocation, tool execution, and chain hop, turning black-box debugging into a queryable timeline.
Prerequisites
- Python 3.10 or newer
- An OpenAI API key (or any OpenAI-compatible endpoint)
- A LangSmith account and API key
- Installed packages:
pip install langchain langchain-openai langgraph langsmith python-dotenvExport the required environment variables before running any code:
export OPENAI_API_KEY=sk-...
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=ls-...
export LANGCHAIN_PROJECT=agent-debugThe LANGCHAIN_TRACING_V2 flag patches LangChain and LangGraph so every run is uploaded. If you route through an OpenAI-compatible gateway like n4n.ai, the same env vars work; the tracing client captures spans regardless of underlying provider, because it operates at the LangChain callback layer.
Build a minimal agent that fails
We’ll use LangGraph’s prebuilt ReAct agent. It loops between a chat model and a tool until the model emits a final answer. To simulate a real outage, we define a tool that raises on a specific input.
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from langchain_core.tools import tool
@tool
def get_weather(city: str) -> str:
"""Return current weather for a city."""
if city.lower() == "paris":
raise ValueError("Upstream weather API timeout")
return f"Sunny in {city}"
model = ChatOpenAI(model="gpt-4o-mini")
agent = create_react_agent(model, [get_weather])Run it against the failing case:
try:
result = agent.invoke({"messages": [("user", "What's the weather in Paris?")]})
except Exception as e:
print("Agent raised:", type(e).__name__, e)Expected output (abbreviated):
Agent raised: ValueError Upstream weather API timeoutBecause tracing is enabled, this invocation is already recorded in the agent-debug project. The top-level exception is visible, but the useful detail is one layer down: which span actually threw.
Inspect the trace programmatically
The LangSmith UI is good for ad-hoc investigation, but in a debugging script you want the spans in code. The client lets you list runs by project and parent.
from langsmith import Client
client = Client()
parent = client.list_runs(project_name="agent-debug", run_type="chain", limit=1)[0]
print("Parent run:", parent.id, parent.name)
children = client.list_runs(parent_run_id=parent.id)
for c in children:
print(f"{c.run_type:10} {c.name:20} error={c.error}")Expected output:
Parent run: 0a1b2c3d-... AgentExecutor
llm ChatOpenAI error=None
tool get_weather error=ValueError('Upstream weather API timeout')This is the core workflow to trace agent failures LangSmith: the error span points at the tool, not the model. Without tracing you’d only see the propagated exception and might wrongly assume the model hallucinated.
You can also dump the full span as JSON to see inputs and outputs:
import json
tool_run = [c for c in children if c.name == "get_weather"][0]
print(json.dumps({
"inputs": tool_run.inputs,
"outputs": tool_run.outputs,
"error": str(tool_run.error)
}, indent=2))This prints the exact arguments the model passed ({"city": "Paris"}) and the raised error, which is everything you need to reproduce the bug in isolation.
Add a fallback and observe the new trace
A production agent should degrade instead of crashing. We wrap the risky call and return a safe string.
@tool
def get_weather_safe(city: str) -> str:
"""Return weather, never raises."""
try:
if city.lower() == "paris":
raise ValueError("Upstream weather API timeout")
return f"Sunny in {city}"
except ValueError:
return "Weather unavailable, please try later"
agent2 = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), [get_weather_safe])
result = agent2.invoke({"messages": [("user", "What's the weather in Paris?")]})
print(result["messages"][-1].content)Expected output:
Weather unavailable, please try laterRe-running the inspection code now shows error=None on the tool span and the fallback string in outputs. You can open both runs in LangSmith and diff them; the model call is identical, only the tool result changed.
Assert on traces in CI
Local debugging is necessary but not sufficient. Regressions appear when someone “fixes” the tool and drops the try/except. A pytest test that fails on any error span closes the loop.
import pytest
from langsmith import Client
from langgraph.prebuilt import create_react_agent
from langchain_openai import ChatOpenAI
from your_module import get_weather_safe
@pytest.fixture
def client():
return Client()
def test_agent_no_tool_errors(client):
agent = create_react_agent(ChatOpenAI(model="gpt-4o-mini"), [get_weather_safe])
agent.invoke({"messages": [("user", "Weather in Paris?")]})
parent = client.list_runs(project_name="agent-debug", run_type="chain", limit=1)[0]
err_runs = client.list_runs(parent_run_id=parent.id, error=True)
assert len(err_runs) == 0, f"Found error spans: {[r.name for r in err_runs]}"Run it:
pytest test_agent.py -qIf a future change makes the tool raise, the test fails with the span name. This is how you trace agent failures LangSmith continuously rather than only after a user complaint.
Async agents and streaming
The same pattern works for async invocation. Use ainvoke and the async client methods:
import asyncio
from langsmith import Client
async def main():
res = await agent2.ainvoke({"messages": [("user", "Weather in London?")]})
client = Client()
parent = (await client.alist_runs(project_name="agent-debug", run_type="chain", limit=1))[0]
children = await client.alist_runs(parent_run_id=parent.id)
for c in children:
print(c.name, c.error)
asyncio.run(main())Streaming does not change span boundaries; each token is attached to the parent LLM span as events, so tool errors remain distinct child spans.
Tag runs with metadata for filtering
When several agents share a project, filter traces by metadata. Pass a config dict at invocation:
agent2.invoke(
{"messages": [("user", "Weather in Paris?")]},
config={"metadata": {"agent": "weather-v1", "env": "ci"}}
)In LangSmith you can query metadata.env = "ci" to separate test runs from production traffic. This keeps your error assertions focused on the build under test.
Takeaways
Tracing is not optional once agents call real tools. The pattern above is reproducible: enable env vars, build a failing case, pull spans via the client, fix the tool, and assert no error spans in CI. Using this approach you can trace agent failures LangSmith across any model provider, catch regressions before release, and turn intermittent agent bugs into queryable postmortems.