Correlating request IDs across LLM retries is the difference between a debuggable production incident and a blind guess. When your client retries a timed-out completion or a gateway silently falls back to a secondary provider, you need one identifier that survives every attempt. This guide shows how to build that correlation into your logging from the edge of your service down to the model response.
Step 1: Generate a stable correlation ID at the entrypoint
Do not wait until you call the model to create a trace ID. Generate it at the HTTP boundary so every downstream system can inherit it. If a client already sends X-Correlation-ID, reuse it; otherwise mint a uuid4.
import uuid
from fastapi import Request
@app.middleware("http")
async def correlation_middleware(request: Request, call_next):
correlation_id = request.headers.get("X-Correlation-ID") or str(uuid.uuid4())
request.state.correlation_id = correlation_id
response = await call_next(request)
response.headers["X-Correlation-ID"] = correlation_id
return responseThis middleware stamps every response with the same ID and stores it on request.state for handlers to pass along. Use a UUIDv4 unless you have a centralized ID generator; randomness avoids collision without coordination.
Step 2: Propagate the ID through your LLM client wrapper
When correlating request IDs across LLM retries, you must ensure the identifier is attached to every outbound call, not just the first. Wrap the OpenAI-compatible client so the header is injected on each attempt and the ID is threaded into logs.
import time, json, logging
import openai
logger = logging.getLogger("llm")
class RetryLLMClient:
def __init__(self, base_url: str, api_key: str, correlation_id: str):
self.client = openai.OpenAI(base_url=base_url, api_key=api_key)
self.correlation_id = correlation_id
def complete(self, model: str, messages: list, max_retries: int = 3):
for attempt in range(max_retries):
start = time.monotonic()
try:
resp = self.client.chat.completions.create(
model=model,
messages=messages,
extra_headers={"X-Correlation-ID": self.correlation_id},
)
logger.info(json.dumps({
"correlation_id": self.correlation_id,
"event": "llm_success",
"attempt": attempt,
"requested_model": model,
"served_model": resp.model,
"provider_request_id": resp.id,
"latency_ms": int((time.monotonic() - start) * 1000),
}))
return resp
except openai.APIError as e:
provider_id = None
if hasattr(e, "response") and e.response is not None:
provider_id = e.response.headers.get("x-request-id")
logger.warning(json.dumps({
"correlation_id": self.correlation_id,
"event": "llm_error",
"attempt": attempt,
"requested_model": model,
"provider_request_id": provider_id,
"error": str(e),
"latency_ms": int((time.monotonic() - start) * 1000),
}))
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt)The extra_headers parameter is honored by any OpenAI-compatible endpoint. If you later switch base URLs, the correlation header travels with the request.
Production code should replace the fixed time.sleep(2 ** attempt) with jittered backoff and a circuit breaker. The tenacity library handles both cleanly. Retries are not free: each attempt consumes tokens and may double-bill if you do not check idempotency. For streaming calls, never retry mid-stream; abort and start a fresh request with the same correlation ID so the log chain stays intact.
Capture provider-assigned IDs
On success, resp.id is the provider’s request identifier (e.g., chatcmpl-...). On failure, inspect e.response.headers for x-request-id or similar before retrying. Add that to the error log so you can cite it in a provider support ticket.
Step 3: Emit structured logs with attempt metadata
String-formatted log lines are useless when you need to query across hundreds of retries. Emit JSON with a fixed schema. At minimum include: correlation_id, event, attempt, requested_model, served_model, provider_request_id, latency_ms, error (when present).
{
"correlation_id": "b3b1f2c4-7a3e-4d9b-8c1a-2f9e0d4b6a1c",
"event": "llm_success",
"attempt": 1,
"requested_model": "gpt-4o-mini",
"served_model": "gpt-4o-mini",
"provider_request_id": "chatcmpl-abc123",
"latency_ms": 812
}Pipe these to stdout and let your collector (Vector, Fluent Bit, or OTel collector) ship them. Do not pretty-print in production. Index correlation_id as a keyword field in your log backend. If you use Datadog or Elasticsearch, set the mapping explicitly; a hashed string that gets analyzed as text will break exact-match queries. The same goes for provider_request_id.
Step 4: Implement explicit client-side fallback
Sometimes you want to fall back to a different model when the primary returns a specific error (e.g., context length exceeded). Treat fallback as another attempt and log the switch explicitly.
FALLBACK_MODEL = "gpt-4o"
def complete_with_fallback(self, model, messages):
try:
return self.complete(model, messages)
except openai.BadRequestError as e:
if "maximum context length" in str(e):
logger.warning(json.dumps({
"correlation_id": self.correlation_id,
"event": "fallback_triggered",
"from_model": model,
"to_model": FALLBACK_MODEL,
}))
return self.complete(FALLBACK_MODEL, messages)
raiseThis keeps the same correlation_id while recording that the model changed. Your future self will thank you when a prompt unexpectedly blows up on one model but not another.
Step 5: Handle gateway-level fallback
If you route through a gateway such as n4n.ai, automatic fallback when a provider is rate-limited or degraded happens upstream of your code. Your X-Correlation-ID still passes through, but the served_model in the response may differ from the model you requested. Log that discrepancy so you can distinguish gateway fallback from your own client-side fallback.
resp = client.chat.completions.create(
model="anthropic/claude-3-haiku",
messages=messages,
extra_headers={"X-Correlation-ID": correlation_id},
)
if resp.model != "anthropic/claude-3-haiku":
logger.info(json.dumps({
"correlation_id": correlation_id,
"event": "gateway_fallback",
"requested": "anthropic/claude-3-haiku",
"served": resp.model,
}))Do not assume the gateway will tell you it fell back via a header. Some return the served model only in the response body. Always diff requested vs served rather than trusting a flag. Correlating request IDs across LLM retries and fallbacks becomes trivial when the gateway returns the served model and you already log it on every attempt.
Step 6: Verify end-to-end with a test harness
You cannot claim correlation works until you force a failure and watch the logs. Write a pytest that points the client at a bad API key or a model that does not exist, then asserts that all log records share the same correlation_id.
import json, logging
from unittest.mock import patch, MagicMock
import pytest
import openai
def mock_response():
class R:
headers = {"x-request-id": "req-err-1"}
status_code = 429
return R()
def test_correlation_id_preserved(caplog):
with patch("openai.OpenAI") as mock_client:
mock_client.return_value.chat.completions.create.side_effect = [
openai.RateLimitError("rate", response=mock_response(), body=None),
MagicMock(id="chatcmpl-2", model="gpt-4o-mini"),
]
client = RetryLLMClient("https://api.example.com/v1", "bad", "test-id-123")
client.complete("gpt-4o-mini", [{"role": "user", "content": "hi"}])
ids = {json.loads(r.message)["correlation_id"] for r in caplog.records}
assert ids == {"test-id-123"}The caplog fixture captures records at the root logger; ensure your logger propagates. Run it with pytest -s and tail your local JSON log file. Search for the correlation ID:
grep "test-id-123" app.log | jq '.event'You should see llm_error followed by llm_success (or fallback_triggered if you forced that path). If every line for that ID appears under one grep, your correlation is solid.
Verification checklist
- Every outbound LLM request includes
X-Correlation-ID. - Each retry and fallback emits a JSON log with the same
correlation_id. - Provider request IDs are captured on both success and error.
- Gateway-level model switches are logged as
gateway_fallback. - A single
grepon the ID reconstructs the full attempt chain.
Correlating request IDs across LLM retries is not optional once you run more than one model in production. Build it at the edge, propagate it relentlessly, and log it as structured data. The next time a provider melts down, you will know exactly which user request suffered and how many attempts it took to recover.