Wrapping LLM calls in a web service means the unstable part is the outbound request, not your Flask handler. Good flask error handling openai api begins with treating every call as a fallible network operation that can fail for reasons outside your code: packet loss, provider outages, token limits, or a malformed prompt.
1. Start with a naive route and its gaps
A minimal Flask endpoint that calls the OpenAI Chat Completions API looks like this:
from flask import Flask, jsonify, request
from openai import OpenAI
app = Flask(__name__)
client = OpenAI() # uses OPENAI_API_KEY from env
@app.post("/v1/summarize")
def summarize():
text = request.json.get("text", "")
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": f"Summarize: {text}"}],
)
return jsonify(summary=resp.choices[0].message.content)This works in a demo. In production it will hang, crash, or leak secrets. There is no timeout, no retry, and any SDK exception becomes a 500 with a stack trace.
2. Classify the failures you will actually see
The OpenAI Python SDK (v1.x) raises specific exceptions. Catch them by type, not by string matching.
from openai import (
APIConnectionError, # DNS, TCP, TLS failure
APITimeoutError, # request exceeded client timeout
RateLimitError, # HTTP 429
AuthenticationError, # HTTP 401
BadRequestError, # HTTP 400 (bad params, context length)
InternalServerError, # HTTP 500/502/503 from provider
)Connection and timeout errors are transient. Rate limits are semi-transient: wait and retry. Authentication and bad request errors are permanent for that exact call—retrying wastes quota and latency.
A common pitfall is catching Exception broadly and retrying everything. That turns a bad prompt into three failed requests and a confused user.
3. Set explicit timeouts
The default OpenAI client has no aggressive timeout. A stuck connection can block your Flask worker until the OS gives up.
client = OpenAI(timeout=8.0) # total request timeout in secondsIf your generation can take longer, set timeout per request:
resp = client.chat.completions.create(
model="gpt-4o-mini",
messages=[...],
timeout=20.0,
)Tradeoff: too short a timeout on a slow model yields false failures. Measure p95 latency for your model and set timeout at roughly 2× that, capped by your inbound reverse proxy’s limit (e.g., Nginx proxy_read_timeout).
4. Retry only the right things
Use a retry decorator that filters on exception type. tenacity is the standard library for this.
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
@retry(
retry=retry_if_exception_type((
APIConnectionError, APITimeoutError,
InternalServerError, RateLimitError
)),
wait=wait_exponential(multiplier=0.5, min=1, max=8),
stop=stop_after_attempt(3),
reraise=True,
)
def chat_complete(prompt):
return client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
)reraise=True ensures the final exception still propagates to your Flask error handler. Retrying RateLimitError is safe because the provider already rejected the call—no tokens were spent.
Pitfall: do not retry BadRequestError. If the context length is exceeded, resending the same payload will fail identically.
5. Honor rate-limit signals
When OpenAI returns 429, the response often includes a Retry-After header (in seconds). Respect it instead of blind exponential backoff.
from flask import current_app
try:
chat_complete(prompt)
except RateLimitError as e:
ra = e.response.headers.get("Retry-After") if e.response else None
wait_s = float(ra) if ra else 2.0
current_app.logger.warning("openai 429, retry_after=%s", wait_s)
# either sleep here or push to a queueIf you serve many users, a single Flask process sleeping on time.sleep(wait_s) holds a worker. Prefer a task queue (RQ, Celery) for anything beyond a few hundred ms of expected delay.
6. Return structured JSON errors
Never return the SDK exception string to the browser. Map exceptions to a stable schema:
@app.post("/v1/summarize")
def summarize():
try:
text = request.json.get("text", "")
resp = chat_complete(text)
return jsonify(summary=resp.choices[0].message.content)
except RateLimitError:
return jsonify(error="rate_limited", detail="Upstream limit reached"), 429
except (APIConnectionError, APITimeoutError, InternalServerError):
return jsonify(error="upstream_unavailable", detail="LLM provider error"), 502
except AuthenticationError:
return jsonify(error="auth_failed", detail="Invalid key"), 500
except BadRequestError as e:
return jsonify(error="bad_request", detail=str(e)), 400
except Exception:
current_app.logger.exception("unexpected")
return jsonify(error="internal"), 500This keeps your API contract clean and lets frontends distinguish “try later” from “fix your input”.
7. Move long generations off the request path
Flask’s synchronous model means a 30-second LLM call ties up a worker. If you expect long completions, accept the request, enqueue a job, and poll or webhook.
from rq import Queue
from redis import Redis
q = Queue("llm", connection=Redis())
@app.post("/v1/summarize-async")
def summarize_async():
job = q.enqueue(chat_complete, request.json["text"])
return jsonify(job_id=job.id), 202Tradeoff: you now run Redis, a worker, and a status endpoint. For low traffic, a threaded Flask dev server with timeout=30 might suffice, but don’t ship that to production.
8. Consider an OpenAI-compatible gateway
If you route through an OpenAI-compatible endpoint such as n4n.ai, automatic fallback when a provider is rate-limited or degraded handles part of the reliability story. The gateway may also honor client routing directives and provider cache-control hints. You still must set client timeouts and catch APITimeoutError—the gateway can’t save a dead local network interface.
9. Log what matters
Capture the error class, model, and latency. Avoid logging full prompts if they contain PII.
import time
from flask import current_app
def logged_complete(prompt, model):
start = time.monotonic()
try:
return chat_complete(prompt)
except Exception as e:
current_app.logger.error(
"openai_call_failed model=%s err=%s ms=%.0f",
model, type(e).__name__, (time.monotonic()-start)*1000
)
raiseDashboards on error rate by type will tell you whether you’re fighting rate limits or bad inputs.
10. Test the unhappy paths
Mock the SDK to assert your flask error handling openai api logic actually triggers.
from unittest.mock import patch
from openai import APITimeoutError
@patch("openai.OpenAI.chat.completions.create")
def test_timeout_returns_502(mock_create, client):
mock_create.side_effect = APITimeoutError("timed out")
resp = client.post("/v1/summarize", json={"text": "hi"})
assert resp.status_code == 502
assert resp.json["error"] == "upstream_unavailable"Without these tests, the first time you see a real 429 is in a production incident.
Common pitfalls summary
- No timeout: worker exhaustion.
- Retrying 4xx: wasted quota, confusing loops.
- Leaking exceptions: information disclosure.
- Sleeping on 429 in request: latency spikes and cascading failures.
- Ignoring Retry-After: avoidable throttling.
Solid flask error handling openai api is not about catching errors—it’s about classifying them, retrying the right ones, and returning a contract your clients can code against. Do that, and your LLM features stay up when the provider hiccups.