Handling timeouts in Ruby LLM API clients

Ruby LLM API timeout handling is the difference between a resilient service and a thread pool full of zombies. When your Rails app calls an OpenAI-compatible endpoint to generate text, a missing or infinite read timeout will eventually stall a worker during a provider slowdown, and the rest of your stack pays for it.

Why LLM calls break differently than CRUD requests

A typical REST call returns in tens of milliseconds. An LLM completion can take seconds to minutes depending on model size, prompt length, and token count. The latency distribution has a long tail, and providers routinely shed load by slowing responses rather than returning 503s. If you only set a connect timeout, a half-open socket that never delivers bytes will hold a Ruby thread indefinitely under GVL contention.

You also have to distinguish three timeout types:

Connect (open) timeout: how long to wait for TCP/TLS handshake.
Write timeout: how long to wait to send the request body.
Read timeout: how long to wait between receiving response bytes.

For LLM APIs, read timeout is where most teams get burned.

Step 1: Choose an HTTP client that exposes raw timeouts

The openai Ruby gem wraps Faraday, but if you talk directly to an OpenAI-compatible gateway you should control the stack. Faraday with the net_http adapter or http.rb both expose the knobs you need. Avoid RestClient defaults—its timeout semantics are murky.

# Using http.rb with explicit tripartite timeouts
require 'http'

client = HTTP.timeout(connect: 5, read: 30, write: 5)

If you prefer Faraday:

require 'faraday'
require 'faraday/net_http'

conn = Faraday.new(url: "https://llm.example.com/v1") do |f|
  f.options.open_timeout = 5   # connect
  f.options.timeout = 30       # read/write
  f.adapter :net_http
end

Pick one and standardize. Mixing clients across services makes timeout policy invisible.

Step 2: Set explicit values per environment

Development can use generous timeouts; production should fail fast. A read timeout of 30–60 seconds is reasonable for non-streaming chat completions on mid-size models. For large context windows or code generation, push to 120s but pair with retries and circuit breakers.

# config/initializers/llm_client.rb
OPENAI_COMPAT_URL = ENV.fetch("LLM_GATEWAY_URL", "https://api.example.com/v1")

LLM_CONN = Faraday.new(url: OPENAI_COMPAT_URL) do |f|
  f.options.open_timeout = Integer(ENV.fetch("LLM_CONNECT_TIMEOUT", 5))
  f.options.timeout      = Integer(ENV.fetch("LLM_READ_TIMEOUT", 45))
  f.adapter :net_http
end

Never leave these as nil. Net::HTTP defaults to no read timeout, which means nil equals “wait forever.”

Step 3: Retry only transient timeouts with backoff

A timeout is not the same as a 429 or 500. Retrying a timeout blindly can amplify load. Wrap only the specific exception classes and use exponential backoff with a hard cap.

require 'retriable'

class LLMClient
  def self.complete(payload)
    Retriable.retry_on(
      [Faraday::TimeoutError, Net::ReadTimeout, Net::OpenTimeout],
      tries: 3,
      base_interval: 0.5,
      multiplier: 2.0,
      max_interval: 4.0
    ) do
      LLM_CONN.post("/chat/completions") do |req|
        req.headers["Content-Type"] = "application/json"
        req.body = payload.to_json
      end
    end
  end
end

Note: if the provider already returned a partial response, retrying may double-generate. For non-idempotent writes, pass an idempotency-key header if your gateway supports it. n4n.ai honors client routing directives and forwards provider cache-control hints, but idempotency is on your request path.

Step 4: Add a circuit breaker for sustained degradation

Retries handle blips. A region-wide outage or a provider’s broken autoscaler needs a circuit breaker so you stop sending traffic entirely for a cool-down window.

require 'circuitbox'

CIRCUIT = Circuitbox.circuit(
  :llm_gateway,
  exceptions: [Faraday::TimeoutError, Net::ReadTimeout],
  volume_threshold: 20,    # min calls before evaluating
  error_threshold: 50,     # % errors to open
  time_window: 60,
  sleep_window: 30         # seconds circuit stays open
)

def self.safe_complete(payload)
  CIRCUIT.run { complete(payload) }
rescue Circuitbox::OpenCircuitError
  Metrics.increment("llm.circuit_open")
  raise LLMUnavailable, "circuit open"
end

In Rails, surface LLMUnavailable to your controller and return a 503 or degraded fallback UI.

Step 5: Treat streaming responses differently

Streaming completions send Server-Sent Events. The read timeout must account for the full generation time, not just the first byte. With http.rb, set a long read timeout and process chunks:

streamer = HTTP.timeout(connect: 5, read: 180, write: 5)
response = streamer.post("#{OPENAI_COMPAT_URL}/chat/completions",
  json: { stream: true, model: "gpt-4o-mini", messages: [...] })

response.body.each_line do |line|
  next if line.strip.empty?
  data = line.sub(/^data: /, "")
  break if data == "[DONE]"
  yield JSON.parse(data)
end

If you use Faraday’s on_data callback, the same read timeout applies per chunk, so keep it generous.

Step 6: Leverage gateway fallback but keep client timeouts

A gateway such as n4n.ai provides automatic fallback when a provider is rate-limited or degraded, but your Ruby client must still enforce its own timeout; otherwise the gateway’s fallback cannot kick in until your request hangs. The gateway can route to a secondary model, yet if your read timeout is 300s you’ve already ruined your p99 latency before the fallback matters.

Set client timeouts tighter than your SLO, and let the gateway’s routing handle provider-level issues:

# Client-side: fail fast
LLM_CONN.options.timeout = 20
# Gateway-side: route to fallback model on upstream 429/5xx
# (configuration lives in gateway, not Ruby)

This separation lets you retry locally and still benefit from upstream redundancy.

Step 7: Verify with fault injection

You cannot claim ruby llm api timeout handling works until you have simulated a slow backend. Use a tiny Sinatra mock that sleeps longer than your timeout, or toxiproxy to add latency.

# spec/support/slow_mock.rb
require 'sinatra/base'

class SlowLLM < Sinatra::Base
  post '/v1/chat/completions' do
    sleep 15  # longer than our 5s test timeout
    content_type :json
    { choices: [{ text: "ok" }] }.to_json
  end
end

Run it on a port, point LLM_GATEWAY_URL there, and assert that Faraday::TimeoutError is raised within the bound:

# spec/llm_client_spec.rb
it "times out before the mock responds" do
  start_slow_mock(port: 9292)
  ENV["LLM_GATEWAY_URL"] = "http://localhost:9292"
  conn = Faraday.new(url: ENV["LLM_GATEWAY_URL"]) do |f|
    f.options.open_timeout = 2
    f.options.timeout = 3
    f.adapter :net_http
  end
  expect {
    conn.post("/v1/chat/completions") { |r| r.body = "{}" }
  }.to raise_error(Faraday::TimeoutError)
end

For circuit breaker tests, drive 20 failing calls and assert the 21st raises OpenCircuitError without hitting the network.

Step 8: Instrument and meter what you enforce

Timeouts are silent killers if unobserved. Emit histograms for llm.request.duration and counters for llm.timeout.count. If you use a gateway with per-token usage metering, correlate timeout events with billed tokens to spot partial generations that cost money but returned nothing.

ActiveSupport::Notifications.instrument("llm.request", model: "gpt-4o") do
  LLMClient.safe_complete(payload)
rescue Faraday::TimeoutError => e
  Metrics.increment("llm.timeout", tags: ["model:gpt-4o"])
  raise
end

A production-grade ruby llm api timeout handling strategy is not a single number; it is connect/read/write separation, scoped retries, a breaker, streaming awareness, and proof via fault injection. Ship the test first, then the client.