ActiveJob retry strategies for failed LLM API calls

Building a reliable activejob retry llm api pipeline starts with classifying failures and separating transient from permanent errors. LLM endpoints throw rate limits, upstream timeouts, and malformed prompt rejections that each demand different treatment. If you blanket-retry every exception, you will amplify cost and duplicate side effects.

1. Classify LLM API failures before retrying

Not every 4xx or 5xx from an LLM provider is retryable. A 400 with invalid_prompt is permanent; a 429 with rate_limit_exceeded is transient; a 502 is usually transient but may indicate provider outage. A 413 for context length is permanent. Define explicit exception types in your client wrapper so ActiveJob can branch on them.

class LLMClient
  class RateLimitError < StandardError; end
  class TimeoutError < StandardError; end
  class InvalidRequestError < StandardError; end
  class ContextLengthError < StandardError; end
  class ServerError < StandardError; end
end

Map HTTP responses to these in your Faraday or Net::HTTP layer. Only then can you write precise retry_on rules instead of catching StandardError and hoping. Parsing streaming chunk errors deserves the same rigor: a mid-stream disconnect is a TimeoutError, not a ServerError.

2. Use retry_on with explicit exceptions and limits

ActiveJob’s retry_on is declarative but unforgiving if misused. Always specify attempts and prefer per-exception wait strategies. Never leave attempts unbounded. Order matters: discard_on should precede retry_on for permanent errors so they short-circuit.

class SummarizeJob < ApplicationJob
  queue_as :llm

  discard_on LLMClient::InvalidRequestError
  discard_on LLMClient::ContextLengthError
  retry_on LLMClient::RateLimitError, wait: 3.seconds, attempts: 5
  retry_on LLMClient::TimeoutError, wait: :exponential_backoff, attempts: 4
  retry_on LLMClient::ServerError, wait: :exponential_backoff, attempts: 3
end

The discard_on prevents wasting cycles on prompts that will never validate. Tradeoff: discarding loses the job entirely. If auditability matters, push to a dead-letter queue instead (see section 7). The default :exponential_backoff algorithm is attempt ** 2 seconds, which is fine for low volume but dangerous at scale.

3. Implement exponential backoff with jitter

The default :exponential_backoff in ActiveJob is deterministic. Under heavy load, synchronized retries cause thundering herds against the same provider. Add jitter at the client or override the wait proc.

def self.backoff_with_jitter(attempt)
  base = 2 ** attempt
  (base + rand(0.0..base.to_f)).seconds
end

class ChatJob < ApplicationJob
  queue_as :llm
  retry_on LLMClient::ServerError,
           wait: ->(attempt) { backoff_with_jitter(attempt) },
           attempts: 5
end

Jitter spreads retries across time. The cost is slightly higher worst-case latency for a single job, but system stability improves. For rate-limit errors, prefer the provider’s Retry-After over jitter (section 5).

4. Make jobs idempotent to survive retries

LLM calls often trigger downstream writes: store a summary, send a message, update a vector index. A retry after a timeout but successful upstream call duplicates that write. Key jobs by an idempotency token derived from business data, not job_id.

def perform(conversation_id, prompt, idempotency_key)
  lock = Redis.current.set("llm_lock:#{idempotency_key}", "1", nx: true, ex: 1.hour)
  return unless lock

  completion = LLMClient.complete(prompt)
  Conversation.find(conversation_id).append_summary(completion)
ensure
  Redis.current.del("llm_lock:#{idempotency_key}") if completion.present?
end

If the job dies after the LLM returns but before the write, the lock prevents a second execution from double-appending. Pitfall: using only job_id as the key fails when the queue re-enqueues a new job instance. For relational integrity, a unique DB constraint on (conversation_id, idempotency_key) is a stronger guarantee than Redis alone.

5. Handle rate limits with custom wait times

Provider 429s often include a Retry-After header. Honor it instead of guessing. Extract the value in your client and raise with a suggested delay attached to the exception.

class RateLimitError < StandardError
  attr_reader :retry_after
  def initialize(msg, retry_after: nil)
    @retry_after = retry_after
    super(msg)
  end
end

# in client
if response.status == 429
  raise LLMClient::RateLimitError.new("rate limited",
    retry_after: response.headers["Retry-After"].to_i)
end

ActiveJob doesn’t natively read exception attributes for wait, so pass a proc that consults the exception instance.

retry_on LLMClient::RateLimitError,
  wait: ->(attempt, exception) { exception.retry_after || 5.seconds },
  attempts: 6

Tradeoff: long Retry-After values (e.g., 30s) block a worker. Use a separate low-concurrency queue for LLM jobs to avoid starving other jobs.

6. Leverage provider fallback at the gateway level

If you route through a gateway such as n4n.ai, which performs automatic fallback when a provider is rate-limited or degraded, you can treat certain 429s as non-retryable at the job level and let the gateway reroute to a healthy endpoint. This simplifies your activejob retry llm api logic: you retry only on hard timeouts, not on transient provider capacity.

For direct-to-provider integrations, implement a similar fallback by swapping model IDs in the job payload. But that adds rescues for each provider’s error shape. A gateway with 240+ models behind one OpenAI-compatible endpoint removes the need to code multi-provider failover and lets you forward provider cache-control hints to avoid recomputing prompt prefixes on retry.

7. Track retry exhaustion and dead-letter jobs

After final attempt, ActiveJob invokes rescue_from or discards. Build a handler that logs and persists to a dead-letter table so failures are observable.

rescue_from LLMClient::ServerError do |exception|
  DeadLetter.create!(
    job_name: self.class.name,
    args: arguments,
    error: exception.message,
    occurred_at: Time.current
  )
  raise # re-raise so Sidekiq/GoodJob marks it failed
end

Without this, silently discarded jobs vanish. Pitfall: rescuing and not re-raising can hide bugs in dev. Use Rails.logger.error plus an external alert (Honeybadger, Sentry). For discard_on cases, hook after_discard to record the invalid request for prompt debugging.

8. Tune queue concurrency and timeouts

Retries compound when workers are saturated. Set queue_adapter (Sidekiq, GoodJob) concurrency limits per queue. Isolate LLM jobs:

# sidekiq.yml
:queues:
  - default
  - llm: 2

A worker pool of two for llm prevents 50 concurrent requests that trigger provider 429s. Also set HTTP client timeouts shorter than job ttl so failures surface fast.

Faraday.new(url: endpoint) do |f|
  f.options.timeout = 8
  f.options.open_timeout = 3
end

If a job’s total possible retry time exceeds your queue’s visibility timeout, the job can be double-executed. Compute max_wait = sum of backoff and keep it under the broker’s limit.

Common pitfalls and tradeoffs

• Retrying non-idempotent writes corrupts data. Always lock or use unique constraints.
• Ignoring context-length errors as retryable wastes calls; they are permanent.
• Using wait: 0 to “just retry fast” defeats backoff and gets you blocked harder.
• Over-retrying increases user-perceived latency; for chat UX, fail after 2 attempts and show a cached fallback.
• Mixing retry strategies without exception hierarchy leads to retry_on StandardError catching ContextLengthError and looping uselessly.

An activejob retry llm api design is not set-and-forget. Measure retry rates per exception class, adjust attempts, and keep idempotency at the core. The goal is not zero failures—it is bounded, observable, and cost-aware recovery.