Background LLM calls in Rails with Sidekiq

Moving rails sidekiq background llm calls out of the request path is non-negotiable once your LLM prompts take longer than your users will wait on a spinner. This walkthrough builds a complete pipeline: a Sidekiq worker that hits an OpenAI-compatible inference endpoint, writes the response to Postgres, and survives retries and provider outages without losing work.

Step 1: Install and configure Sidekiq

Sidekiq requires Redis. Add the gems and pin versions you have tested.

# Gemfile
gem "sidekiq", "~> 7.0"
gem "redis", "~> 5.0"

Run bundle install, then start Redis locally for development:

redis-server --daemonize yes

Configure the connection explicitly so both server and client use the same Redis DB. Put this in config/sidekiq.rb:

redis_url = ENV.fetch("REDIS_URL", "redis://localhost:6379/0")

Sidekiq.configure_server do |config|
  config.redis = { url: redis_url }
end

Sidekiq.configure_client do |config|
  config.redis = { url: redis_url }
end

Mount the Web UI behind authentication. Ops needs visibility into queue depth and dead jobs.

# config/routes.rb
require "sidekiq/web"
Rails.application.routes.draw do
  authenticate :user, ->(u) { u.admin? } do
    mount Sidekiq::Web => "/sidekiq"
  end
end

Set Sidekiq concurrency based on your worker pool size and the blocking behavior of your HTTP client. Net::HTTP blocks a thread, so a concurrency of 10 means 10 simultaneous LLM calls. Tune in config/sidekiq.yml or via -c flag.

Step 2: Wrap the LLM endpoint in a tiny client

Scatter HTTP calls across workers and you will rewrite error handling three times. Write one client that speaks the OpenAI chat completions shape. If you want a single base URL that fronts 240+ models with automatic fallback when a provider is rate-limited or degraded, point this at a gateway like n4n.ai; otherwise use the vendor URL directly.

# app/clients/llm_client.rb
class LlmClient
  def initialize(base_url:, api_key:, model:)
    @base_url = base_url
    @api_key = api_key
    @model = model
  end

  def complete(prompt, temperature: 0.7, timeout: 30)
    uri = URI.join(@base_url, "/v1/chat/completions")
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = uri.scheme == "https"
    http.open_timeout = 5
    http.read_timeout = timeout

    req = Net::HTTP::Post.new(uri)
    req["Authorization"] = "Bearer #{@api_key}"
    req["Content-Type"] = "application/json"
    req.body = {
      model: @model,
      messages: [{ role: "user", content: prompt }],
      temperature: temperature
    }.to_json

    res = http.request(req)
    return JSON.parse(res.body).dig("choices", 0, "message", "content") if res.code.start_with?("2")
    raise "LLM HTTP #{res.code}: #{res.body}"
  end
end

Keep the client free of Rails.application references. That makes it unit-testable with a stubbed TCP socket or WebMock.

Why a gateway instead of direct vendor calls

A background worker should not care which provider is healthy at 3 a.m. An OpenRouter-class gateway hands you one endpoint and routes around degraded models, which removes the need to write your own fallback branch inside the job. You still pass model per call, but the gateway resolves it to a live provider.

Step 3: Persist the request and response

You need a durable record so a worker crash does not lose the prompt. Generate a table:

rails g model LlmCompletion prompt:text model:string status:string result:text job_id:string
rails db:migrate

Add an index on status and job_id for queue inspection queries. In the model, enforce a strict state machine:

class LlmCompletion < ApplicationRecord
  STATUSES = %w[pending running completed failed].freeze
  validates :status, inclusion: { in: STATUSES }
  before_create { self.status ||= "pending" }

  def completed? = status == "completed"
  def running?   = status == "running"
end

Do not store the API key or full request payload here; the prompt is enough for debugging.

Step 4: Write the Sidekiq worker

Use a plain Sidekiq worker, not ActiveJob, to get direct access to retry metadata. The worker loads the record, guards against double execution, calls the client, and writes the result.

# app/workers/llm_call_worker.rb
class LlmCallWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5, backtrace: true

  def perform(completion_id)
    completion = LlmCompletion.find(completion_id)
    return if completion.completed? || completion.running?

    completion.update!(status: "running")
    client = LlmClient.new(
      base_url: ENV.fetch("LLM_BASE_URL"),
      api_key: ENV.fetch("LLM_API_KEY"),
      model: completion.model
    )

    result = client.complete(completion.prompt)
    completion.update!(status: "completed", result: result)
  rescue => e
    completion&.update!(status: "failed", result: e.message)
    raise
  end
end

The early return is the idempotency guard. Sidekiq may redeliver a job after a crash; if the row is already running or completed, we skip. Never wrap the whole perform in a DB transaction—long-running HTTP calls inside a transaction hold locks and kill Postgres connections.

Step 5: Tune retries and dead-letter behavior

Sidekiq uses exponential backoff by default (10s, 30s, 60s, …). For LLM rate limits, that is reasonable. Cap retries so a permanently broken prompt (e.g., malformed JSON schema) moves to the Dead set instead of retrying for an hour:

sidekiq_options retry: 5

After 5 failures the job appears in the Dead tab of the Web UI. From there you can inspect the exception and requeue manually. If you need custom backoff, use the sidekiq-retry gem or a small middleware that sets msg["retry_at"].

Do not set retry: false blindly; transient 429s from the provider are normal and should be retried.

Step 6: Enqueue from a controller without blocking

The controller creates the row and fires the job. It never awaits the LLM response.

# app/controllers/completions_controller.rb
class CompletionsController < ApplicationController
  def create
    completion = LlmCompletion.create!(
      prompt: params[:prompt],
      model: params[:model] || "gpt-4o-mini"
    )
    LlmCallWorker.perform_async(completion.id)
    render json: { id: completion.id, status: completion.status }, status: :accepted
  end

  def show
    render json: LlmCompletion.find(params[:id])
  end
end

The browser polls show or you push via ActionCable when the status flips. Either way the HTTP request returns in milliseconds. This is the core benefit of rails sidekiq background llm calls: the web tier stays responsive while the heavy inference runs elsewhere.

Step 7: Run and deploy the worker

Start Sidekiq pointing at your queue:

bundle exec sidekiq -q default -c 10 -v

In production use a process manager. A minimal systemd unit:

[Service]
ExecStart=/app/bin/bundle exec sidekiq -q default -c 10
Restart=on-failure
Environment=REDIS_URL=redis://redis.internal:6379/0

Ensure Redis is up before Sidekiq starts. If the client cannot connect, Sidekiq buffers enqueues in memory, which masks outages. Health-check Redis in your deploy script.

Step 8: Verify the pipeline end to end

Test the synchronous path in a console first:

c = LlmCompletion.create!(prompt: "Summarize Sidekiq in one sentence.", model: "gpt-4o-mini")
LlmCallWorker.new.perform(c.id)
c.reload.status # => "completed"
c.result        # => non-empty string

Then exercise the real queue:

rails runner 'LlmCallWorker.perform_async(LlmCompletion.last.id)'
tail -f log/sidekiq.log

Expect to see the job fetch, the HTTP POST, and the completed update. Open /sidekiq; the Dead tab should be empty. For automated tests, stub the client:

allow_any_instance_of(LlmClient).to receive(:complete).and_return("stubbed")
completion = LlmCompletion.create!(prompt: "test")
LlmCallWorker.new.perform(completion.id)
expect(completion.reload.status).to eq("completed")

Verifying with curl

Hit the controller to confirm the async contract:

curl -X POST localhost:3000/completions -d 'prompt=hi' -d 'model=gpt-4o-mini'
# => {"id":12,"status":"pending"}
curl localhost:3000/completions/12
# => {"id":12,"status":"completed","result":"..."}

Step 9: Pass provider cache and routing hints

OpenAI-compatible gateways accept headers for cache control and routing. If your gateway honors client routing directives and forwards provider cache-control hints, extend the client:

def complete(prompt, temperature: 0.7, route: nil, cache: false)
  # ... build request ...
  req["X-Route"] = route if route
  req["Cache-Control"] = "max-age=3600" if cache
  # ...
end

Now a worker can pin a job to a specific provider or reuse a prompt cache without changing the worker body. This matters when you run rails sidekiq background llm calls across multiple tenants with different compliance requirements.

Operational notes

• Set http.read_timeout to something smaller than your Sidekiq death timeout (default 25s for signal handling). A hung connection should fail the job, not the process.
• Log completion_id and model on every attempt. When you audit spend later, per-token metering from your gateway maps to those ids.
• Batch by enqueuing many small jobs instead of one giant prompt. Sidekiq concurrency parallelizes better than a single long call, and a failure isolates to one row.
• Alert on Dead job count. A pile-up of failed rails sidekiq background llm calls usually means an expired API key, a deprecated model name, or a provider-side outage.
• Use sidekiq_options queue: "llm" to isolate LLM jobs from transactional email jobs so a slow model does not delay password resets.

Following these steps gives you a resilient pattern for rails sidekiq background llm calls that keeps the web tier fast and the inference tier observable.