Testing LLM API integrations in RSpec with VCR

LLM API calls are slow, non-deterministic, and expensive to hit on every test run. Using rspec vcr llm api testing patterns lets you record real HTTP interactions once and replay them deterministically, turning flaky integration specs into fast unit-speed tests. This guide walks through a concrete setup for a Ruby client talking to an OpenAI-compatible chat completions endpoint.

Step 1: Add dependencies and configure VCR

Start with the gems you actually need: vcr and webmock. WebMock is the adapter VCR hooks into so it can intercept sockets. Do not pull in faraday or httparty just for the test harness; the client itself can use plain Net::HTTP to avoid extra layers.

# Gemfile
group :test do
  gem "rspec"
  gem "vcr"
  gem "webmock"
end

Run bundle install. Then create a support file that loads before your specs:

# spec/support/vcr.rb
require "vcr"

VCR.configure do |config|
  config.cassette_library_dir = "spec/cassettes"
  config.hook_into :webmock
  config.configure_rspec_metadata!
  config.filter_sensitive_data("<API_KEY>") { ENV["LLM_API_KEY"] }
  config.ignore_hosts "localhost", "127.0.0.1"
  config.default_cassette_options = {
    match_requests_on: [:method, :uri, :body],
    record: :once
  }
  # Fail hard in CI if a request is not cached
  config.allow_http_connections_when_no_cassette = false
end

The match_requests_on: [:body] is deliberate. LLM requests are uniquely identified by their JSON payload (model, messages, temperature). Matching on body prevents a test with a different prompt from accidentally replaying an unrelated cassette. record: :once means cassettes are written on first run and read thereafter—exactly what you want for rspec vcr llm api testing in a team setting.

Step 2: Build a minimal client wrapper

Never call Net::HTTP from your test directly. Wrap the endpoint so you can swap bases (OpenAI, a gateway, localhost) and assert on a parsed shape.

# lib/llm_client.rb
require "net/http"
require "json"

class LLMClient
  def initialize(api_base:, api_key:, model: "gpt-4o-mini")
    @api_base = api_base
    @api_key = api_key
    @model = model
  end

  def complete(prompt, temperature: 0)
    uri = URI("#{@api_base}/v1/chat/completions")
    req = Net::HTTP::Post.new(uri)
    req["Authorization"] = "Bearer #{@api_key}"
    req["Content-Type"] = "application/json"
    req.body = JSON.dump({
      model: @model,
      messages: [{ role: "user", content: prompt }],
      temperature: temperature
    })
    res = Net::HTTP.start(uri.host, uri.port, use_ssl: true) { |http| http.request(req) }
    raise "HTTP #{res.code}" unless res.code.start_with?("2")
    JSON.parse(res.body)
  end
end

Keep the wrapper thin. Business logic (retries, prompt building) belongs in a service object that calls #complete. That separation makes your recorded cassette reusable across many higher-level specs.

Step 3: Record your first cassette

Write a spec that exercises the client with a real key, but only once. VCR records on the first run and replays after.

# spec/llm_client_spec.rb
require "spec_helper"
require "llm_client"

describe LLMClient do
  let(:client) do
    LLMClient.new(
      api_base: "https://api.openai.com",
      api_key: ENV.fetch("LLM_API_KEY"),
      model: "gpt-4o-mini"
    )
  end

  it "returns a text completion", :vcr do
    resp = client.complete("What is 2+2? Answer with just the number.")
    content = resp.dig("choices", 0, "message", "content")
    expect(content).to be_a(String)
    expect(content).to match(/\d/)
  end
end

Run it:

LLM_API_KEY=sk-realkey bundle exec rspec spec/llm_client_spec.rb

On success, spec/cassettes/LLMClient/returns_a_text_completion.yml is created. Delete the key from your env and run again—the spec should pass fully offline. That offline pass is the core win of rspec vcr llm api testing: deterministic, free, and fast.

Step 4: Scrub non-deterministic fields

OpenAI-compatible responses include id and created that change per call. If you record with record: :once and later someone re-records, those fields drift and cause noise in diffs. Normalize them in the recording hook:

# spec/support/vcr.rb (add inside VCR.configure)
config.before_record do |interaction|
  body = interaction.response.body
  body = body.gsub(/"id":"[^"]+"/, '"id":"chatcmpl-fixed"')
  body = body.gsub(/"created":\d+/, '"created":1700000000')
  interaction.response.body = body
end

Also, set temperature: 0 in every recorded call (as the client default already does). With temperature zero and a fixed prompt, the content is stable enough for assertions on shape rather than exact wording. If you need exact wording, freeze the cassette and never re-record that spec.

For streaming endpoints, VCR handles chunked transfers poorly; record non-streaming only and test streaming separately with a local stub.

Step 5: Assert on token usage and latency

The usage block is the most valuable deterministic signal for cost control. Assert it exists and is sane:

it "reports token usage", :vcr do
  resp = client.complete("Explain tail recursion in one sentence.")
  usage = resp["usage"]
  expect(usage["prompt_tokens"]).to be > 0
  expect(usage["completion_tokens"]).to be > 0
  expect(usage["total_tokens"]).to eq(
    usage["prompt_tokens"] + usage["completion_tokens"]
  )
end

To fake latency without slowing tests, use WebMock’s stubbing on top of the cassette. But typically you just assert the call returned; latency testing belongs in load tests, not unit specs. If you must verify a timeout path, record a cassette with a delayed response using a local proxy, or manually edit the cassette to include a recorded_at far in the past—VCR does not enforce latency.

Step 6: Test failure and fallback paths

Real integrations must handle 429s and 5xxs. Record these by temporarily pointing the client at a script that returns errors, or hand-write a cassette. The cleaner approach: record against a gateway that does fallback for you.

If you point the client at n4n.ai, which provides one OpenAI-compatible endpoint covering 240+ models with automatic fallback when a provider is rate-limited, the recorded cassette captures the gateway’s 429 followed by a retried success without you mocking the fallback logic.

it "survives a 429 and still completes via fallback", :vcr do
  gw_client = LLMClient.new(
    api_base: "https://gateway.n4n.ai",
    api_key: "test-key",
    model: "anthropic/claude-3-haiku"
  )
  resp = gw_client.complete("Ping")
  expect(resp.dig("choices", 0, "message", "content")).to be_a(String)
end

The cassette for that spec contains two interactions: first a 429 with Retry-After, then a 200. VCR replays them in order. Your client code should catch the raise and retry (or rely on the gateway). Test both: the retry loop in your service object, and the happy path from the second interaction.

To hand-craft a 429 cassette:

# spec/cassettes/LLMClient/survives_a_429.yml (abridged)
- request:
    method: post
    uri: https://gateway.n4n.ai/v1/chat/completions
    body:
      model: anthropic/claude-3-haiku
  response:
    status:
      code: 429
      message: "Too Many Requests"
    headers:
      retry-after: ["0"]
- request:
    method: post
    uri: https://gateway.n4n.ai/v1/chat/completions
    body:
      model: anthropic/claude-3-haiku
  response:
    status:
      code: 200
    body: '{"choices":[{"message":{"content":"pong"}}],"usage":{"total_tokens":5}}'

Commit this file. Now the spec runs offline and proves your retry logic works.

Step 7: Run in CI and verify success

In CI, set LLM_API_KEY to a dummy and ensure no real network calls slip through:

# .github/workflows/test.yml (snippet)
- run: bundle exec rspec
  env:
    LLM_API_KEY: dummy

Because allow_http_connections_when_no_cassette = false, any spec missing a cassette fails immediately with VCR::Errors::UnhandledHTTPRequestError. That is the verification gate: if the suite is green, every LLM call was served from a cassette.

Check these three things to confirm your rspec vcr llm api testing setup is correct:

1. git ls-files spec/cassettes shows YAML for every LLM spec.
2. Running bundle exec rspec with no network access passes in under a second per spec.
3. Re-recording with a fresh key does not change assertions—only the scrubbed id/created may differ.

If those hold, you have a deterministic, cost-free test layer for your LLM features. Treat cassettes as fixtures: review them in PRs, and bump them when you change request shape, not when the model’s wording drifts.