n4nAI
Open console →
API integrationHow-toRuby on Rails LLM Integration

Rate limiting LLM requests in Rails with rack-attack

Implement rails rack-attack llm rate limiting to protect your Rails app and LLM provider quotas. Step-by-step throttling, token-aware guards, and tests.

n4n Team4 min read874 words

Audio narration

Coming soon — every post will get a voice note here.

Rate limiting LLM calls in a Rails app is different from throttling a typical JSON endpoint. Token costs, slow responses, and provider quotas mean you need a layered defense. This guide walks through implementing rails rack-attack llm rate limiting that protects your app and your upstream bill without sacrificing legitimate traffic.

Step 1: Install and mount rack-attack correctly

Add the gem to your Gemfile:

gem 'rack-attack'

Run bundle install. For a Rails API or a full app, mount the middleware as early as possible in the stack so it runs before the router and before any expensive controller logic:

# config/initializers/rack_attack.rb
Rails.application.config.middleware.use Rack::Attack

If you run Puma with multiple workers or you deploy on more than one node, the default in-memory store will give each process its own counters and your limits will be effectively multiplied. Use Redis:

Rack::Attack.cache.store = Redis.new(url: ENV.fetch('REDIS_URL'))

For Rails 7.1+ with the new default Rails.application.config.cache_store, you can reuse the app’s Redis by assigning Rack::Attack.cache.store = Rails.cache.redis if you use RedisCacheStore. Either way, shared state is non-negotiable for accurate rails rack-attack llm rate limiting.

Step 2: Scope throttles to LLM routes

You rarely want to throttle your entire app with the same rule. LLM proxy endpoints are usually POST paths like /v1/chat/completions or your own /api/generate. Use the request object inside the throttle block to match:

Rack::Attack.throttle('llm/requests', limit: 20, period: 60) do |req|
  if req.post? && req.path.start_with?('/v1/chat')
    req.env['HTTP_X_API_KEY'] || req.ip
  end
end

The block returns the discriminator. Returning nil means “don’t throttle this request”. Here we key by API key header, falling back to IP. A key point: rack-attack counts requests, not tokens. Twenty requests per minute per key is a sane starting ceiling for chat completions, but a single request can still bankrupt you if it asks for 100k tokens.

Step 3: Add a token budget guard

Request counting alone is weak for LLMs. Read the body to extract max_tokens and approximate prompt size, then enforce a separate hard cap in application code. In the middleware you can only block; you cannot sum across requests without custom Redis logic. A practical compromise:

Rack::Attack.blocklist('llm/oversized_payload') do |req|
  next false unless req.post? && req.path.start_with?('/v1/chat')
  
  length = req.content_length.to_i
  length > 500_000 # ~125k tokens of JSON
end

And in the controller, enforce the generation cap:

# app/controllers/v1/chat_controller.rb
before_action :enforce_token_cap

private

def enforce_token_cap
  requested = params[:max_tokens].to_i
  render json: {
    error: { message: 'max_tokens exceeds policy', type: 'invalid_request_error' }
  }, status: :unprocessable_entity if requested > 8_192
end

If you truly need a rolling token budget (e.g., 100k tokens/min per key), write a small custom filter using Rails.cache with increment and a TTL, but keep it outside rack-attack’s throttle DSL. The rails rack-attack llm rate limiting setup should stay simple enough that you can reason about it at 3 a.m.

Step 4: Return provider-shaped errors

Rack-attack’s default response is a bare 429 text page. OpenAI-compatible clients expect JSON. Override the throttled response globally:

Rack::Attack.throttled_response = lambda do |env|
  match_data = env['rack.attack.match_data']
  retry_after = (match_data && match_data[:period]) || 60
  [
    429,
    { 'Content-Type' => 'application/json', 'Retry-After' => retry_after.to_s },
    [{ error: { message: 'Rate limit reached', type: 'rate_limit_error' } }.to_json]
  ]
end

This makes your gateway indistinguishable from the upstream on limit errors, which matters when you have retry logic in the client that keys off error.type.

Step 5: Safelist and blocklist deliberately

Never throttle your own infrastructure. Health checks, metrics scrapers, and internal admin calls should bypass the rules:

Rack::Attack.safelist('allow/internal') do |req|
  req.path.start_with?('/health', '/metrics') || req.ip == '127.0.0.1'
end

If you terminate TLS at a load balancer, req.ip is the LB IP. Configure config.action_dispatch.trusted_proxies and use req.remote_ip so the real client IP is used. Otherwise a single NAT gateway will share one limit across thousands of users, defeating the purpose of rails rack-attack llm rate limiting.

Step 6: Handle streaming connections

Streaming chat completions hold the TCP connection open for the duration of generation. Rack-attack’s period counter only sees the initial request. You must throttle the POST itself (already covered) and limit concurrent connections at the web server layer. In Puma, set a low max_threads per worker and use NGINX limit_conn. Critically, return the 429 before you start streaming:

# In the throttle block, include streaming paths
req.path.start_with?('/v1/chat', '/v1/completions')

Once you write the HTTP/1.1 200 OK header and begin SSE chunks, you cannot later send a 429. The throttle runs before the controller, so you are safe as long as you don’t bypass it with safelist.

Step 7: Write a boundary test

A request spec that crosses the limit proves the rule works and documents the expected behavior:

# spec/requests/llm_rate_limit_spec.rb
RSpec.describe 'LLM rate limiting' do
  let(:headers) { { 'HTTP_X_API_KEY' => 'test-key' } }

  it 'allows 20 requests per minute then blocks' do
    20.times do
      post '/v1/chat', params: { messages: [] }, headers: headers
      expect(response).not_to have_http_status(:too_many_requests)
    end

    post '/v1/chat', params: { messages: [] }, headers: headers
    expect(response).to have_http_status(:too_many_requests)
    body = JSON.parse(response.body)
    expect(body.dig('error', 'type')).to eq('rate_limit_error')
    expect(response.headers['Retry-After']).to be_present
  end
end

Run this with RAILS_ENV=test and a Redis instance. If your test suite uses the memory store, set Rack::Attack.cache.store = ActiveSupport::Cache::MemoryStore.new in the test env only, and clear it between examples with Rack::Attack.cache.store.clear.

Step 8: Verify in production

Ship the initializer, then watch logs. Subscribe to the notification:

ActiveSupport::Notifications.subscribe('rack.attack') do |_name, _start, _finish, _id, payload|
  req = payload[:request]
  Rails.logger.info("[rack-attack] #{req.env['rack.attack.match_type']} #{req.path} key=#{req.env['rack.attack.match_data']&.dig(:discriminator)}")
end

Look for throttle and blocklist matches. If you see a spike from one corporate NAT IP, move the discriminator to the API key or a signed session claim. The point of rails rack-attack llm rate limiting is to protect the upstream, not to punish legitimate shared egress.

If you front Rails with an OpenAI-compatible gateway such as n4n.ai, its automatic fallback when a provider is degraded and per-token metering handle the upstream side. Your local throttling still matters: it prevents a queue storm from ever reaching the gateway and incurring metered tokens on rejected retries.

Step 9: Tune the limits with data

Start conservative: 10 chat requests/min per key, 2 embeddings requests/min. After a week, inspect Redis key cardinality:

redis-cli --scan --pattern 'rack_attack:llm/*' | wc -l

And sample hit counts:

redis-cli get 'rack_attack:llm/requests:test-key'

Bump limits based on p95 usage, not averages. A viral feature will blow your provider quota in seconds flat; the local throttle is your circuit breaker.

Final checklist

  • Middleware mounted early, Redis store shared.
  • Path-scoped throttles for /v1/chat and /v1/completions.
  • max_tokens cap enforced in controller; oversized payloads blocked.
  • 429 response shaped like the provider’s error.
  • Safelist for health and internal paths; correct trusted proxies.
  • Streaming POST throttled pre-flush.
  • Boundary spec green.
  • Logging subscription active in production.

That is the full path to production-grade rails rack-attack llm rate limiting. Ship it, watch the logs, and adjust before the bill does.

Tagsrailsrack-attackrate-limitingerror-handling

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationActiveJob retry strategies for failed LLM API callsPractical ActiveJob retry patterns for resilient Rails integrations with LLM APIs, covering backoff, idempotency, provider fallback, and queue tuning.API integrationBackground LLM calls in Rails with SidekiqStep-by-step guide to implementing rails sidekiq background llm calls: job design, retries, idempotency, and streaming without blocking web requests.API integrationHandling timeouts in Ruby LLM API clientsLearn practical ruby llm api timeout handling: configure connect/read timeouts, retry with backoff, circuit breakers, and verify with fault injection.API integrationStoring LLM API keys with Rails encrypted credentialsA practical guide to managing Rails encrypted credentials API key storage for LLM services, including setup, rotation, and verification steps.API integrationStreaming LLM responses to the browser with ActionCableA practical guide to rails actioncable llm streaming: wire Ruby on Rails WebSockets to an LLM API and push token deltas to the browser live.API integrationBuilding a chat feature in Rails with Hotwire and n4nHands-on tutorial: build a Rails Hotwire chat feature backed by n4n's OpenAI-compatible LLM gateway. Models, controllers, Turbo streams, and code.

All ruby on rails llm integration posts →