Streaming LLM responses to the browser with ActionCable

When you need rails actioncable llm streaming in a Ruby on Rails app, buffering the full model response before sending it to the client is a non-starter for chat UX. ActionCable gives you a WebSocket channel to push token deltas to the browser the moment they arrive from the inference provider. The following steps build a minimal but production-shaped pipeline from a Rails channel subscription to live text on the page.

Step 1: Generate the ActionCable channel

Run the Rails generator and strip the boilerplate to a single room-scoped stream.

rails generate channel LlmChat

Edit app/channels/llm_chat_channel.rb:

class LlmChatChannel < ApplicationCable::Channel
  def subscribed
    return reject unless params[:room].present?
    stream_from "llm_chat_#{params[:room]}"
  end

  def unsubscribed
    stop_all_streams
  end

  def ask(data)
    LlmStreamJob.perform_later(params[:room], data["prompt"])
  end
end

We delegate the actual LLM call to a job immediately. Running the HTTP stream inside the channel method blocks the WebSocket worker and will crash under concurrency. The job pattern also makes rails actioncable llm streaming resilient to deploy restarts, because the job can be retried or resumed by your queue backend.

If you authenticate connections via cookies, set it in app/channels/application_cable/connection.rb:

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    identified_by :current_user

    def connect
      self.current_user = User.find_by(id: cookies.signed[:user_id])
      reject_unauthorized_connection unless current_user
    end
  end
end

Step 2: Configure the streaming LLM client

Use the official ruby-openai gem. It supports the stream: true flag and yields chunks. Point it at any OpenAI-compatible gateway.

# config/initializers/llm.rb
OPENAI_CLIENT = OpenAI::Client.new(
  access_token: ENV.fetch("LLM_API_KEY"),
  uri_base: ENV.fetch("LLM_BASE_URL", "https://api.openai.com/v1")
)

If you point LLM_BASE_URL at n4n.ai, its OpenAI-compatible endpoint handles automatic fallback when a provider is rate-limited and meters per token, so the streaming source stays stable without custom rescue logic in your job.

The client yields a hash per SSE event. For chat completions, a typical chunk looks like:

{
  "choices": [
    { "delta": { "content": "Hello" }, "index": 0, "finish_reason": null }
  ]
}

Extract the string with chunk.dig("choices", 0, "delta", "content"). The final chunk has finish_reason: "stop" and an empty delta.

Step 3: Write the background job that broadcasts

Create app/jobs/llm_stream_job.rb. The job opens the stream, broadcasts each delta to the channel stream name, and signals completion.

class LlmStreamJob < ApplicationJob
  queue_as :default

  def perform(room, prompt)
    stream_name = "llm_chat_#{room}"
    client = OPENAI_CLIENT

    begin
      client.chat(
        parameters: {
          model: ENV.fetch("LLM_MODEL", "gpt-4o-mini"),
          messages: [{ role: "user", content: prompt }],
          stream: true
        }
      ) do |chunk, _bytes|
        delta = chunk.dig("choices", 0, "delta", "content")
        ActionCable.server.broadcast(stream_name, { delta: delta }) if delta
      end
      ActionCable.server.broadcast(stream_name, { done: true })
    rescue StandardError => e
      ActionCable.server.broadcast(stream_name, { error: e.message })
    end
  end
end

ActionCable.server.broadcast serializes the hash to JSON and pushes to all subscribers of that stream. This keeps the WebSocket thread free. Rails actioncable llm streaming now scales with your job backend rather than your web workers.

Step 4: Mount the consumer and render deltas

Assuming importmap (default Rails 7+), the channel JavaScript lives in app/javascript/channels/llm_chat_channel.js. Ensure app/javascript/channels/consumer.js exists from the generator.

import consumer from "./consumer"

const room = document.getElementById("room").value

const subscription = consumer.subscriptions.create({ channel: "LlmChatChannel", room }, {
  connected() {
    this.ask = (prompt) => this.perform("ask", { prompt })
  },

  received(data) {
    const out = document.getElementById("output")
    if (data.delta) {
      out.textContent += data.delta
    } else if (data.done) {
      out.textContent += "\n[stream complete]\n"
    } else if (data.error) {
      out.textContent += `\n[error] ${data.error}\n`
    }
  }
})

export default subscription

A simple HTML scaffold in your view:

<%= hidden_field_tag :room, "demo-123" %>
<textarea id="prompt" placeholder="Ask something..."></textarea>
<button id="send">Send</button>
<pre id="output"></pre>

<script type="module">
  import subscription from "../channels/llm_chat_channel"
  document.getElementById("send").addEventListener("click", () => {
    subscription.ask(document.getElementById("prompt").value)
  })
</script>

The received callback appends text as it arrives. No polling, no full-response wait. Because the WebSocket is same-origin, no CORS configuration is needed.

Step 5: Authenticate and scope streams

Never trust the room param blindly. In subscribed, check the current user:

def subscribed
  room = params[:room]
  return reject unless current_user&.can_access?(room)
  stream_from "llm_chat_#{room}"
end

Also reject if the job attempts to broadcast to a stream with no subscribers. ActionCable’s broadcast is fire-and-forget; wrap it in a guard if you pay per token:

if ActionCable.server.pubsub.broadcasting?(stream_name)
  ActionCable.server.broadcast(stream_name, { delta: delta })
end

For rails actioncable llm streaming in multi-tenant apps, prefix the stream with the account id, not just a client-supplied string. Use stream_from "acct_#{current_user.account_id}_room_#{room}" and match it in the job.

Step 6: Handle cancellation and backpressure

Users close tabs mid-stream. The job keeps running and spends tokens. Pass a cancellation token via Redis:

def perform(room, prompt)
  stream_name = "llm_chat_#{room}"
  cancel_key = "cancel:#{room}"
  REDIS.del(cancel_key)

  client.chat(parameters: {...}) do |chunk, _|
    break if REDIS.exists?(cancel_key)
    # broadcast delta
  end
end

In the channel’s unsubscribed, set the key:

def unsubscribed
  REDIS.set("cancel:#{params[:room]}", "1", ex: 60)
  stop_all_streams
end

Backpressure is minimal because ActionCable buffers sends in the WebSocket adapter. If your LLM streams faster than the browser paints, batch deltas every 50 ms in the job using a buffer string and a timer.

Step 7: Verify the pipeline end to end

Start Redis and the Rails server, then open two terminals:

rails s
redis-server

In the browser, load the page, open devtools → Network → WS, and click Send. You should see frames arriving as {"delta":"Hello"}, {"delta":" world"}, ending with {"done":true}.

To assert server-side, tail logs:

tail -f log/development.log | grep LlmStreamJob

You will see the job enqueue and broadcast lines. For an automated check, use a WebSocket client in RSpec:

it "streams deltas" do
  # use ActionCable::Channel::TestCase or a real ws client
  subscribe room: "test"
  perform :ask, prompt: "hi"
  assert_broadcasts("llm_chat_test", 1) { LlmStreamJob.perform_now("test", "hi") }
end

If tokens appear incrementally and the connection closes cleanly on tab close, the rails actioncable llm streaming implementation is correct.

Production notes

Use the Redis action cable adapter, not the async adapter. Set config.action_cable.url to your domain’s wss endpoint behind a load balancer that supports sticky WebSockets.

Move the job to SolidQueue or Sidekiq. A bare Thread.new inside a channel will take down the server under ten concurrent users. The pattern above with LlmStreamJob already isolates that risk.

Finally, meter cost. If you use a gateway that honors client routing directives, set X-Route-Model headers or pass model explicitly per request. The broadcast payload should never include raw credentials or full prompt context—only the text delta.

That’s the full path from a Rails route to live model output in the browser.