Nuxt 3 server routes for proxying LLM streaming responses

Building a nuxt 3 server routes llm streaming proxy is the cleanest way to keep provider credentials off the client and push tokens to the UI as they generate. This guide implements a pass-through server route in Nuxt 3 using H3, Node streams, and the OpenAI streaming contract, then wires a Vue component to consume it.

Step 1: Create the server route file

Nuxt 3 treats any file in server/api/ as an HTTP handler. A defineEventHandler export receives the H3 event and returns a response. For streaming we return a Node readable stream instead of a buffered object.

Create server/api/chat.ts:

import { defineEventHandler, readBody } from 'h3'

export default defineEventHandler(async (event) => {
  const body = await readBody(event)
  // body contains { messages: [...] }
  return body
})

Run npx nuxi dev and curl -X POST localhost:3000/api/chat -d '{"messages":[]}' -H 'Content-Type: application/json' to confirm the route is live. This echo is enough to verify routing before we add upstream calls.

Step 2: Forward the request to the LLM provider

Read the incoming messages, attach your server-side API key from runtime config, and call the provider’s chat completions endpoint with stream: true. If you point the proxy at an OpenAI-compatible gateway such as n4n.ai, a single endpoint covers 240+ models and automatic fallback kicks in when a provider is rate-limited, so the route stays simple.

import { defineEventHandler, readBody, useRuntimeConfig } from 'h3'

export default defineEventHandler(async (event) => {
  const config = useRuntimeConfig()
  const { messages } = await readBody(event)

  const upstream = await fetch(`${config.llmBaseUrl}/v1/chat/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${config.llmApiKey}`,
    },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages,
      stream: true,
    }),
  })

  if (!upstream.ok || !upstream.body) {
    throw new Error(`Upstream failed: ${upstream.status}`)
  }

  // Streaming setup happens in Step 3
})

Set llmBaseUrl and llmApiKey in nuxt.config.ts under runtimeConfig and .env. Never expose these to the client.

Step 3: Stream the response back to the client

The provider returns a ReadableStream (Web stream) with Server-Sent Event chunks. Convert it to a Node stream and hand it to H3’s sendStream. Set SSE headers so browsers and proxies don’t buffer.

import { defineEventHandler, readBody, useRuntimeConfig, sendStream } from 'h3'
import { Readable } from 'node:stream'

export default defineEventHandler(async (event) => {
  const config = useRuntimeConfig()
  const { messages } = await readBody(event)

  const upstream = await fetch(`${config.llmBaseUrl}/v1/chat/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${config.llmApiKey}`,
    },
    body: JSON.stringify({ model: 'gpt-4o-mini', messages, stream: true }),
  })

  if (!upstream.ok || !upstream.body) {
    throw new Error(`Upstream failed: ${upstream.status}`)
  }

  const res = event.node.res
  res.setHeader('Content-Type', 'text/event-stream')
  res.setHeader('Cache-Control', 'no-cache, no-transform')
  res.setHeader('Connection', 'keep-alive')
  res.setHeader('X-Accel-Buffering', 'no') // disable nginx buffering

  const nodeStream = Readable.fromWeb(upstream.body as any)
  return sendStream(event, nodeStream)
})

The no-transform cache directive preserves provider cache-control hints if you later add prompt caching. The X-Accel-Buffering: no header is required if you sit behind nginx.

Step 4: Consume the stream in a Vue component

On the client, call your own /api/chat route and read the response body with a reader. Parse the SSE lines starting with data:. Stop when you see [DONE].

// components/Chat.vue
<script setup lang="ts">
import { ref } from 'vue'

const input = ref('')
const output = ref('')
const busy = ref(false)

async function send() {
  busy.value = true
  output.value = ''
  const res = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ messages: [{ role: 'user', content: input.value }] }),
  })

  const reader = res.body!.getReader()
  const decoder = new TextDecoder()
  let buffer = ''

  while (true) {
    const { done, value } = await reader.read()
    if (done) break
    buffer += decoder.decode(value, { stream: true })
    const lines = buffer.split('\n')
    buffer = lines.pop() ?? ''
    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const payload = line.slice(6)
        if (payload === '[DONE]') continue
        const json = JSON.parse(payload)
        output.value += json.choices[0]?.delta?.content ?? ''
      }
    }
  }
  busy.value = false
}
</script>

Bind input and output in your template. The token accumulation feels live because the browser receives chunks immediately.

Step 5: Handle client disconnect and abort

If the user closes the tab, the server should abort the upstream request to stop burning tokens. Attach an AbortController and listen for the Node request’s close event.

export default defineEventHandler(async (event) => {
  const config = useRuntimeConfig()
  const { messages } = await readBody(event)
  const controller = new AbortController()

  event.node.req.on('close', () => {
    if (!event.node.res.writableEnded) controller.abort()
  })

  const upstream = await fetch(`${config.llmBaseUrl}/v1/chat/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${config.llmApiKey}`,
    },
    body: JSON.stringify({ model: 'gpt-4o-mini', messages, stream: true }),
    signal: controller.signal,
  })

  if (!upstream.ok || !upstream.body) throw new Error('upstream failed')

  const res = event.node.res
  res.setHeader('Content-Type', 'text/event-stream')
  res.setHeader('Cache-Control', 'no-cache, no-transform')
  res.setHeader('Connection', 'keep-alive')

  return sendStream(event, Readable.fromWeb(upstream.body as any))
})

Without this, a discarded fetch on the client leaves the server streaming to a dead socket until the provider finishes.

Step 6: Lock down production config

Keep secrets in runtimeConfig (server-only). In nuxt.config.ts:

export default defineNuxtConfig({
  runtimeConfig: {
    llmApiKey: process.env.LLM_API_KEY,
    llmBaseUrl: process.env.LLM_BASE_URL || 'https://api.openai.com',
  },
  // Disable compression for the streaming route to avoid buffering
  nitro: {
    compress: false,
  },
})

If you deploy behind a CDN, ensure it does not buffer /api/chat. For Nuxt’s built-in Nitro server, compress: false prevents GZip from holding chunks.

Step 7: Verify the nuxt 3 server routes llm streaming proxy end-to-end

Start the dev server and run a streaming curl:

curl -N -X POST http://localhost:3000/api/chat \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hi in 3 words"}]}'

You should see incremental data: {...} lines, each containing a choices[0].delta.content fragment, ending with data: [DONE].

In the browser, type a prompt in your Chat.vue component. The output ref updates token-by-token. If you see the full response only after a delay, check that no proxy is buffering and that Content-Type: text/event-stream is present in the response headers (use the Network tab).

A nuxt 3 server routes llm streaming proxy also simplifies model switching: change one env var or forward a model field from the client and pass it to the upstream. Because the route speaks plain SSE, any OpenAI-compatible SDK on the client works unchanged.

One last pitfall: Node 18+ exposes Readable.fromWeb, but TypeScript may complain about the stream type. Cast as any only at the boundary; keep the rest of your code typed. If you upgrade Nitro, prefer event.node.res writes only when sendStream does not fit your transform needs—for a pure pass-through, sendStream is the least code that works.