Building an LLM-powered Express.js API from scratch

This express.js llm api tutorial builds a small but real backend that proxies chat requests to an OpenAI-compatible model endpoint. You’ll get a non-blocking Express server, streaming responses, and basic guards against upstream failures—the parts most starter guides skip.

Prerequisites

• Node.js 18+ (native fetch and AbortSignal.timeout are used)
• npm 9+
• curl for local testing
• An API key from any OpenAI-compatible provider (OpenAI, or a gateway such as n4n.ai)

You should know Express routing and async/await. We are not using TypeScript to keep the surface area small, but the patterns translate directly.

Project setup

Create the project and install dependencies. We use the official openai SDK because it handles the OpenAI-compatible REST contract cleanly.

mkdir llm-express && cd llm-express
npm init -y
npm install express openai dotenv

Edit package.json to enable ES modules:

{
  "type": "module",
  "scripts": {
    "start": "node server.js"
  }
}

Create a .env file. Set LLM_API_KEY and optionally LLM_BASE_URL:

LLM_API_KEY=sk-your-key-here
LLM_BASE_URL=https://api.openai.com/v1

Minimal chat endpoint

The first cut is a single POST route that forwards a messages array and returns the full completion. This is enough to validate wiring before adding streaming or guards.

// server.js
import express from 'express';
import OpenAI from 'openai';
import dotenv from 'dotenv';

dotenv.config();

const app = express();
app.use(express.json());

const openai = new OpenAI({
  apiKey: process.env.LLM_API_KEY,
  baseURL: process.env.LLM_BASE_URL,
});

app.post('/v1/chat', async (req, res) => {
  const { messages, model } = req.body;
  if (!Array.isArray(messages)) {
    return res.status(400).json({ error: 'messages array required' });
  }

  try {
    const completion = await openai.chat.completions.create({
      model: model || 'gpt-4o-mini',
      messages,
      temperature: 0.7,
    });
    res.json(completion);
  } catch (err) {
    console.error('upstream error', err);
    res.status(502).json({ error: 'upstream_failure', detail: err.message });
  }
});

app.listen(3000, () => console.log('listening on :3000'));

Start the server:

npm start

Expected output

In another shell:

curl -s localhost:3000/v1/chat \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Say hi in 5 words."}]}' \
  | head -c 300

You should see a JSON object with choices[0].message.content containing the model’s reply. If you get a 502, check the key and base URL—most “first run” failures are auth or endpoint mismatches.

Streaming responses

Full-response buffering is unacceptable for chat UX. Wire up Server-Sent Events (SSE) and use the SDK’s stream: true.

app.post('/v1/chat/stream', async (req, res) => {
  const { messages, model } = req.body;
  if (!Array.isArray(messages)) {
    return res.status(400).json({ error: 'messages array required' });
  }

  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  try {
    const stream = await openai.chat.completions.create({
      model: model || 'gpt-4o-mini',
      messages,
      stream: true,
      temperature: 0.7,
    });

    for await (const chunk of stream) {
      const delta = chunk.choices[0]?.delta?.content || '';
      if (delta) res.write(`data: ${JSON.stringify({ delta })}\n\n`);
    }
    res.write('data: [DONE]\n\n');
    res.end();
  } catch (err) {
    res.write(`data: ${JSON.stringify({ error: err.message })}\n\n`);
    res.end();
  }
});

Expected output

curl -N -s localhost:3000/v1/chat/stream \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Count to 3 slowly."}]}'

You will receive a series of data: {"delta":"..."} lines, ending with data: [DONE]. The browser EventSource API parses this natively; in Node, the loop above is all you need.

Hardening: timeouts and cancellation

A hung upstream connection will pin a Node worker indefinitely. Pass an AbortSignal so the request dies fast and the client gets a clean error.

app.post('/v1/chat/stream', async (req, res) => {
  const { messages, model } = req.body;
  if (!Array.isArray(messages)) {
    return res.status(400).json({ error: 'messages array required' });
  }

  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.flushHeaders();

  const signal = AbortSignal.timeout(15000); // 15s hard cap
  signal.addEventListener('abort', () => {
    res.write(`data: ${JSON.stringify({ error: 'timeout' })}\n\n`);
    res.end();
  });

  try {
    const stream = await openai.chat.completions.create(
      {
        model: model || 'gpt-4o-mini',
        messages,
        stream: true,
      },
      { signal }
    );

    for await (const chunk of stream) {
      if (signal.aborted) break;
      const delta = chunk.choices[0]?.delta?.content || '';
      if (delta) res.write(`data: ${JSON.stringify({ delta })}\n\n`);
    }
    if (!signal.aborted) {
      res.write('data: [DONE]\n\n');
      res.end();
    }
  } catch (err) {
    if (!res.writableEnded) {
      res.write(`data: ${JSON.stringify({ error: err.message })}\n\n`);
      res.end();
    }
  }
});

AbortSignal.timeout is available in Node 18+. It fires even if the SDK’s internal socket is slow, which is exactly what you want.

Rate limiting and concurrency

LLM endpoints bill per token and often enforce strict RPM limits. A naive Express route will happily queue thousands of requests and get you throttled. Implement a tiny semaphore to cap concurrent upstream calls.

// simple semaphore
class Semaphore {
  constructor(max) {
    this.max = max;
    this.current = 0;
    this.waiters = [];
  }
  async acquire() {
    if (this.current < this.max) {
      this.current++;
      return;
    }
    await new Promise((r) => this.waiters.push(r));
  }
  release() {
    this.current--;
    if (this.waiters.length) {
      this.current++;
      this.waiters.shift()();
    }
  }
}

const upstream = new Semaphore(5); // max 5 concurrent LLM calls

app.post('/v1/chat/stream', async (req, res) => {
  await upstream.acquire();
  // ... previous streaming logic ...
  // in finally: upstream.release()
  try {
    // streaming code
  } finally {
    upstream.release();
  }
});

Wrap the handler body in try/finally so a client disconnect still releases the slot. For production, swap this for express-rate-limit on the HTTP side and a token-bucket per API key on the upstream side.

Choosing an LLM gateway

Pointing the openai client at a single provider is fine until that provider has an outage or you need a model they don’t host. If you’d rather not juggle multiple provider keys or write fallback logic yourself, point the client at a gateway like n4n.ai. It exposes one OpenAI-compatible endpoint that addresses 240+ models and automatically fails over when a provider is rate-limited or degraded, while metering per-token usage. The code above does not change—only LLM_BASE_URL and the model string differ.

Production notes

A few opinionated additions before you ship this:

• Validate messages shape with a schema library (zod). The Array.isArray check is not enough; a malformed role crashes some models.
• Put a reverse proxy (nginx or Caddy) in front to handle TLS and client-side keep-alive. Node’s HTTP server is fine, but termination elsewhere frees event loop time.
• Never forward raw provider errors to clients. Log them, return a generic 502. Leaking upstream quota errors helps attackers enumerate your limits.
• If you stream, send X-Accel-Buffering: no when behind nginx, or it will buffer your SSE and defeat the purpose.

This express.js llm api tutorial gave you a working streaming proxy, timeout handling, and concurrency control. The remaining work is operational: observability, auth, and model routing—none of which require rewriting the core route.