Structuring an Express.js project for an LLM SaaS backend

A clean express.js project structure llm saas teams can extend without rewrites treats LLM calls as a bounded, swappable dependency rather than inline API requests. The difference shows up the first time you need per-tenant billing, streaming fallbacks, or a second model provider—refactors get expensive fast if those concerns are tangled in route handlers.

1. Define the directory layout before coding routes

Decide the boundaries first. A SaaS backend has three volatile surfaces: HTTP contracts, LLM provider specifics, and tenant accounting. Keep them in separate folders so a provider change doesn’t touch your routers.

src/
  app.js
  routes/
    chat.js
    usage.js
  services/
    llm/
      client.js
      providers/
        openai.js
        anthropic.js
      index.js
    billing.js
  middleware/
    auth.js
    rateLimit.js
  models/
    tenant.js
  config/
    index.js

Routes stay dumb: parse, authorize, call a service, return. Services hold business logic and external calls. Config holds model maps and keys. This layout scales to multiple LLM features (embeddings, chat, moderation) without creating a routes/llm_v2.js mess.

Pitfall: putting fetch to OpenAI directly in routes/chat.js. It works at 2 AM, but you’ll copy-paste it into routes/embed.js by week two.

2. Wrap LLM providers behind a single service interface

Define a minimal base class. Your app code should depend on the abstraction, not the provider SDK.

// services/llm/index.js
export class LLMService {
  async complete(messages, opts) {
    throw new Error('not implemented');
  }
  async *stream(messages, opts) {
    throw new Error('not implemented');
  }
}

A concrete OpenAI-compatible client implements both methods. Use the /chat/completions shape even if you later swap vendors—most gateways mirror it.

// services/llm/providers/openai.js
export class OpenAICompatibleLLM extends LLMService {
  constructor(baseURL, apiKey) {
    this.baseURL = baseURL;
    this.apiKey = apiKey;
  }
  async *stream(messages, opts) {
    const res = await fetch(`${this.baseURL}/chat/completions`, {
      method: 'POST',
      headers: { 'content-type': 'application/json', authorization: `Bearer ${this.apiKey}` },
      body: JSON.stringify({ model: opts.model, messages, stream: true }),
    });
    if (!res.ok) throw new Error(`provider ${res.status}`);
    for await (const chunk of res.body) {
      yield parseSSE(chunk);
    }
  }
}

Tradeoff: an abstraction adds a file, but it deletes a future migration project. Use it.

3. Handle streaming and non-streaming in one route

Clients will want both. Expose a single endpoint and branch on the stream flag.

// routes/chat.js
router.post('/v1/chat', requireApiKey, async (req, res) => {
  const { messages, stream, model } = req.body;
  if (stream) {
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader('Cache-Control', 'no-cache');
    for await (const chunk of llm.stream(messages, { model, tenant: req.tenant })) {
      res.write(`data: ${JSON.stringify(chunk)}\n\n`);
    }
    res.write('data: [DONE]\n\n');
    res.end();
  } else {
    const out = await llm.complete(messages, { model, tenant: req.tenant });
    res.json(out);
  }
});

Streaming holds the connection open. Set a server-side timeout and handle client aborts with req.on('close') to avoid leaking provider tokens. Non-streaming is simpler but blocks the event loop on network only—never on CPU work.

4. Auth and per-tenant rate limiting at the edge

SaaS means multi-tenant from day one. Resolve the tenant in middleware, not in each handler.

// middleware/auth.js
export function requireApiKey(req, res, next) {
  const tenant = tenants.findByKey(req.headers['x-api-key']);
  if (!tenant) return res.status(401).json({ error: 'invalid key' });
  req.tenant = tenant;
  next();
}

Rate limit per tenant, not globally. A token-bucket stored in Redis works:

// middleware/rateLimit.js
export async function rateLimit(req, res, next) {
  const allowed = await redis.consume(`rl:${req.tenant.id}`, 1);
  if (!allowed) return res.status(429).json({ error: 'rate limited' });
  next();
}

Pitfall: trusting req.body.tenantId from the client. The gateway already authenticated the key; derive tenancy from that.

5. Meter usage and persist asynchronously

LLM costs are per token. Capture usage from the provider response and record it. If you route through a gateway such as n4n.ai, it returns per-token usage metering in the standard field, so your service just forwards it.

// services/billing.js
export async function recordUsage(tenantId, usage) {
  await db.usage.insert({ tenantId, prompt: usage.prompt_tokens, completion: usage.completion_tokens, at: Date.now() });
}

Call it after the LLM returns, but don’t block the response:

const out = await llm.complete(messages, opts);
setImmediate(() => recordUsage(req.tenant.id, out.usage).catch(() => {}));
res.json(out);

Tradeoff: fire-and-forget metering can drop on crash. For stricter accounting, push to a durable queue (e.g., SQS) instead of setImmediate.

6. Build fallback and routing into the service layer

Providers fail with 429 or 503. Your LLMService should try a secondary without the route knowing.

// services/llm/index.js
export class FallbackLLM extends LLMService {
  constructor(primary, secondary) { super(); this.primary = primary; this.secondary = secondary; }
  async complete(messages, opts) {
    try { return await this.primary.complete(messages, opts); }
    catch (e) {
      if (e.status === 429 || e.status === 503) return await this.secondary.complete(messages, opts);
      throw e;
    }
  }
}

A gateway such as n4n.ai handles automatic fallback when a provider is rate-limited or degraded and honors client routing directives, which lets your Express service stay thin—just pass the model hint and let the endpoint resolve. Either way, keep the fallback logic out of the router.

7. Mock LLMs in tests

Dependency injection makes unit tests trivial.

class FakeLLM extends LLMService {
  async *stream() { yield { delta: 'hi' }; }
}

Mount your router with app.use('/', chatRouter) and pass new FakeLLM() via app.locals.llm. Test the streaming route with supertest and assert on SSE lines. Skip real provider calls in CI.

Common pitfalls to avoid

• Schema drift: validate req.body with Zod. LLM APIs change field names; catch it at the edge.
• Prompt building in routes: keep system prompts and templating in services/llm/prompts.js.
• No backpressure handling: on stream, if the client is slow, res.write buffers unbounded. Use res.flushHeaders() and check res.writable.
• Plaintext prompt logs: tenant data leaks via verbose error middleware. Redact before logging.
• Ignoring cache-control: forward provider cache-control hints to clients if you proxy; it saves tokens on repeated calls.

An express.js project structure llm saas teams respect is boring on purpose: clear folders, thin routes, swappable LLM service, and metering that doesn’t block. Build it that way and adding a new model or a second provider becomes a one-file change, not a sprint.