NestJS microservice pattern for multi-provider LLM routing

Most teams bolt LLM provider calls directly into their request handlers and regret it once they need a second model or a fallback path. A clean nestjs microservices llm routing layer separates provider concerns from business logic, gives you one place to enforce quotas, and makes multi-model orchestration testable. Here’s the pattern we ship.

1. Choose the transport and service topology

Keep the external face as a standard HTTP API (NestJS controller) and run the routing logic as an internal microservice over TCP. TCP avoids the JSON-over-HTTP overhead on every intra-cluster call and plays well with NestJS’s built-in ClientProxy.

// main.ts (routing service)
import { NestFactory } from '@nestjs/core';
import { MicroserviceOptions, Transport } from '@nestjs/microservices';
import { RoutingModule } from './routing.module';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<RoutingOptions>(
    RoutingModule,
    {
      transport: Transport.TCP,
      options: { host: '0.0.0.0', port: 4001 },
    },
  );
  await app.listen();
}
bootstrap();

The HTTP gateway calls this service via a ClientTCP instance. This hop is sub-millisecond on localhost and lets you scale routers independently from provider adapters.

2. Define the routing contract

Treat routing as a typed message, not a loose HTTP passthrough. Define a DTO that carries both the inference payload and routing hints.

// llm-request.dto.ts
export interface LlmRequestDto {
  model: string;
  messages: { role: 'system' | 'user' | 'assistant'; content: string }[];
  temperature?: number;
  routing?: {
    prefer?: string[]; // provider ids in priority order
    exclude?: string[];
    cacheTtl?: number;
  };
}

The response should normalize across providers:

export interface LlmResponseDto {
  provider: string;
  model: string;
  content: string;
  usage: { promptTokens: number; completionTokens: number };
  cached?: boolean;
}

This contract is the seam that lets you swap a provider without touching callers.

3. Build the provider registry

A registry maps provider IDs to adapter instances. Adapters implement a common interface so the router stays dumb.

// provider.interface.ts
export interface LlmProvider {
  id: string;
  chat(req: LlmRequestDto): Promise<LlmResponseDto>;
}

// registry.service.ts
@Injectable()
export class RegistryService {
  private providers = new Map<string, LlmProvider>();

  register(p: LlmProvider) {
    this.providers.set(p.id, p);
  }

  get(id: string) | undefined {
    return this.providers.get(id);
  }

  orderedByRequest(req: LlmRequestDto): LlmProvider[] {
    const all = [...this.providers.values()];
    if (!req.routing?.prefer) return all;
    return req.routing.prefer
      .map((id) => this.providers.get(id))
      .filter(Boolean as unknown as (p: LlmProvider) => p is LlmProvider)
      .concat(all.filter((p) => !req.routing!.prefer!.includes(p.id)));
  }
}

Client routing directives (prefer, exclude) are honored here. If you forward to a gateway that already aggregates providers, this same logic works against its single endpoint.

4. Implement fallback and timeouts

Provider APIs fail intermittently. Wrap each attempt in a timeout and cascade to the next provider on rejection or 429-class errors.

// router.service.ts
async route(req: LlmRequestDto): Promise<LlmResponseDto> {
  const candidates = this.registry.orderedByRequest(req)
    .filter((p) => !req.routing?.exclude?.includes(p.id));

  for (const provider of candidates) {
    try {
      return await Promise.race([
        provider.chat(req),
        new Promise((_, rej) => setTimeout(() => rej(new Error('timeout')), 8000)),
      ]);
    } catch (err) {
      // log and continue to next
      this.logger.warn(`provider ${provider.id} failed: ${err.message}`);
    }
  }
  throw new HttpException('all providers failed', 502);
}

If you’d rather not operate the fallback logic yourself, an OpenAI-compatible gateway like n4n.ai handles automatic fallback when a provider is rate-limited or degraded, while still honoring your routing directives. That trades some in-house control for less operational toil.

Tradeoff: cascading increases tail latency. Set a hard budget (e.g., 8s) and return partial results if your product allows.

5. Meter usage and forward cache hints

Per-token metering is non-negotiable for cost control. Capture usage from each adapter response and emit an event the billing service consumes.

// metering.service.ts
@EventPattern('llm.usage')
handleUsage(data: { provider: string; usage: LlmResponseDto['usage'] }) {
  this.redis.incrby(`cost:${data.provider}`, data.usage.promptTokens + data.usage.completionTokens);
}

Provider cache-control hints should pass through. If the client sends routing.cacheTtl, forward it as a header or body field to providers that support prompt caching (e.g., Anthropic, OpenAI). Don’t invent cache behavior; only forward what the underlying API accepts.

{
  "model": "claude-3-5-sonnet",
  "messages": [{"role": "user", "content": "Summarize"}],
  "cache_control": { "ttl": 300 }
}

6. Streaming is a different beast

The TCP request/response pattern above breaks for token streaming. If you need SSE or WebSocket streams, bypass the microservice hop and let the HTTP gateway proxy the stream directly to the chosen provider. Run the router logic synchronously to pick a provider, then return a signed URL or internal address for the gateway to stream from.

Trying to serialize every token over TCP adds latency and wastes CPU on framing. Accept that the routing layer is for decision-making, not bulk transfer.

7. Deploy and observe

Package the routing service as its own container. It has different scaling needs than stateless API servers—CPU light, network bound.

docker build -t llm-router -f Dockerfile.router .
docker run -p 4001:4001 llm-router

Instrument with traces on the route method. You want to see per-provider latency and fallback counts. A simple counter works:

this.metrics.inc('llm_fallback_total', { from: prevId, to: provider.id });

Without this, you’ll silently route 100% to a backup provider after a config typo and blow your budget.

8. Common pitfalls and tradeoffs

Serialization cost. TCP in NestJS uses JSON by default. Large message arrays get re-parsed at every hop. If you pass 32k-token contexts internally, consider a binary transport or keep the router co-located with the gateway.

Provider SDK drift. Each adapter must map provider-specific error shapes to your LlmResponseDto. Don’t lean on a universal SDK; write thin clients. You’ll otherwise spend days waiting for the abstraction library to support a new model.

Routing directive trust. If you honor prefer from end users, a malicious client can pin an expensive provider and exhaust quota. Validate directives against an allowlist per API key.

Metering lag. Emit usage asynchronously but persist within the same process boundary. Losing 0.1% of usage events makes finance unhappy; making the request block on a DB write makes users unhappy.

Test the fallback path. Most teams test the happy path and skip the cascade. Inject a failing mock provider in CI to prove the router actually tries the next one.

The nestjs microservices llm routing pattern is not free—it adds a network hop and a contract to maintain. But once you run three providers and need to shift traffic at 3 a.m. without a deploy, the separation pays for itself.