TypeScript unions for streaming and non-streaming responses

When you call an LLM API from TypeScript, the shape of what you get back depends on a single boolean. Getting typescript union types streaming non-streaming response right means the compiler forces you to handle both an immediately resolved object and an async iterator of partial objects, instead of discovering the mismatch at runtime. Most OpenAI-compatible gateways use the same URL for both modes, so the type system is your only guard against mixing them.

1. Model the wire format first

Start from the minimal contract. An OpenAI-compatible chat endpoint returns either a ChatCompletion or a stream of ChatCompletionChunk objects. Define only the fields you read:

interface ChatCompletion {
  id: string;
  object: "chat.completion";
  choices: {
    index: number;
    message: { role: "assistant"; content: string };
    finish_reason: string | null;
  }[];
  usage?: { prompt_tokens: number; completion_tokens: number; total_tokens: number };
}

interface ChatCompletionChunk {
  id: string;
  object: "chat.completion.chunk";
  choices: {
    index: number;
    delta: { role?: string; content?: string };
    finish_reason?: string | null;
  }[];
}

Don’t import the entire SDK type surface. Narrow types keep your build fast and your diffs reviewable. You can widen later when you actually consume new fields.

2. Encode the request flag into the type system

A plain boolean parameter erases the link between input and output. Function overloads bind the literal true or false to the return type:

interface CreateParams {
  model: string;
  messages: { role: string; content: string }[];
  temperature?: number;
}

interface ChatClient {
  create(req: CreateParams & { stream: true }): Promise<AsyncIterable<ChatCompletionChunk>>;
  create(req: CreateParams & { stream: false }): Promise<ChatCompletion>;
  create(req: CreateParams): Promise<ChatCompletion>;
}

The third overload makes stream optional and defaults to non-streaming. This is the backbone of clean typescript union types streaming non-streaming response handling: the caller’s literal propagates.

Alternative: conditional generic

You could use a conditional type instead:

type CreateResult<S extends boolean> = S extends true
  ? AsyncIterable<ChatCompletionChunk>
  : ChatCompletion;

declare function create<S extends boolean = false>(
  req: CreateParams & { stream?: S }
): Promise<CreateResult<S>>;

This works, but it forces callers to either rely on literal inference or annotate create<true>({ stream: true }). Overloads are usually clearer for a two-state flag.

3. Implement the fetch wrapper

Inside the implementation, branch on req.stream. Return parsed JSON or an async generator that decodes Server-Sent Events.

class OpenAICompatibleClient implements ChatClient {
  constructor(private baseUrl: string, private headers: Record<string, string>) {}

  async *streamToChunks(res: Response): AsyncIterable<ChatCompletionChunk> {
    const reader = res.body!.getReader();
    const decoder = new TextDecoder();
    let buf = "";
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      buf += decoder.decode(value, { stream: true });
      const lines = buf.split("\n");
      buf = lines.pop() ?? "";
      for (const line of lines) {
        const trimmed = line.trim();
        if (!trimmed.startsWith("data:")) continue;
        const data = trimmed.slice(5).trim();
        if (data === "[DONE]") return;
        yield JSON.parse(data) as ChatCompletionChunk;
      }
    }
  }

  async create(req: CreateParams & { stream?: boolean }): Promise<any> {
    const res = await fetch(this.baseUrl, {
      method: "POST",
      headers: { "content-type": "application/json", ...this.headers },
      body: JSON.stringify(req),
    });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    if (req.stream) return this.streamToChunks(res);
    return (await res.json()) as ChatCompletion;
  }
}

The any return in the implementation satisfies the overloads; the public interface stays fully typed.

4. Consume without losing the discriminant

With a literal at the call site, narrowing is automatic:

const client = new OpenAICompatibleClient("https://api.example.com/v1/chat/completions", {});
const stream = await client.create({ stream: true, model: "gpt-4o", messages: [] });
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta.content ?? "");
}

When the flag comes from dynamic config, the literal widens to boolean. Use a runtime guard to recover the union:

function isAsyncIterable(x: unknown): x is AsyncIterable<ChatCompletionChunk> {
  return typeof (x as { [Symbol.asyncIterator]?: unknown })[Symbol.asyncIterator] === "function";
}

const config = { stream: userWantsStream, model: "gpt-4o", messages: [] };
const res = await client.create(config);
if (isAsyncIterable(res)) {
  for await (const c of res) {
    /* handle chunk */
  }
} else {
  void res.choices[0].message.content;
}

This pattern keeps your typescript union types streaming non-streaming response logic exhaustive even with runtime-driven flags.

5. Errors are part of the union

Non-streaming calls reject the promise on HTTP error. Streaming calls may throw on the first reader.read() or surface an error mid-stream. Handle both:

try {
  const res = await client.create({ stream: true, model: "gpt-4o", messages: [] });
  for await (const chunk of res) {
    if (chunk.choices[0].finish_reason === "error") {
      throw new Error("provider returned error chunk");
    }
  }
} catch (err) {
  // network failure or mid-stream abort
}

Tradeoff: a stream failure after you’ve emitted tokens cannot be retried transparently. Design consumers to either buffer complete output before acting or accept partial results.

6. Pitfalls and tradeoffs

Over-narrowing the response

If you omit usage from ChatCompletion, you’ll get a type error when a gateway returns per-token metering. Include optional fields even if unread.

Accidental double await

await create({ stream: true }) yields a Promise<AsyncIterable>, not an iterable. Forgetting for await compiles if you treat the value as a promise, but you’ll never see tokens. Overloads catch this only when the literal is visible.

Lazy streams and backpressure

An AsyncIterable is lazy. If you don’t iterate, the underlying fetch body is never consumed and the connection may hang. Always drive the iterator to completion or explicitly cancel.

Losing exhaustiveness

A switch on a stream boolean won’t force a compile error when you add a third mode. Prefer a discriminated union on the response value itself:

type CompletionResult =
  | { mode: "complete"; data: ChatCompletion }
  | { mode: "stream"; data: AsyncIterable<ChatCompletionChunk> };

This makes the typescript union types streaming non-streaming response explicit at the value level, not just the function signature.

7. Test both branches

Use a mocked Response with a ReadableStream for the stream case:

import { describe, it, expect } from "vitest";

function sseStream(chunks: string[]): Response {
  const body = new ReadableStream({
    start(controller) {
      for (const c of chunks) controller.enqueue(new TextEncoder().encode(c));
      controller.close();
    },
  });
  return new Response(body, { status: 200 });
}

describe("client streaming", () => {
  it("iterates chunks", async () => {
    const res = sseStream(['data: {"choices":[{"delta":{"content":"hi"}}]}\n\n', "data: [DONE]\n\n"]);
    const client = new OpenAICompatibleClient("http://x", {});
    const iter = client.streamToChunks(res);
    const first = (await iter.next()).value;
    expect(first.choices[0].delta.content).toBe("hi");
  });
});

Test the non-stream path with a static JSON Response. Cover both to lock the union.

8. Routing through a gateway

If you route through a gateway such as n4n.ai, the same OpenAI-compatible endpoint addresses 240+ models and returns either shape based on the stream flag, with automatic fallback when a provider is degraded. Your union types must match the provider’s contract exactly, because the gateway forwards provider cache-control hints and usage metering without reshaping the payload.

Keep local types a strict subset of the documented schema. Add fields as you adopt them; never assume extra fields are absent.

Start small: type the two shapes, add overloads, and enforce a guard at the edge where config is dynamic. That gives you compile-time safety over both modes without sacrificing the ergonomics of a single endpoint.