Why most LLM APIs use REST instead of gRPC

The question of why llm apis use rest not grpc comes down to ecosystem gravity and the specific mechanics of inference workloads. REST—usually JSON over HTTP with Server-Sent Events for streaming—won because it matches how models are called from browsers, notebooks, and polyglot backends, not because it is technically superior in a vacuum.

The request profile of inference

Most LLM calls are unary-ish: send a prompt, get a completion. Even with streaming, the client opens one HTTP request and reads a stream of tokens. Here is the typical REST shape:

curl https://api.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Explain gRPC."}],
    "stream": true
  }'

The response is a sequence of SSE frames:

{"choices":[{"delta":{"content":"g"}}]}
{"choices":[{"delta":{"content":"RPC"}}]}

This is dead simple. Any language with an HTTP client can call it. That is a large part of why llm apis use rest not grpc: the client side is zero-config.

Strict contracts vs. fast-moving schemas

gRPC forces a Protobuf contract upfront. You define services, messages, and fields with numeric tags. That is great for stable internal systems. LLM APIs, however, mutate monthly: new sampling parameters, tool-calling schemas, response formats, and provider-specific extensions.

Adding a field in Protobuf requires regenerating stubs for every client. In JSON, you add a key and old clients ignore it.

// Hypothetical gRPC definition
service LLM {
  rpc Complete(CompletionRequest) returns (stream CompletionChunk);
}
message CompletionRequest {
  string model = 1;
  repeated Message messages = 2;
  float temperature = 3;
  // added later: optional string response_format = 4;
}

With REST, the same evolution is just:

{
  "model": "gpt-4o-mini",
  "messages": [{"role": "user", "content": "Hi"}],
  "response_format": {"type": "json_object"}
}

Old SDKs skip the unknown key. The server ignores missing keys. No codegen step. This flexibility is a quiet reason why llm apis use rest not grpc in public surfaces.

Streaming: SSE vs gRPC streams

Token generation is inherently a stream. gRPC supports server-streaming RPCs natively, with binary framing and backpressure. But the dominant consumer is a browser or a Python script. Browsers cannot speak raw HTTP/2 gRPC; they need grpc-web, which adds a proxy and a JS library. SSE, by contrast, is native to fetch and EventSource.

const res = await fetch("/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ model: "gpt-4o-mini", messages: [{role:"user",content:"Stream"}], stream: true })
});
const reader = res.body!.getReader();
// read chunks, parse SSE

gRPC-streaming would require generated client stubs and a websocket-like transport. For a public API, that friction kills adoption.

Tooling and language reach

REST is just HTTP. Every language ships an HTTP client, often in stdlib. gRPC needs protoc, language plugins, and generated code. In a world where an LLM app might be written in Go, Ruby, Swift, and Node in the same week, REST removes a build step.

Consider a Python caller:

import requests
r = requests.post("https://api.example.com/v1/chat/completions",
                  json={"model":"gpt-4o-mini","messages":[{"role":"user","content":"Hi"}]},
                  stream=True)
for line in r.iter_lines():
    if line: print(line)

The gRPC equivalent needs a compiled _pb2.py and _pb2_grpc.py. That is fine inside one org; it is friction across thousands of external developers.

Where gRPC actually earns its keep

I am not arguing gRPC is useless. For internal service-to-service communication inside an inference platform, gRPC is often the right call. Binary encoding cuts payload size, HTTP/2 multiplexing reduces connection overhead, and strict schemas catch bugs at compile time.

An inference gateway such as n4n.ai exposes one OpenAI-compatible REST endpoint across 240+ models; its internal routing between aggregator and provider adapters may well use gRPC or similar binary transports, but the edge is REST because that is what client SDKs expect. The split is pragmatic: REST at the boundary, efficiency inside.

Header semantics and caching

REST uses HTTP headers for routing, auth, and cache hints. An OpenAI-compatible gateway can forward x-route or cache-control directives to providers with zero extra protocol machinery. With gRPC, you pack metadata into a separate map, which is functional but less inspectable in browser devtools.

curl https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "X-Route: provider-a" \
  -H "Cache-Control: max-age=3600" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Cached?"}]}'

This transparency is a small but real factor in why llm apis use rest not grpc.

The latency myth

A common claim is that gRPC is faster, so LLM APIs should use it. Measure the phases: network RTT, request serialization, queue time, time-to-first-token (TTFT), and generation. Serialization of a 1KB JSON prompt vs a protobuf message is sub-millisecond. TTFT is hundreds of milliseconds to seconds. Generation is seconds. Transport efficiency is lost in the noise.

If you are shipping a public API, optimizing the serialization format is premature. The bottleneck is the model, not the wire.

Versioning and breaking changes

Protobuf has a philosophy: never break the contract. You add fields, you don’t remove. But LLM providers do rename, deprecate, and restructure (e.g., the shift from prompt to messages). With REST, you bump the URL path (/v1/, /v2/) and both coexist. With gRPC, you either stand up a new service or maintain parallel message types.

This is another facet of why llm apis use rest not grpc: URL-based versioning is trivial and visible.

Browser and edge constraints

Cloudflare Workers, Vercel Edge, and browser extensions all speak fetch. They do not speak HTTP/2 gRPC without extra layers. LLM apps are increasingly edge-deployed to reduce latency. Forcing grpc-web through a proxy negates that benefit.

Decision matrix

Use REST/HTTP+JSON when:

• The API is public or cross-team.
• Clients include browsers, scripts, and unknown languages.
• Streaming via SSE is sufficient.
• Schema evolves quickly.

Use gRPC when:

• Both ends are owned by you.
• You need strict contracts and codegen.
• High-frequency internal calls justify binary savings.
• Bidirectional streaming is required (rare for LLM inference).

Takeaway

The answer to why llm apis use rest not grpc is not about raw performance; it is about reach, evolution, and the reality of token streaming over the open web. REST with SSE is good enough at the boundary and universally supported. Adopt gRPC only where you control the stack and can harvest its efficiencies. Build your public LLM interface as OpenAI-compatible REST, and keep gRPC for the plumbing.