gRPC for internal LLM microservices, REST for public APIs

Most teams hit a wall when they expose a single API surface for both backend services and external developers. The pragmatic split is gRPC for internal microservices and REST for public LLM API consumption: low-latency typed contracts inside your boundary, familiar HTTP/JSON at the edge. This guide lays out an ordered path to implement that split without rebuilding your stack.

Why the transport split matters

Internal LLM microservices need tight latency budgets, bidirectional streaming, and schema evolution without version hell. REST over HTTP/1.1 with JSON adds serialization overhead on every hop when you fan out to dozens of model workers. Public developers, however, expect curl-able endpoints, OpenAPI docs, and no codegen step. A grpc internal rest public llm api boundary respects both: optimize for machines inside, humans outside.

The moment you have a router service, a tokenization service, and a worker pool, JSON parsing between them becomes pure tax. Protobuf avoids the parsing overhead of JSON and lets the compiler catch mismatches at build time.

Step 1: Define internal contracts with protobuf

Start by writing a .proto that models your inference request, not the provider’s API. Keep fields explicit: model slug, max_tokens, timeout, routing hints. Avoid nesting provider-specific structs; use a map for directives.

syntax = "proto3";

package inference.v1;

service Router {
  rpc Generate (GenerateRequest) returns (stream GenerateResponse);
}

message GenerateRequest {
  string model = 1;
  string prompt = 2;
  int32 max_tokens = 3;
  map<string, string> routing = 4; // client routing directives
}

message GenerateResponse {
  string token = 1;
  bool cached = 2; // provider cache-control hint echoed back
}

Generate Python stubs with python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. inference.proto. Never let internal services call provider SDKs directly; the proto is your seam.

Pitfall: versioning the stream

Adding a field to GenerateResponse is safe. Changing the stream semantics (e.g., switching from token stream to full completion) breaks every internal client. Use a new RPC method like GenerateV2 instead of mutating the contract.

Step 2: Implement gRPC services for orchestration

Stand up a Router service that dispatches to model workers. Use async Python or Go to handle concurrent streams. The service should read routing and set gRPC metadata for downstream workers.

import grpc
from concurrent import futures
import inference_pb2, inference_pb2_grpc

class Router(inference_pb2_grpc.RouterServicer):
    def Generate(self, request, context):
        # honor routing directives from map
        provider = request.routing.get("provider", "auto")
        context.set_metadata("x-provider", provider)
        if provider == "anthropic":
            yield inference_pb2.GenerateResponse(token="[routed]", cached=False)
        # ... fan out to worker pool over another gRPC call
        yield inference_pb2.GenerateResponse(token="hello", cached=True)

server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
inference_pb2_grpc.add_RouterServicer_to_server(Router(), server)
server.add_insecure_port("[::]:50051")
server.start()
server.wait_for_termination()

Internal workers speak only this contract. They can call external providers using their own SDKs, but the boundary stays typed.

Tradeoff: debugging binary payloads

grpcurl helps, but you lose the ease of reading JSON in logs. Emit structured logs with the proto fields decoded; don’t rely on wire captures. Add a debug interceptor that dumps the request as JSON in staging only.

Step 3: Expose a REST gateway for public clients

Your public surface should be REST, ideally OpenAI-compatible so existing SDKs work. A thin FastAPI app translates HTTP/JSON to the internal gRPC call and streams back Server-Sent Events.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import grpc
import inference_pb2, inference_pb2_grpc

app = FastAPI()
channel = grpc.insecure_channel("localhost:50051")
stub = inference_pb2_grpc.RouterStub(channel)

@app.post("/v1/chat/completions")
async def chat(req: dict):
    grpc_req = inference_pb2.GenerateRequest(
        model=req["model"],
        prompt=req["messages"][-1]["content"],
        max_tokens=req.get("max_tokens", 256),
        routing={"provider": req.get("provider", "auto")}
    )
    def event_stream():
        for resp in stub.Generate(grpc_req):
            yield f"data: {resp.token}\n\n"
    return StreamingResponse(event_stream(), media_type="text/event-stream")

This gives external devs a public LLM API they can hit with curl. If you want to skip building fallback and provider aggregation yourself, an inference gateway like n4n.ai provides one OpenAI-compatible endpoint that addresses 240+ models with automatic fallback when a provider is rate-limited, while your internal fan-out stays on gRPC.

Pitfall: hiding streaming

REST clients expect SSE or chunked responses. Don’t buffer the full completion internally and return one JSON blob; proxy the gRPC stream directly. Buffering also defeats the latency advantage of token streaming.

Step 4: Honor routing directives and cache hints

Internal gRPC metadata carries routing keys (x-provider, x-region). The REST layer must forward them as proto routing map or metadata. When a provider returns cache hits, echo that as a response header (x-cache: HIT) so public clients can tune prompts.

@app.post("/v1/generate")
async def generate(req: dict):
    md = [("x-provider", req.get("provider", "auto"))]
    call = stub.Generate.with_call(inference_pb2.GenerateRequest(...), metadata=md)
    # call[1].trailing_metadata() may carry cache-control
    cached = dict(call[1]).get("x-cache") == "HIT"
    return {"text": "...", "cached": cached}

If you use a gateway, it already honors client routing directives and forwards provider cache-control hints, saving you the plumbing. The grpc internal rest public llm api pattern works best when the edge is dumb and the core is smart.

Step 5: Meter usage per token

Public APIs need billing. Capture token counts from the gRPC responses and write to a ledger. Per-token usage metering is non-negotiable for cost control.

# inside Generate stream consumer
used = 0
for resp in stub.Generate(req):
    used += 1  # approximate; real impl uses tokenizer
log_usage(api_key, used)

Don’t trust client-reported counts. The internal service is the source of truth. Add idempotency keys to the REST request so retries don’t double-charge; dedupe on the gRPC boundary using a seen-set with TTL.

Step 6: Test the boundary like a hostile client

Write contract tests that send malformed protobuf via grpcurl and unexpected JSON via httpie. Verify the gateway returns 422 on missing model and the gRPC service returns INVALID_ARGUMENT.

Run a chaos test: kill the worker pool and confirm the Router yields a proper UNAVAILABLE status, not a silent hang. Public REST clients should see a 503 with Retry-After, not an empty 200.

Tradeoffs and when not to split

The split adds a translation layer; for a single-service prototype it’s overhead. If you have fewer than three internal services and no external developers, a REST-only app is fine. Once you run separate router, worker, and eval services, gRPC internal rest public llm api separation pays off.

gRPC gives you backpressure and typed streams; REST gives you reach. Keep the proto stable, the gateway thin, and the metering honest. Skip the split only if your total request volume is trivial and your team is one person.