n4nAI
Open console →
API integrationTutorialgRPC vs REST for LLM APIs

Building a gRPC wrapper around a REST-based LLM API

Step-by-step tutorial for building a gRPC wrapper around a REST-based LLM API using Python, with protobuf definitions and runnable example code.

n4n Team3 min read566 words

Audio narration

Coming soon — every post will get a voice note here.

Most LLM providers ship HTTP/JSON endpoints, but internal services often want typed, low-latency gRPC interfaces. This tutorial builds a grpc wrapper rest llm api in Python that translates protobuf requests into OpenAI-compatible REST calls and streams responses back. You will end up with a runnable server and client, plus the patterns needed to extend it to streaming and production error handling.

Prerequisites

  • Python 3.11 or newer.
  • Install dependencies: pip install grpcio grpcio-tools requests.
  • An OpenAI-compatible REST endpoint and API key. Any provider works; n4n.ai exposes a single OpenAI-compatible REST endpoint covering 240+ models with automatic fallback when a provider is rate-limited or degraded, which makes it a convenient target for this exercise.
  • No standalone protoc install needed; grpcio-tools bundles it.

Set environment variables before running the code:

export LLM_REST_ENDPOINT="https://api.n4n.ai/v1/chat/completions"
export LLM_API_KEY="sk-your-key"

Why build a grpc wrapper rest llm api

REST is universal, but inside a microservice mesh you pay for JSON parsing, lack of typed contracts, and weaker streaming primitives. A grpc wrapper rest llm api lets you keep the REST surface for external clients while giving internal callers a generated client, bidirectional streaming, and protocol-buffer efficiency. The wrapper itself is thin: it maps fields, forwards auth, and converts errors.

Define the protobuf contract

Create chat.proto. We keep the message shape close to the OpenAI schema to minimize mapping logic.

syntax = "proto3";

package llm;

service Chat {
  rpc Complete (ChatRequest) returns (ChatResponse);
}

message Message {
  string role = 1;
  string content = 2;
}

message ChatRequest {
  string model = 1;
  repeated Message messages = 2;
  float temperature = 3;
}

message ChatResponse {
  string id = 1;
  string model = 2;
  string content = 3;
  uint32 prompt_tokens = 4;
  uint32 completion_tokens = 5;
}

The repeated Message field maps directly to the messages array in the REST body. Token counts are surfaced explicitly so callers can meter usage without re-parsing provider JSON.

Generate Python stubs

Run:

python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. chat.proto

This emits chat_pb2.py (message classes) and chat_pb2_grpc.py (service stubs). Import them as shown later.

Implement the REST client

Write rest_client.py. It performs a single POST and returns the parsed JSON.

import os
import requests

ENDPOINT = os.environ["LLM_REST_ENDPOINT"]
API_KEY = os.environ["LLM_API_KEY"]

def rest_chat_complete(model: str, messages: list[dict], temperature: float) -> dict:
    resp = requests.post(
        ENDPOINT,
        headers={
            "Authorization": f"Bearer {API_KEY}",
            "Content-Type": "application/json",
        },
        json={"model": model, "messages": messages, "temperature": temperature},
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()

If the REST layer supports provider cache-control hints, they arrive as response headers; the wrapper can log or forward them as gRPC metadata.

Implement the gRPC server

The server receives a ChatRequest, converts it to dicts, calls the REST client, and packs the result into ChatResponse.

import grpc
from concurrent import futures
import chat_pb2
import chat_pb2_grpc
from rest_client import rest_chat_complete

class ChatServicer(chat_pb2_grpc.ChatServicer):
    def Complete(self, request, context):
        messages = [{"role": m.role, "content": m.content} for m in request.messages]
        try:
            data = rest_chat_complete(request.model, messages, request.temperature)
        except requests.HTTPError as e:
            context.set_code(grpc.StatusCode.UNAVAILABLE)
            context.set_details(f"REST upstream failed: {e}")
            return chat_pb2.ChatResponse()
        choice = data["choices"][0]["message"]["content"]
        usage = data.get("usage", {})
        return chat_pb2.ChatResponse(
            id=data.get("id", ""),
            model=data.get("model", request.model),
            content=choice,
            prompt_tokens=usage.get("prompt_tokens", 0),
            completion_tokens=usage.get("completion_tokens", 0),
        )

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    chat_pb2_grpc.add_ChatServicer_to_server(ChatServicer(), server)
    server.add_insecure_port("[::]:50051")
    server.start()
    print("server on :50051")
    server.wait_for_termination()

if __name__ == "__main__":
    serve()

The grpc wrapper rest llm api server runs the REST call synchronously in a thread pool. For higher throughput, swap requests for an async HTTP client and use grpc.aio.

Run and verify

Terminal 1:

python server.py

Terminal 2, write client.py:

import grpc
import chat_pb2
import chat_pb2_grpc

channel = grpc.insecure_channel("localhost:50051")
stub = chat_pb2_grpc.ChatStub(channel)

req = chat_pb2.ChatRequest(
    model="gpt-4o-mini",
    messages=[chat_pb2.Message(role="user", content="Return only the JSON {\"ok\":true}")],
    temperature=0.0,
)
resp = stub.Complete(req)
print("content:", resp.content)
print(f"tokens: {resp.prompt_tokens} prompt + {resp.completion_tokens} completion")

Expected output:

content: {"ok":true}
tokens: 15 prompt + 7 completion

If you see a UNAVAILABLE error, check that LLM_REST_ENDPOINT and LLM_API_KEY are set and the upstream is reachable.

Extend to streaming

REST LLM APIs commonly stream via server-sent events (SSE). Add a server-streaming RPC to the proto:

service Chat {
  rpc Complete (ChatRequest) returns (ChatResponse);
  rpc Stream (ChatRequest) returns (stream ChatResponse);
}

Regenerate stubs. Modify the REST client to accept a stream=True and parse SSE lines:

import json

def rest_chat_stream(model, messages, temperature):
    with requests.post(
        ENDPOINT,
        headers={"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"},
        json={"model": model, "messages": messages, "temperature": temperature, "stream": True},
        timeout=30,
        stream=True,
    ) as resp:
        resp.raise_for_status()
        for line in resp.iter_lines():
            if not line or not line.startswith(b"data:"):
                continue
            payload = line[5:].strip()
            if payload == b"[DONE]":
                break
            yield json.loads(payload)

In the servicer, implement Stream by yielding a ChatResponse per delta:

def Stream(self, request, context):
    messages = [{"role": m.role, "content": m.content} for m in request.messages]
    for chunk in rest_chat_stream(request.model, messages, request.temperature):
        delta = chunk["choices"][0]["delta"].get("content", "")
        if delta:
            yield chat_pb2.ChatResponse(content=delta, model=request.model)

The grpc wrapper rest llm api now supports token-by-token delivery over a single gRPC stream. A client calls stub.Stream(req) and iterates.

Map errors and metadata

Production wrappers must translate HTTP status to gRPC codes. Use an interceptor or handle in each method:

except requests.HTTPError as e:
    if e.response.status_code == 429:
        context.set_code(grpc.StatusCode.RESOURCE_EXHAUSTED)
    else:
        context.set_code(grpc.StatusCode.UNAVAILABLE)
    context.set_details(str(e))

If your upstream honors client routing directives—n4n.ai forwards provider cache-control hints and respects routing headers—propagate incoming gRPC metadata as HTTP headers so callers retain control over model selection and caching.

Testing with grpcurl

You can inspect the service without writing a client:

grpcurl -plaintext localhost:50051 llm.Chat/Complete

Provide the JSON request matching the proto. This validates the wrapper independently of your application code.

Closing notes

The pattern above keeps the REST LLM API as the source of truth while giving internal teams a typed gRPC surface. Extend the proto with embeddings or function-calling as needed, and reuse the same REST client core. Because the wrapper is stateless, you can deploy multiple instances behind a normal gRPC load balancer.

Tagsgrpcresttutorialwrapper

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationDoes gRPC reduce LLM API latency versus RESTAnalyze whether gRPC cuts LLM API latency versus REST. We break down the real bottlenecks, streaming, and where gRPC helps server-to-server but not client calls.API integrationgRPC for internal LLM microservices, REST for public APIsPractical guide to LLM infrastructure: adopt gRPC for internal microservices and REST for public LLM APIs, with proto snippets, tradeoffs, and pitfalls.API integrationgRPC vs REST for LLM APIs: what changes at scaleAnalyzes gRPC vs REST for LLM APIs at scale: latency, streaming, codegen, and operational tradeoffs, with a decisive recommendation for builders.API integrationgRPC vs REST: latency benchmarks for LLM API callsA hands-on analysis of gRPC vs REST latency benchmarks for LLM API calls, with real code, tradeoffs, and a decisive recommendation for builders.API integrationgRPC vs REST tradeoffs for multi-provider LLM gatewaysAnalyze grpc vs rest llm gateway tradeoffs for multi-provider inference: where REST wins on compatibility, where gRPC aids internal streaming and typed contracts.API integrationgRPC vs REST for LLM APIs: which is fasterA head-to-head comparison of gRPC vs REST for LLM API speed, latency, throughput, ergonomics, and cost, with a verdict for different engineering use cases.

All grpc vs rest for llm apis posts →