Writing a reliable Go service that calls an LLM API demands rigorous tests against the HTTP contract, not just unit tests on structs. With go httptest llm client testing, you spin up an in-process mock server that emulates OpenAI-compatible responses—including streaming and error states—without ever hitting a real provider. This guide walks through building that harness end to end, from a minimal client to verified streaming and retry logic.
Step 1: Define a minimal client surface
Start by isolating the transport concerns. A thin wrapper around net/http is easier to test than a fat SDK. Define the request and response types you actually use, and keep the client struct injectable.
type Message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type ChatRequest struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
Stream bool `json:"stream,omitempty"`
}
type ChatResponse struct {
ID string `json:"id"`
Object string `json:"object"`
Choices []struct {
Message Message `json:"message"`
} `json:"choices"`
}
type ChatClient struct {
baseURL string
apiKey string
http *http.Client
}
func NewChatClient(baseURL, apiKey string) *ChatClient {
return &ChatClient{baseURL: baseURL, apiKey: apiKey, http: http.DefaultClient}
}The key detail: baseURL is a field, not a hardcoded constant. That single decision is what makes go httptest llm client testing possible.
Step 2: Stand up a baseline httptest.Server
httptest.NewServer gives you a real TCP listener and a http.Client configured to talk to it. Use it to replace the production endpoint. If you point your client at a unified gateway like n4n.ai, which serves 240+ models through one OpenAI-compatible endpoint with automatic fallback, your mock must reproduce that single contract rather than per-provider shapes.
func TestCreateChat(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
w.Write([]byte(`{"id":"x","object":"chat.completion","choices":[{"message":{"role":"assistant","content":"ok"}}]}`))
}))
defer srv.Close()
client := &ChatClient{baseURL: srv.URL, apiKey: "test", http: srv.Client()}
// ... call and assert
}The srv.Client() returned by httptest already trusts the local cert and uses the right base URL. Never use http.DefaultClient against the test server; it works but loses the locality guarantee.
Step 3: Implement the non-streaming call
Write the method under test. Keep error handling explicit—status codes and malformed JSON are exactly what you want to catch with mocks.
func (c *ChatClient) CreateChat(ctx context.Context, req ChatRequest) (*ChatResponse, error) {
b, err := json.Marshal(req)
if err != nil {
return nil, err
}
httpReq, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/v1/chat/completions", bytes.NewReader(b))
if err != nil {
return nil, err
}
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Authorization", "Bearer "+c.apiKey)
resp, err := c.http.Do(httpReq)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(resp.Body)
return nil, fmt.Errorf("status %d: %s", resp.StatusCode, body)
}
var out ChatResponse
if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
return nil, err
}
return &out, nil
}Step 4: Assert the outgoing request shape and headers
A mock server is not just a responder; it is a spy. Capture the request body and headers inside the handler and assert them after the client returns. This is the core of go httptest llm client testing: you verify what hits the wire, not just what comes back.
func TestCreateChatAssertsRequest(t *testing.T) {
var gotReq ChatRequest
var gotAuth string
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
gotPath = r.URL.Path
json.NewDecoder(r.Body).Decode(&gotReq)
w.WriteHeader(200)
w.Write([]byte(`{"id":"x","object":"chat.completion","choices":[{"message":{"role":"assistant","content":"ok"}}]}`))
}))
defer srv.Close()
client := &ChatClient{baseURL: srv.URL, apiKey: "secret", http: srv.Client()}
_, err := client.CreateChat(context.Background(), ChatRequest{
Model: "gpt-4o-mini",
Messages: []Message{{Role: "user", Content: "hi"}},
})
if err != nil {
t.Fatal(err)
}
if gotPath != "/v1/chat/completions" {
t.Fatalf("unexpected path: %s", gotPath)
}
if gotAuth != "Bearer secret" {
t.Fatalf("auth header wrong: %q", gotAuth)
}
if gotReq.Model != "gpt-4o-mini" {
t.Fatalf("model not forwarded: %q", gotReq.Model)
}
if len(gotReq.Messages) != 1 || gotReq.Messages[0].Content != "hi" {
t.Fatalf("messages not forwarded: %+v", gotReq.Messages)
}
}Run it with go test -run TestCreateChatAssertsRequest -v. Success means a green pass and no diff in the captured fields.
Step 5: Test streaming with chunked JSON
Real LLM calls stream Server-Sent Events (SSE). Your mock must flush chunks, and your client must parse data: lines. Use http.Flusher in the handler; without it, the test will deadlock waiting for a buffer that never flushes.
Client method:
func (c *ChatClient) StreamChat(ctx context.Context, req ChatRequest) (<-chan string, error) {
req.Stream = true
b, _ := json.Marshal(req)
httpReq, _ := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/v1/chat/completions", bytes.NewReader(b))
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Authorization", "Bearer "+c.apiKey)
resp, err := c.http.Do(httpReq)
if err != nil {
return nil, err
}
if resp.StatusCode != 200 {
resp.Body.Close()
return nil, fmt.Errorf("status %d", resp.StatusCode)
}
ch := make(chan string)
go func() {
defer resp.Body.Close()
defer close(ch)
sc := bufio.NewScanner(resp.Body)
for sc.Scan() {
line := sc.Text()
if !strings.HasPrefix(line, "data: ") {
continue
}
data := strings.TrimPrefix(line, "data: ")
if data == "[DONE]" {
return
}
var chunk struct {
Choices []struct {
Delta Message `json:"delta"`
} `json:"choices"`
}
if err := json.Unmarshal([]byte(data), &chunk); err != nil {
return
}
if len(chunk.Choices) > 0 {
ch <- chunk.Choices[0].Delta.Content
}
}
}()
return ch, nil
}Handler with streaming:
func TestStreamChat(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
t.Error("server does not support flushing")
return
}
w.Header().Set("Content-Type", "text/event-stream")
w.WriteHeader(200)
for _, piece := range []string{"Hello", " world"} {
fmt.Fprintf(w, "data: {\"choices\":[{\"delta\":{\"content\":%q}}]}\n\n", piece)
flusher.Flush()
time.Sleep(5 * time.Millisecond)
}
fmt.Fprint(w, "data: [DONE]\n\n")
flusher.Flush()
}))
defer srv.Close()
client := &ChatClient{baseURL: srv.URL, apiKey: "test", http: srv.Client()}
ch, err := client.StreamChat(context.Background(), ChatRequest{Model: "gpt-4o", Messages: []Message{{Role: "user", Content: "say hi"}}})
if err != nil {
t.Fatal(err)
}
var got strings.Builder
for piece := range ch {
got.WriteString(piece)
}
if got.String() != "Hello world" {
t.Fatalf("streamed content wrong: %q", got.String())
}
}When you run this, the test completes in a few milliseconds. If you omit flusher.Flush(), the Scanner blocks until the handler returns, defeating the purpose of the test.
Step 6: Simulate degradation and retry logic
Production LLM gateways rate-limit. Your client should retry on 429 or 503. Mock a server that fails the first call and succeeds on the second using a closure counter.
func TestRetryOn429(t *testing.T) {
var hits int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
hits++
if hits == 1 {
w.WriteHeader(429)
w.Write([]byte(`{"error":"rate limited"}`))
return
}
w.WriteHeader(200)
w.Write([]byte(`{"id":"y","object":"chat.completion","choices":[{"message":{"role":"assistant","content":"recovered"}}]}`))
}))
defer srv.Close()
client := &ChatClient{baseURL: srv.URL, apiKey: "test", http: srv.Client()}
// Wrap with a simple retry loop in the test or in the client.
var resp *ChatResponse
var err error
for attempt := 0; attempt < 3; attempt++ {
resp, err = client.CreateChat(context.Background(), ChatRequest{Model: "gpt-4o"})
if err == nil {
break
}
time.Sleep(time.Duration(attempt+1) * 10 * time.Millisecond)
}
if err != nil {
t.Fatalf("retry failed: %v", err)
}
if resp.Choices[0].Message.Content != "recovered" {
t.Fatalf("unexpected content: %q", resp.Choices[0].Message.Content)
}
if hits != 2 {
t.Fatalf("expected 2 hits, got %d", hits)
}
}This pattern proves your retry budget and backoff actually trigger. In go httptest llm client testing, error injection is as important as happy-path mocking.
Step 7: Verify context cancellation
A forgotten context propagation bug will hang your service under load. Test it by canceling the context before the mock sleeps.
func TestContextCancel(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(100 * time.Millisecond)
w.WriteHeader(200)
}))
defer srv.Close()
ctx, cancel := context.WithCancel(context.Background())
cancel() // already canceled
client := &ChatClient{baseURL: srv.URL, http: srv.Client()}
_, err := client.CreateChat(ctx, ChatRequest{Model: "gpt-4o"})
if err == nil {
t.Fatal("expected context error")
}
}If the client ignores ctx, the call succeeds and the test fails—exactly the bug you want to catch locally.
Step 8: Run the suite and verify success
Put each test in client_test.go alongside the client. Run the full package:
go test ./... -v -raceExpected output includes PASS for TestCreateChatAssertsRequest, TestStreamChat, TestRetryOn429, and TestContextCancel. The -race flag confirms no data races in your streaming goroutine. If you see flaky failures on TestStreamChat, increase the flush sleep or use httptest.NewUnstartedServer with a longer timeout—but never suppress the flush.
The foundation of go httptest llm client testing is treating HTTP as a boundary, not an implementation detail. Mock at the transport layer, assert on bytes, and exercise streaming and failure modes. That gives you a client you can ship against any OpenAI-compatible endpoint with confidence.