A hung LLM request can burn tokens and block your service indefinitely if you don’t enforce a go context cancellation llm timeout. Go’s context package gives you precise control over request lifetimes, but the standard net/http client needs explicit wiring to respect it.
Step 1: Separate transport-level timeouts from request deadlines
The net/http client has two distinct failure domains. Transport timeouts (dial, TLS handshake, idle connection) protect against slow network setup. The context deadline protects against a slow or stalled upstream model inference.
Set transport timeouts conservatively so connections don’t hang before your request even starts:
package llm
import (
"net"
"net/http"
"time"
)
func newTransport() *http.Transport {
return &http.Transport{
DialContext: (&net.Dialer{
Timeout: 5 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
TLSHandshakeTimeout: 5 * time.Second,
ResponseHeaderTimeout: 10 * time.Second,
IdleConnTimeout: 90 * time.Second,
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
}
}These values are starting points. If your go context cancellation llm timeout is 30 seconds, the transport headers must arrive well before that.
Step 2: Build a minimal LLM client that takes context
Wrap http.Client with a method that always uses http.NewRequestWithContext. Never call http.Post or http.Get—they ignore your context.
type Client struct {
baseURL string
http *http.Client
apiKey string
}
func NewClient(baseURL, apiKey string) *Client {
return &Client{
baseURL: baseURL,
http: &http.Client{Transport: newTransport()},
apiKey: apiKey,
}
}
type chatReq struct {
Model string `json:"model"`
Messages []struct {
Role string `json:"role"`
Content string `json:"content"`
} `json:"messages"`
}
func (c *Client) Complete(ctx context.Context, model, prompt string) ([]byte, error) {
body := chatReq{Model: model}
body.Messages = append(body.Messages, struct {
Role string `json:"role"`
Content string `json:"content"`
}{Role: "user", Content: prompt})
buf, _ := json.Marshal(body)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/v1/chat/completions", bytes.NewReader(buf))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+c.apiKey)
resp, err := c.http.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
b, _ := io.ReadAll(resp.Body)
return nil, fmt.Errorf("llm status %d: %s", resp.StatusCode, b)
}
return io.ReadAll(resp.Body)
}This skeleton already respects context cancellation: if ctx is cancelled, c.http.Do returns immediately.
Step 3: Apply a go context cancellation llm timeout at the call site
Never hardcode the timeout inside the client. Let the caller decide based on the endpoint’s latency profile. For a chat completion behind a gateway, 20–40 seconds is typical.
func handleChat(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// 25s budget for the LLM call, derived from go context cancellation llm timeout needs
ctx, cancel := context.WithTimeout(ctx, 25*time.Second)
defer cancel()
client := NewClient("https://api.example.com", os.Getenv("LLM_KEY"))
resp, err := client.Complete(ctx, "gpt-4o-mini", "Explain RAFT")
if err != nil {
if errors.Is(err, context.DeadlineExceeded) {
http.Error(w, "llm timeout", http.StatusGatewayTimeout)
return
}
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
w.Write(resp)
}If you route through a single OpenAI-compatible endpoint such as n4n.ai, the same context propagates and cancels in-flight requests when the provider is slow or rate-limited, saving tokens on abandoned generations.
Step 4: Stream responses without leaking goroutines
Streaming is where most teams get context handling wrong. The server sends tokens slowly; your read loop must observe cancellation.
func (c *Client) Stream(ctx context.Context, model, prompt string, fn func(token string)) error {
body := map[string]interface{}{
"model": model,
"stream": true,
"messages": []map[string]string{{"role": "user", "content": prompt}},
}
buf, _ := json.Marshal(body)
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/v1/chat/completions", bytes.NewReader(buf))
req.Header.Set("Authorization", "Bearer "+c.apiKey)
req.Header.Set("Content-Type", "application/json")
resp, err := c.http.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
scanner := bufio.NewScanner(resp.Body)
scanner.Buffer(make([]byte, 1024*1024), 1024*1024)
for {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
if !scanner.Scan() {
if err := scanner.Err(); err != nil && !errors.Is(err, context.Canceled) {
return err
}
return nil
}
line := scanner.Text()
if strings.HasPrefix(line, "data: ") {
fn(strings.TrimPrefix(line, "data: "))
}
}
}The select checks cancellation every iteration. Without it, a slow network write on the server side can stall scanner.Scan() and ignore a cancelled context until the next read—which might be never.
Step 5: Test cancellation with httptest
You cannot claim your go context cancellation llm timeout works without a test that proves the client aborts before the server responds.
func TestClientTimeout(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(2 * time.Second) // simulate slow model
w.Write([]byte(`{"ok":true}`))
}))
defer srv.Close()
c := NewClient(srv.URL, "test")
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := c.Complete(ctx, "test", "hi")
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("expected deadline exceeded, got %v", err)
}
}Run go test -run TestClientTimeout -v. If it passes, your client respects the deadline. If it hangs for two seconds, you wired context incorrectly.
Step 6: Avoid the body-close race on cancellation
When context cancels mid-read, resp.Body.Close() is called by defer. That tears down the connection. But if you’ve spawned a separate goroutine to read the body, it may write to a closed pipe. Keep reads in the same goroutine as the context check, as shown in Step 4.
Also, do not reuse the same http.Client with a zero Timeout field and rely solely on context. The client-level Timeout covers the entire exchange including reading the body; context covers it too but with clearer error typing. Set http.Client{Timeout: 0} and depend on context, or set both—just be aware that client.Timeout returns http.ErrHandlerTimeout, not a context error.
Step 7: Verify success in production
After deployment, confirm the following:
- Metrics show
context.DeadlineExceedederrors correlated with your configured go context cancellation llm timeout, not with transport errors. - No goroutine leaks:
runtime.NumGoroutine()stable under load tests that force cancellations. - Token spend drops for aborted requests (most LLM providers stop billing once the connection closes, but verify with per-token metering).
A simple curl loop with a proxy that delays responses will exercise the path:
# use a local proxy or toxiproxy to add 30s latency
for i in {1..100}; do curl -m 5 http://localhost:8080/chat -d '{"prompt":"test"}'; doneIf your service returns 504 within 5 seconds consistently, the timeout is enforced.
Step 8: Advanced pattern—per-attempt timeout with fallback
When you have multiple model providers, wrap the call in a retry that uses a fresh context for each attempt but a parent context for total budget:
func completeWithFallback(parent context.Context, c *Client, models []string) ([]byte, error) {
for _, m := range models {
attemptCtx, cancel := context.WithTimeout(parent, 8*time.Second)
resp, err := c.Complete(attemptCtx, m, "hello")
cancel()
if err == nil {
return resp, nil
}
if errors.Is(err, context.DeadlineExceeded) {
continue // try next model
}
return nil, err
}
return nil, errors.New("all models timed out")
}This bounds each provider call while preserving the overall go context cancellation llm timeout from the inbound request.
Final notes
Context cancellation is not optional for LLM clients. A single stuck generation can exhaust your worker pool and rack up cost. Wire context.WithTimeout at every boundary, use NewRequestWithContext, and test the failure path explicitly. The code above is production-grade minus your auth and retry telemetry—drop it in and measure.