Building a proxy or direct API server for language models demands protecting upstream compute from runaway clients. A practical way to enforce quotas is with gin middleware rate limiting llm endpoints by request count or estimated token usage. This guide walks through implementing token-aware limits in Go using Gin, with runnable code you can drop into an existing service.
Step 1: Pick the right limit dimension
Rate limiting an LLM endpoint is not the same as throttling a CRUD API. A single chat completion request can consume 10 tokens or 10,000 depending on the prompt and max_tokens. If you limit only on requests per second, a client can still exhaust your GPU budget with one massive prompt.
Decide between three dimensions:
- Requests per minute (RPM) – simple, protects against connection floods.
- Tokens per minute (TPM) – aligns with provider billing and compute.
- Concurrent requests – caps in-flight inferences.
For most gateways, combine RPM with a token estimate. You can refine after responses using the usage field from the provider.
Define a config struct:
type LimitConfig struct {
RPM float64 // requests per minute
TPM float64 // tokens per minute
Burst int // allowed burst
}Step 2: Scaffold the Gin service
Initialize a module and pull Gin plus the standard rate package.
go mod init llm-gateway
go get github.com/gin-gonic/gin
go get golang.org/x/time/rateA minimal server looks like this:
package main
import "github.com/gin-gonic/gin"
func main() {
r := gin.Default()
r.POST("/v1/chat/completions", func(c *gin.Context) {
c.JSON(200, gin.H{"echo": "not implemented"})
})
r.Run(":8080")
}We will replace the handler with a forwarding proxy later.
Step 3: Implement request-rate middleware
Start with RPM limiting keyed by API key or IP. The golang.org/x/time/rate package gives a token bucket per limiter. Store limiters in a map guarded by a mutex.
type rateLimiter struct {
limiters map[string]*rate.Limiter
mu sync.Mutex
cfg LimitConfig
}
func newRateLimiter(cfg LimitConfig) *rateLimiter {
return &rateLimiter{
limiters: make(map[string]*rate.Limiter),
cfg: cfg,
}
}
func (rl *rateLimiter) getLimiter(key string) *rate.Limiter {
rl.mu.Lock()
defer rl.mu.Unlock()
l, ok := rl.limiters[key]
if !ok {
l = rate.NewLimiter(rate.Limit(rl.cfg.RPM/60.0), rl.cfg.Burst)
rl.limiters[key] = l
}
return l
}The gin middleware rate limiting llm requests by RPM checks Allow():
func (rl *rateLimiter) middleware() gin.HandlerFunc {
return func(c *gin.Context) {
key := c.GetHeader("X-API-Key")
if key == "" {
key = c.ClientIP()
}
if !rl.getLimiter(key).Allow() {
c.Header("Retry-After", "60")
c.AbortWithStatusJSON(429, gin.H{"error": "rate limit exceeded"})
return
}
c.Next()
}
}Attach it:
rl := newRateLimiter(LimitConfig{RPM: 60, TPM: 100000, Burst: 5})
r.Use(rl.middleware())Step 4: Add token-aware throttling
RPM alone is not enough. We need to estimate tokens before forwarding. A rough heuristic: 1 token ≈ 4 characters for English text. Parse the incoming JSON body to read max_tokens and the prompt content.
type chatRequest struct {
Messages []struct {
Role string `json:"role"`
Content string `json:"content"`
} `json:"messages"`
MaxTokens int `json:"max_tokens"`
}
### Token estimation heuristic
func estimateTokens(req chatRequest) int {
chars := 0
for _, m := range req.Messages {
chars += len(m.Content)
}
promptTokens := chars / 4
if req.MaxTokens > 0 {
return promptTokens + req.MaxTokens
}
return promptTokens + 512 // default completion guess
}We extend the limiter to track a separate token bucket. For simplicity, use a second rate.Limiter with TPM converted to per-second:
func (rl *rateLimiter) getTokenLimiter(key string) *rate.Limiter {
rl.mu.Lock()
defer rl.mu.Unlock()
// reuse map with suffix to avoid collision
l, ok := rl.limiters[key+":tpm"]
if !ok {
l = rate.NewLimiter(rate.Limit(rl.cfg.TPM/60.0), rl.cfg.Burst*1000)
rl.limiters[key+":tpm"] = l
}
return l
}In the middleware, read body, then check token bucket:
func (rl *rateLimiter) tokenMiddleware() gin.HandlerFunc {
return func(c *gin.Context) {
key := c.GetHeader("X-API-Key")
if key == "" {
key = c.ClientIP()
}
var req chatRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.AbortWithStatusJSON(400, gin.H{"error": "bad request"})
return
}
needed := estimateTokens(req)
tl := rl.getTokenLimiter(key)
if !tl.AllowN(time.Now(), needed) {
c.Header("Retry-After", "60")
c.AbortWithStatusJSON(429, gin.H{"error": "token rate limit exceeded"})
return
}
c.Next()
}
}Note: ShouldBindJSON consumes the body. If you need to forward it later, store the parsed struct in context and re-marshal, or use c.Request.Body with a buffer. For clarity, we re-marshal in the handler.
Step 5: Proxy to an LLM provider
Now replace the stub handler with a forwarder. If you front an inference gateway like n4n.ai, which provides an OpenAI-compatible endpoint across 240+ models with automatic fallback, you still need per-client limits before traffic hits it. The middleware above already enforces that.
func proxyHandler(upstream string, apiKey string) gin.HandlerFunc {
return func(c *gin.Context) {
var req chatRequest
c.ShouldBindJSON(&req) // already bound in middleware, but rebind safe
body, _ := json.Marshal(req)
url := upstream + "/v1/chat/completions"
httpReq, _ := http.NewRequest("POST", url, bytes.NewReader(body))
httpReq.Header.Set("Content-Type", "application/json")
httpReq.Header.Set("Authorization", "Bearer "+apiKey)
// forward provider cache-control hints if present
if cc := c.GetHeader("Cache-Control"); cc != "" {
httpReq.Header.Set("Cache-Control", cc)
}
resp, err := http.DefaultClient.Do(httpReq)
if err != nil {
c.AbortWithStatusJSON(502, gin.H{"error": "upstream failure"})
return
}
defer resp.Body.Close()
c.Status(resp.StatusCode)
io.Copy(c.Writer, resp.Body)
}
}Wire it:
r.POST("/v1/chat/completions", proxyHandler("https://api.openai.com", "sk-..."))For an OpenRouter-class gateway, swap the base URL and key.
Step 6: Return proper limit headers
Clients integrate better when you advertise remaining quota. Extend the RPM middleware to set headers:
func (rl *rateLimiter) middleware() gin.HandlerFunc {
return func(c *gin.Context) {
key := c.GetHeader("X-API-Key")
if key == "" {
key = c.ClientIP()
}
l := rl.getLimiter(key)
if !l.Allow() {
c.Header("Retry-After", "60")
c.AbortWithStatusJSON(429, gin.H{"error": "rate limit exceeded"})
return
}
c.Header("X-RateLimit-Limit", fmt.Sprintf("%.0f", rl.cfg.RPM))
c.Header("X-RateLimit-Remaining", fmt.Sprintf("%.0f", l.Tokens()))
c.Next()
}
}This gives clients visibility into the gin middleware rate limiting llm posture.
Step 7: Verify the setup
Run the server and fire requests with curl. First, a valid call:
curl -i -X POST localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-API-Key: test" \
-d '{"messages":[{"role":"user","content":"hi"}],"max_tokens":50}'Expect 200 OK and the upstream JSON.
Now exceed the limit. A loop sending 100 requests quickly:
for i in {1..100}; do curl -s -o /dev/null -w "%{http_code}\n" \
-X POST localhost:8080/v1/chat/completions \
-H "X-API-Key: test" \
-d '{"messages":[{"role":"user","content":"spam"}],"max_tokens":2000}'; doneYou should see 429 responses after the burst and RPM threshold. Check the Retry-After and X-RateLimit-Remaining headers on successful calls to confirm the buckets drain.
If you wired the token middleware, a single request with max_tokens: 100000 for key test should immediately return 429 because the TPM bucket lacks capacity. That confirms token-aware enforcement.
Production notes
The in-memory limiter works for a single instance. For multiple replicas, back it with Redis using a fixed-window or sliding-log algorithm. Libraries like github.com/go-redis/redis_rate fit well.
Also consider:
- Key selection: API key is better than IP to prevent NAT collisions.
- Dynamic config: load limits from env or a control plane.
- Response usage reconciliation: after upstream returns, if actual
usage.total_tokensis less than estimated, refund the difference. - Streaming: for SSE responses, you cannot easily know completion tokens upfront; estimate conservatively and reconcile post-stream.
The gin middleware rate limiting llm pattern shown here keeps abusive clients off your expensive inference boxes while remaining ~150 lines of Go. Swap the proxy target to any OpenAI-compatible endpoint and you have a guarded edge.