Rate limiting outbound LLM calls in Spring Boot

Third-party LLM APIs fail loud when you exceed quotas: 429 responses, dropped connections, and latency spikes that cascade into your own services. Implementing spring boot rate limiting llm calls at the outbound HTTP layer lets you cap concurrency, shape traffic to provider limits, and degrade gracefully instead of guessing. This walkthrough builds a working throttle for WebClient using Bucket4j, then adds retry and fallback so a single saturated model doesn’t take down the caller.

Step 1: Add dependencies and choose a rate limit strategy

Token bucket is the right primitive for outbound LLM throttling. It admits bursts up to a capacity and refills at a steady rate, matching how most providers publish per-minute or per-second quotas. For a single-process Spring Boot service, in-memory buckets are enough; if you run multiple replicas, swap the bucket store for Redis later without changing the calling code.

Add the core Bucket4j library and WebFlux (we use WebClient for non-blocking I/O):

<dependency>
  <groupId>com.bucket4j</groupId>
  <artifactId>bucket4j-core</artifactId>
  <version>8.10.1</version>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

The pattern for spring boot rate limiting llm calls shown here stays the same whether you target OpenAI, Anthropic, or an OpenAI-compatible gateway.

Step 2: Configure token buckets per model

Different models often have different quotas. Build one bucket per logical destination and key it by a header your client sets. The numbers below are placeholders—set them from application.yml after reading your provider’s documented limits.

@Configuration
public class RateLimitConfig {

    @Bean
    public Map<String, Bucket> llmBuckets() {
        Map<String, Long> ratesPerSecond = Map.of(
            "gpt-4o", 10L,
            "mistral-large", 20L,
            "default", 5L
        );
        Map<String, Bucket> buckets = new HashMap<>();
        ratesPerSecond.forEach((model, rps) -> {
            Bandwidth bw = Bandwidth.simple(rps, Duration.ofSeconds(1));
            buckets.put(model, Bucket.builder().addLimit(bw).build());
        });
        return buckets;
    }
}

If you skip per-model keys, every call competes for the default bucket and you will underutilize high-quota models.

Step 3: Intercept outbound WebClient requests

A Reactor ExchangeFilterFunction runs before the HTTP request leaves the process. Consume one token; if the bucket is empty, fail fast with a domain exception instead of hammering the network.

@Component
public class LlmRateLimitFilter implements ExchangeFilterFunction {
    private final Map<String, Bucket> buckets;

    public LlmRateLimitFilter(Map<String, Bucket> buckets) {
        this.buckets = buckets;
    }

    @Override
    public Mono<ClientResponse> filter(ClientRequest request, ExchangeFunction next) {
        String model = request.headers().getFirst("X-Model");
        Bucket bucket = buckets.getOrDefault(model, buckets.get("default"));
        if (bucket.tryConsume(1)) {
            return next.exchange(request);
        }
        return Mono.error(new RateLimitExceededException(model));
    }
}

Wire the filter into a named WebClient:

@Bean
public WebClient llmWebClient(LlmRateLimitFilter filter) {
    return WebClient.builder()
        .filter(filter)
        .baseUrl("https://api.example.com/v1")
        .defaultHeader("Content-Type", "application/json")
        .build();
}

This is the core of spring boot rate limiting llm calls: the limiter sits at the edge, not inside every service method.

Step 4: Handle 429 and retry with backoff

Client-side throttling reduces but does not eliminate upstream 429s—shared tenants, provider incidents, or clock skew can still trip a quota. Convert error statuses to exceptions and retry with capped exponential backoff.

public class RateLimitExceededException extends RuntimeException {
    public RateLimitExceededException(String model) { super("Limit hit for " + model); }
}

public class UpstreamError extends RuntimeException {
    public UpstreamError(HttpStatus status) { super("Upstream returned " + status); }
}

public Mono<String> complete(String model, String prompt) {
    return llmWebClient.post()
        .uri("/chat/completions")
        .header("X-Model", model)
        .bodyValue(Map.of("model", model, "messages", List.of(Map.of("role","user","content",prompt))))
        .retrieve()
        .onStatus(HttpStatus::isError, r -> Mono.error(new UpstreamError(r.statusCode())))
        .bodyToMono(String.class)
        .retryWhen(Retry.backoff(3, Duration.ofMillis(200))
            .filter(e -> e instanceof RateLimitExceededException || e instanceof UpstreamError));
}

Without spring boot rate limiting llm calls plus retry caps, a retry storm will amplify load exactly when the provider is weakest.

Step 5: Fallback for degraded providers

When a primary model is returning 429s persistently, route to a cheaper or secondary endpoint. A gateway such as n4n.ai honors client routing directives and provides automatic fallback when a provider is rate-limited or degraded, but you still need local throttling to stay within your own token budget. Implement a minimal client-side fallback:

public Mono<String> completeWithFallback(String model, String prompt) {
    return complete(model, prompt)
        .onErrorResume(e -> complete("default", prompt));
}

For multi-base-URL setups, inject a second WebClient bean pointed at the backup host and call it in the onErrorResume branch.

Step 6: Verify success with a local test

Stand up WireMock to stub the LLM endpoint and assert the bucket actually blocks. The test below fires 20 concurrent requests and expects at most the bucket capacity to succeed within the first second.

@WireMockTest
class LlmThrottleTest {

    @Autowired WebClient llmWebClient;

    @Test
    void limitsToBucketCapacity() {
        stubFor(post("/v1/chat/completions").willReturn(okJson("{\"ok\":true}")));
        AtomicInteger success = new AtomicInteger();
        List<Mono<Void>> calls = IntStream.range(0, 20)
            .mapToObj(i -> llmWebClient.post()
                .uri("/v1/chat/completions")
                .header("X-Model", "gpt-4o")
                .bodyValue("{}")
                .retrieve()
                .toBodilessEntity()
                .doOnSuccess(e -> success.incrementAndGet())
                .then())
            .toList();
        StepVerifier.create(Mono.when(calls))
            .expectError(RateLimitExceededException.class)
            .verify();
        assertThat(success.get()).isLessThanOrEqualTo(10);
    }
}

Run it with ./mvnw test -Dtest=LlmThrottleTest. A green build with the assertion holding proves the throttle works. For manual confirmation, loop curl against a local stub:

for i in {1..20}; do curl -s -o /dev/null -w "%{http_code}\n" \
  -H "X-Model: gpt-4o" -d '{}' http://localhost:8080/v1/chat/completions & done

You should see a mix of 200 and 429-equivalent client errors rather than 20 successful posts.

Step 7: Operational notes

Expose bucket state via a @Scheduled logger or Micrometer gauge so you can see refill lag in production. A simple metric:

@Scheduled(fixedDelay = 5000)
void logBucketUsage() {
    llmBuckets().forEach((k, b) -> {
        long available = b.getAvailableTokens();
        Metrics.gauge("llm.bucket.tokens", Tags.of("model", k), available);
    });
}

Treat rate limits as tunable config, not compiled constants. Load them from application.yml and reload on signal if your provider raises quotas. Per-token usage metering belongs at the billing layer; the throttle only controls request rate, not response size, so pair it with response-size guards if you call models that stream unbounded output.

The setup above gives you predictable outbound traffic, clean 429 handling, and a fallback path—everything required to run spring boot rate limiting llm calls safely in a production service.