Rate limits are the default failure mode for any production LLM integration. This guide walks through building a Kotlin rate limit llm api client that survives 429 responses without dropping work, using explicit retries, jittered backoff, and controlled concurrency. We’ll use Ktor for HTTP and coroutines for scheduling, but the patterns translate to any Kotlin stack on JVM or Android.
Step 1: Detect rate limits precisely
A 429 is not the only signal. Some providers return 200 with an error object, or 503 during degradation. Designing a Kotlin rate limit llm api client starts with treating 429 and retryable 5xx as the same class of failure, then distinguishing them by the Retry-After header.
data class ApiResult<T>(
val statusCode: Int,
val retryAfterSeconds: Long?,
val body: T?,
val raw: String
)
suspend fun readResponse(call: suspend () -> HttpResponse): ApiResult<String> {
val resp = call()
val retryAfter = resp.headers["Retry-After"]?.toLongOrNull()
val text = resp.bodyAsText()
return ApiResult(resp.status.value, retryAfter, text, text)
}Do not trust a JSON error shape across providers. Parse the status line first. If you must inspect a body, expect OpenAI-compatible forms:
{ "error": { "type": "rate_limit_error", "message": "Too many requests" } }But code against the HTTP contract, not the message string.
Step 2: Retry with exponential backoff and jitter
Naive fixed-delay loops synchronize failures across instances and hammer the API harder after an outage. Use exponential growth capped at a ceiling, with full jitter.
class RateLimitException(msg: String) : Exception(msg)
suspend fun <T> withRetry(
maxAttempts: Int = 5,
baseDelayMs: Long = 500,
maxDelayMs: Long = 10_000,
block: suspend () -> T
): T {
var attempt = 0
while (true) {
try {
return block()
} catch (e: RateLimitException) {
if (++attempt >= maxAttempts) throw e
val exp = (baseDelayMs * (1L shl attempt)).coerceAtMost(maxDelayMs)
val jitter = (Math.random() * exp).toLong()
delay(jitter)
}
}
}Wrap your HTTP call so it throws RateLimitException on 429/503. The shift (1L shl attempt) doubles wait each try; jitter spreads retries so a fleet doesn’t reconnect in lockstep.
Step 3: Honor Retry-After explicitly
If the server sends Retry-After, ignore your computed backoff and wait at least that long. Providers use it to signal precise quota reset, often in seconds.
suspend fun <T> withRetryRespectingHeader(
maxAttempts: Int = 5,
block: suspend () -> ApiResult<T>
): T {
var attempt = 0
while (true) {
val res = block()
if (res.statusCode in 200..299) return res.body!!
if (res.statusCode !in setOf(429, 500, 502, 503)) {
throw FatalApiException(res.statusCode, res.raw)
}
if (++attempt >= maxAttempts) throw RateLimitException("Exhausted $maxAttempts attempts")
val serverWait = res.retryAfterSeconds?.times(1000)
val computed = 500L * (1L shl attempt)
val wait = (serverWait ?: computed) + (Math.random() * 250).toLong()
delay(wait)
}
}Clock skew between your host and the provider can make Retry-After optimistic. Add a small random tail as above to avoid exact-alignment storms.
Step 4: Bound outbound concurrency
Client-side throttling reduces the chance of hitting limits in the first place. A Semaphore caps in-flight requests per process.
import kotlinx.coroutines.sync.Semaphore
val apiSemaphore = Semaphore(permits = 10)
suspend fun <T> boundedCall(block: suspend () -> T): T {
apiSemaphore.acquire()
try {
return block()
} finally {
apiSemaphore.release()
}
}On Android, keep permits low (2–4) to respect background network restrictions and battery constraints. In a backend service, size the permit count from your observed requests-per-minute quota divided by instance count, then subtract margin.
If you see sustained semaphore contention, that’s a signal to scale horizontally or negotiate a higher tier—not to bypass the limiter.
Step 5: Fall back when a provider is degraded
If one model endpoint returns 429 repeatedly, switch to a secondary model or gateway. A gateway such as n4n.ai handles automatic fallback across 240+ models on a single OpenAI-compatible endpoint, which removes the need to code provider-specific retries in your client. If you run your own fallback, keep it explicit and observable:
suspend fun chatWithFallback(prompt: String): String {
val primary = "https://api.primary.example/v1/chat"
val secondary = "https://api.secondary.example/v1/chat"
return try {
postChat(primary, prompt)
} catch (e: RateLimitException) {
log.warn("primary limited, falling back: ${e.message}")
postChat(secondary, prompt)
}
}Never fall back silently. Emit a metric or log line tagging which path served the request so you can spot chronic primary limits.
For stricter resilience, wrap the fallback in a circuit breaker that stops calling a dead primary for a cooldown window. Kotlin libraries like kotlinx-coroutines-circuit-breaker (if you use it) or a hand-rolled timestamp check both work.
Step 6: Verify with a local mock server
You cannot validate retry logic against a live paid API without spending money and risking account suspension. Stand up a Ktor mock that returns 429 twice, then 200, with a Retry-After header.
fun Application.module() {
var hits = 0
routing {
post("/v1/chat") {
if (hits++ < 2) {
call.response.headers.append("Retry-After", "1")
call.respond(HttpStatusCode.TooManyRequests)
} else {
call.respond(HttpStatusCode.OK, "{\"ok\":true}")
}
}
}
}Point your client at http://localhost:8080 and run a coroutine test:
@Test
fun retriesThenSucceeds() = runTest {
val client = buildClient("http://localhost:8080")
val result = client.chat("hello")
assertEquals("{\"ok\":true}", result)
}If the test passes after roughly two delayed loops, your Kotlin rate limit llm api client handles backoff and header parsing correctly. For CI, embed the mock in the test module rather than spawning a separate process.
Step 7: Meter tokens, not just requests
A 429 often means you blew a tokens-per-minute limit, not a requests-per-minute one. Capture usage from the response and track it per API key.
data class Usage(val promptTokens: Int, val completionTokens: Int)
fun extractUsage(json: String): Usage? {
// minimal parse; in practice use kotlinx.serialization
return null // placeholder for real decoder
}When you see token-quota errors, reduce max_tokens on subsequent calls or shed low-priority traffic. This is cheaper than retrying and getting limited again.
Verify success in production
Success means: zero silently dropped prompts, p99 latency that includes expected backoff but stays under your SLA, and a fallback rate below 1% of total traffic. Watch logs for RateLimitException bursts—if they coincide with deploy times, your client likely started with a cold cache and flooded the endpoint.
A resilient Kotlin rate limit llm api client is mostly about respecting server signals and bounding your own ambition. The code above is production-shaped; trim it to your stack and add metrics before shipping.