n4nAI
Open console →
API integrationHow-toKotlin & Android LLM Integration

Kotlin coroutines for async LLM API calls

Step-by-step guide to using Kotlin coroutines for async LLM API calls: build a suspended, streaming OpenAI-compatible client with Ktor, timeouts, and verification.

n4n Team3 min read733 words

Audio narration

Coming soon — every post will get a voice note here.

Kotlin coroutines llm api integration should be the default for any JVM or Android app talking to language models. Blocking HTTP clients waste threads and make cancellation a nightmare; suspend functions let you write linear async code that respects structured concurrency. This walkthrough builds a small, production-shaped client for an OpenAI-compatible chat endpoint using Kotlin coroutines and Ktor, covering streaming, timeouts, and verification.

Step 1: Configure the project and dependencies

Use Gradle with Kotlin DSL. You need kotlinx-coroutines-core, ktor-client-core, ktor-client-cio, ktor-client-content-negotiation, ktor-client-serialization, and kotlinx-serialization-json.

// build.gradle.kts
plugins {
    kotlin("jvm") version "1.9.23"
    kotlin("plugin.serialization") version "1.9.23"
}

dependencies {
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.8.0")
    implementation("io.ktor:ktor-client-core:2.3.10")
    implementation("io.ktor:ktor-client-cio:2.3.10")
    implementation("io.ktor:ktor-client-content-negotiation:2.3.10")
    implementation("io.ktor:ktor-client-serialization:2.3.10")
    implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.6.3")
    testImplementation("io.ktor:ktor-client-mock:2.3.10")
}

If you are on Android, swap the JVM plugin for the Android Gradle plugin and add androidx.lifecycle:lifecycle-viewmodel-ktx for viewModelScope. Keep the Ktor version aligned across all io.ktor artifacts to avoid bytecode mismatches.

Step 2: Model the request and response

OpenAI-compatible chat completions accept a JSON body with model, messages, and optional stream. Define serializable data classes that ignore unknown fields so backend additions don’t break you.

import kotlinx.serialization.*

@Serializable
data class ChatMessage(val role: String, val content: String)

@Serializable
data class ChatRequest(
    val model: String,
    val messages: List<ChatMessage>,
    val stream: Boolean = false,
    val temperature: Double = 0.7
)

@Serializable
data class ChatChoice(val index: Int, val message: ChatMessage, val finish_reason: String? = null)

@Serializable
data class ChatResponse(
    val id: String,
    val `object`: String,
    val created: Long,
    val model: String,
    val choices: List<ChatChoice>,
    val usage: Usage? = null
)

@Serializable
data class Usage(val prompt_tokens: Int, val completion_tokens: Int, val total_tokens: Int)

Keep the types tight. If you later switch to a gateway that addresses 240+ models through one endpoint, the same ChatRequest works unchanged because the request shape is part of the OpenAI compatibility contract.

Step 3: Write a suspended client

Create a single HttpClient instance and wrap the POST in a suspend fun. Never use GlobalScope; inject a CoroutineScope or just expose suspend functions and let callers provide the scope.

import io.ktor.client.*
import io.ktor.client.request.*
import io.ktor.client.statement.*
import io.ktor.http.*
import io.ktor.client.plugins.contentnegotiation.*
import io.ktor.serialization.kotlinx.json.*
import kotlinx.serialization.json.Json

class LLMClient(
    private val baseUrl: String,
    private val apiKey: String,
    private val http: HttpClient = HttpClient(CIO) {
        install(ContentNegotiation) {
            json(Json { ignoreUnknownKeys = true })
        }
    }
) {
    suspend fun complete(req: ChatRequest): ChatResponse {
        return http.post("$baseUrl/v1/chat/completions") {
            contentType(ContentType.Application.Json)
            headers { append("Authorization", "Bearer $apiKey") }
            setBody(req)
        }.body<ChatResponse>()
    }
}

This is the core of kotlin coroutines llm api usage: a normal-looking function that suspends instead of blocking. Under the hood Ktor maps the IO to non-blocking sockets on the CIO engine, so a handful of threads service thousands of concurrent inflight requests.

Step 4: Invoke from a structured scope

On Android, use viewModelScope. On a backend service, create a CoroutineScope(Dispatchers.IO + SupervisorJob()) tied to your component lifecycle.

fun CoroutineScope.loadAnswer(prompt: String) = launch {
    val client = LLMClient("https://api.openai.com", System.getenv("OPENAI_KEY")!!)
    val resp = client.complete(
        ChatRequest(
            model = "gpt-4o-mini",
            messages = listOf(ChatMessage("user", prompt))
        )
    )
    println(resp.choices.first().message.content)
}

If you call complete from a suspended context, the HTTP call inherits the caller’s cancellation. Cancel the parent job and the socket read is aborted. That behavior is why coroutines beat callback-based SDKs for LLM calls where users back out of a screen mid-generation.

Step 5: Timeouts and supervised error handling

Network calls fail. Wrap with withTimeout and catch TimeoutCancellationException. Use supervisorScope so one failed call doesn’t kill sibling tasks.

suspend fun safeComplete(client: LLMClient, req: ChatRequest): ChatResponse? =
    supervisorScope {
        try {
            withTimeout(10_000) {
                client.complete(req)
            }
        } catch (e: TimeoutCancellationException) {
            null // treat timeout as empty result, or rethrow domain error
        } catch (e: ResponseException) {
            // 4xx/5xx: inspect e.response.status
            null
        }
    }

Do not swallow CancellationException; rethrow it so coroutine cancellation stays transparent. If you need parallel fan-out (e.g., summarize three documents at once), use async inside the same supervisorScope and awaitAll(); a single failure won’t cancel the others unless you explicitly do so.

Step 6: Stream tokens with Flow

For chat UIs you want incremental tokens. OpenAI-compatible endpoints support stream: true returning Server-Sent Events. Ktor’s preparePost with execute lets you read the raw stream inside a coroutine.

import io.ktor.client.request.*
import io.ktor.client.statement.*
import kotlinx.coroutines.flow.*
import kotlinx.serialization.json.*

fun LLMClient.stream(req: ChatRequest): Flow<String> = flow {
    val streamReq = req.copy(stream = true)
    val response = http.preparePost("$baseUrl/v1/chat/completions") {
        contentType(ContentType.Application.Json)
        headers { append("Authorization", "Bearer $apiKey") }
        setBody(streamReq)
    }.execute()
    val channel = response.bodyAsChannel()
    while (!channel.isClosedForRead) {
        val line = channel.readUTF8Line() ?: continue
        if (line.startsWith("data:")) {
            val json = line.removePrefix("data:").trim()
            if (json == "[DONE]") break
            val delta = Json.decodeFromString<StreamChunk>(json)
            delta.choices.firstOrNull()?.delta?.content?.let { emit(it) }
        }
    }
}

@Serializable
data class StreamChunk(val choices: List<DeltaChoice>)

@Serializable
data class DeltaChoice(val delta: DeltaContent, val finish_reason: String? = null)

@Serializable
data class DeltaContent(val content: String? = null)

Collect the flow in a scope and update UI state. Because it is a cold flow, collection cancellation closes the HTTP channel automatically. On Android, map the flow to StateFlow with stateIn(viewModelScope, SharingStarted.WhileSubscribed(5_000), "") so the UI gets the latest partial text even across rotation.

Step 7: Retries and provider fallback

Transient 429s and 5xx are normal. Implement a small retry with backoff:

suspend fun <T> retryIO(times: Int = 3, block: suspend () -> T): T {
    var last: Exception? = null
    repeat(times) { i ->
        try { return block() }
        catch (e: Exception) {
            if (e is CancellationException) throw e
            last = e; delay((i + 1) * 500L)
        }
    }
    throw last!!
}

If you point your baseUrl at a gateway such as n4n.ai, automatic fallback when a provider is rate-limited or degraded happens server-side, so the client retry above becomes a secondary safety net rather than your primary resilience mechanism. The client code stays identical because the endpoint is OpenAI-compatible.

Step 8: Verify success

You need both unit and integration checks.

Unit test with MockEngine confirms serialization and auth header:

@Test
fun `completes chat`() = runTest {
    val mock = HttpClient(MockEngine) {
        engine {
            addHandler { request ->
                respond(
                    content = """{"id":"1","object":"chat.completion","created":1,"model":"gpt-4o-mini","choices":[{"index":0,"message":{"role":"assistant","content":"hi"}}]}""",
                    status = HttpStatusCode.OK,
                    headers = headersOf(HttpHeaders.ContentType, "application/json")
                )
            }
        }
        install(ContentNegotiation) { json() }
    }
    val client = LLMClient("https://x", "key", mock)
    val r = client.complete(ChatRequest("gpt-4o-mini", listOf(ChatMessage("user","hi"))))
    assertEquals("hi", r.choices.first().message.content)
}

Integration test (manual or CI with secret) hits the real endpoint and prints token usage:

suspend fun main() {
    val client = LLMClient("https://api.openai.com", System.getenv("KEY")!!)
    val r = client.complete(ChatRequest("gpt-4o-mini", listOf(ChatMessage("user","ping"))))
    println("Used ${r.usage?.total_tokens} tokens")
}

Success criteria: the unit test passes; the integration call returns a ChatResponse with non-null choices and a usage block showing positive token counts. If you enabled streaming, the flow emits multiple strings and concatenation matches the final non-streamed content. Run the streaming variant and confirm that cancelling the collect job stops new tokens within a few hundred milliseconds.

Practical notes for production

  • Pin the Dispatchers appropriately. Ktor already uses IO threads; don’t wrap suspend calls in withContext(Dispatchers.IO) unnecessarily.
  • For Android, expose StateFlow from the ViewModel; collect the streaming flow and reduce into a single StringBuilder for the displayed message.
  • Honor provider cache-control hints if your gateway forwards them; send extra_headers or extension fields only if your endpoint supports them. The OpenAI-compatible shape keeps client code stable even when the backend routes to different models.
  • Avoid runBlocking in services; it defeats the kotlin coroutines llm api advantage. Use runTest only in tests.

Following these steps gives you a cancellable, streaming, timeout-aware LLM client that drops into any Kotlin stack. The same patterns apply whether you target a single provider or a unified gateway.

Tagskotlincoroutinesasyncllm-client

Written by

n4n Team

The team building n4n — a single OpenAI-compatible API in front of 240+ models, with automatic fallback, load balancing and pay-per-token metering.

More from n4n Team →
API integrationHandling rate limits in Kotlin LLM API clientsPractical patterns for building resilient Kotlin LLM API clients that handle rate limits with retries, backoff, and fallback to keep workloads running.API integrationParsing LLM responses in Kotlin with kotlinx.serializationHands-on Kotlin tutorial: parse LLM API responses with kotlinx.serialization llm api parsing, covering data models, streaming, and error handling.API integrationStreaming LLM responses in Android with Kotlin FlowLearn how to implement android kotlin flow streaming llm responses over SSE with OkHttp, coroutines, and Compose for live token updates.API integrationUnit testing LLM API calls in Kotlin with MockWebServerStep-by-step kotlin mockwebserver llm api testing tutorial: mock OpenAI-compatible endpoints, assert requests, and run deterministic LLM client unit tests.API integrationViewModel and StateFlow patterns for LLM chat appsPractical patterns for building Android LLM chat apps with ViewModel and StateFlow: model state, stream tokens, handle cancellation, and avoid common pitfalls.API integrationBuilding a Jetpack Compose chat UI for an LLM APIStep-by-step tutorial to build a Jetpack Compose chat UI for an LLM API in Kotlin, covering streaming, state management, and error handling.

All kotlin & android llm integration posts →