Async/await patterns for concurrent LLM calls in C#

Calling an LLM API from C# is trivial until you need to do it fifty times per second. The right c# async await concurrent llm calls pattern determines whether your service scales linearly or collapses under thread-pool starvation. This guide walks through the concurrency primitives you actually need, the mistakes that bite in production, and how to keep p99 latency under control.

Start with a non-blocking single call

Before fanning out, prove the happy path is async all the way down. In .NET, HttpClient is the only network primitive you should be wrapping. Do not instantiate a new HttpClient per request; use a singleton or IHttpClientFactory. The code below posts to an OpenAI-compatible chat endpoint and deserializes the minimal shape we care about.

public sealed class LlmClient
{
    private readonly HttpClient _http;
    public LlmClient(HttpClient http) => _http = http;

    public async Task<string> CompleteAsync(string prompt, CancellationToken ct)
    {
        var req = new
        {
            model = "gpt-4o-mini",
            messages = new[] { new { role = "user", content = prompt } }
        };
        using var resp = await _http.PostAsJsonAsync("v1/chat/completions", req, ct);
        resp.EnsureSuccessStatusCode();
        var doc = await resp.Content.ReadFromJsonAsync<ChatResponse>(cancellationToken: ct);
        return doc!.Choices[0].Message.Content;
    }
}

public record ChatResponse(Choice[] Choices);
public record Choice(Message Message);
public record Message(string Content);

The critical rule: never call .Result or .Wait() on these tasks. In ASP.NET Core, that synchronously blocks a thread-pool thread that is needed to resume the continuation, and you get the classic deadlock where the response never arrives. If you are writing a library, avoid ConfigureAwait(false) noise; in modern .NET the default is fine for app code, but in a shared library it can still reduce context switches.

Fan out with Task.WhenAll

The simplest c# async await concurrent llm calls pattern is to launch a task per independent prompt and await them together.

var prompts = new[] { "summarize A", "summarize B", "summarize C" };
var tasks = prompts.Select(p => client.CompleteAsync(p, ct)).ToArray();
string[] results = await Task.WhenAll(tasks);

Materialize the tasks with ToArray() before passing to WhenAll. Select is lazy; if you enumerate twice you will issue duplicate calls. Task.WhenAll will enumerate once, but being explicit costs nothing and makes the concurrency boundary visible during code review.

Trade-off: this pattern issues prompts.Length simultaneous HTTP requests. If you are calling a single provider with a 20 RPM limit, you will trip rate limits instantly. Unbounded fan-out is fine for a handful of items, disastrous at scale.

Bound concurrency with SemaphoreSlim

The correct throttle is SemaphoreSlim, not Task.Run or Parallel.ForEachAsync (though the latter is acceptable in .NET 6+ if you prefer it). A semaphore gives you a maximum concurrent request count and respects cancellation.

private readonly SemaphoreSlim _sem = new(5, 5); // max 5 concurrent

public async Task<string> CompleteThrottledAsync(string prompt, CancellationToken ct)
{
    await _sem.WaitAsync(ct);
    try
    {
        return await CompleteAsync(prompt, ct);
    }
    finally
    {
        _sem.Release();
    }
}

Now fan out safely:

var tasks = prompts.Select(p => CompleteThrottledAsync(p, ct)).ToArray();
await Task.WhenAll(tasks);

Pitfall: if CompleteAsync throws, the finally still releases the semaphore—good. But if you forget WaitAsync(ct) and use Wait(ct) you block a thread. Always use the async wait. Also, SemaphoreSlim is not fair; tasks queue in an internal list and wake in arbitrary order. For LLM workloads that is fine; fairness rarely matters when all requests cost similar tokens.

If you are on .NET 6+, Parallel.ForEachAsync is a concise alternative:

await Parallel.ForEachAsync(prompts, new ParallelOptions { MaxDegreeOfParallelism = 5, CancellationToken = ct },
    async (p, ct2) => { var r = await client.CompleteAsync(p, ct2); /* store */ });

It handles the semaphore internally. I still prefer the explicit SemaphoreSlim when I need to mix throttled and unthrottled paths.

Isolate failures per task

Task.WhenAll surfaces an AggregateException only after all tasks finish, but if you need partial success you must catch inside each task. Otherwise one bad prompt cancels the entire batch mentally, even though the others completed.

public async Task<(string? Result, Exception? Error)> TryCompleteAsync(string p, CancellationToken ct)
{
    try { return (await CompleteThrottledAsync(p, ct), null); }
    catch (OperationCanceledException) { throw; }
    catch (Exception ex) { return (null, ex); }
}

var results = await Task.WhenAll(prompts.Select(p => TryCompleteAsync(p, ct)).ToArray());
var successes = results.Where(r => r.Error == null).Select(r => r.Result!).ToArray();

Do not swallow OperationCanceledException; let cancellation propagate so the caller can short-circuit. For other exceptions, log the status code. A 429 means you should back off; a 5xx might mean the model is degraded. The c# async await concurrent llm calls code should feed that signal into a retry policy (e.g., Polly or a simple exponential backoff), but never retry blindly inside the throttled loop without incrementing a counter.

Cancellation and timeouts

Every method above takes a CancellationToken. Wire it from your ASP.NET endpoint or background service. For per-call timeouts, link a short token:

using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(TimeSpan.FromSeconds(30));
await CompleteAsync(prompt, cts.Token);

The default HttpClient.Timeout is 100 seconds, which is far too long for an interactive LLM feature. Set it to 30–60s on the client, and use token timeouts for finer control. If you cancel mid-flight, the underlying TCP connection is released back to the pool; do not assume the provider stopped generating—you may still be billed.

Concurrent streaming without deadlocks

When you need tokens as they arrive, set stream: true and parse server-sent events. Use ResponseHeadersRead so you do not buffer the whole response.

public async IAsyncEnumerable<string> StreamAsync(string prompt, [EnumeratorCancellation] CancellationToken ct)
{
    using var req = new HttpRequestMessage(HttpMethod.Post, "v1/chat/completions");
    req.Content = JsonContent.Create(new
    {
        model = "gpt-4o-mini",
        stream = true,
        messages = new[] { new { role = "user", content = prompt } }
    });
    using var resp = await _http.SendAsync(req, HttpCompletionOption.ResponseHeadersRead, ct);
    resp.EnsureSuccessStatusCode();
    using var stream = await resp.Content.ReadAsStreamAsync(ct);
    using var reader = new StreamReader(stream);
    while (!reader.EndOfStream && !ct.IsCancellationRequested)
    {
        var line = await reader.ReadLineAsync(ct);
        if (line?.StartsWith("data:") == true) yield return line[5..];
    }
}

Consuming multiple streams concurrently is just another WhenAll:

async Task DrainAsync(IAsyncEnumerable<string> s, string tag, CancellationToken ct)
{
    await foreach (var chunk in s.WithCancellation(ct))
    {
        // process or buffer chunk
    }
}
var streams = prompts.Select(p => StreamAsync(p, ct)).ToArray();
await Task.WhenAll(streams.Select((s, i) => DrainAsync(s, $"p{i}", ct)));

Pitfall: if you buffer every chunk from every stream into a List before processing, memory grows with token count × concurrency. Consume and persist incrementally, or cap the number of concurrent streams to something your RAM budget allows.

Routing and fallback

If you front your inference with a gateway, you can send provider routing hints in the request body. For example, n4n.ai exposes an OpenAI-compatible endpoint that automatically falls back when a provider is rate-limited or degraded; your c# async await concurrent llm calls should still cap concurrency to avoid hammering the fallback path. Keep client timeouts slightly above expected fallback latency so a slow failover does not look like a hang.

Even without a gateway, honor the cache-control hints some providers accept to reuse prompt prefixes. Forwarding those headers from your client is a one-line req.Headers.Add and can cut tail latency dramatically on repeated system prompts.

Logging token usage

The response JSON includes a usage object. Capture it in your ChatResponse and log per call:

public record Usage(int PromptTokens, int CompletionTokens);
// after deserialize: _logger.LogInformation("tok={P}+{C}", u.PromptTokens, u.CompletionTokens);

In a batch, sum the successes to report cost. Per-token metering at the gateway is convenient for billing, but local logs let you correlate spikes with specific prompt templates. Do not log full prompt contents in production unless you have a redaction policy; LLM inputs are frequently PII.

Ordered checklist

1. Register a single HttpClient (or use IHttpClientFactory).
2. Write CompleteAsync returning Task<string> with CancellationToken.
3. Fan out with .Select(...).ToArray() then Task.WhenAll.
4. Throttle with SemaphoreSlim (or Parallel.ForEachAsync) to a limit your provider allows.
5. Wrap each call in a try/catch that preserves cancellation and records errors.
6. Apply linked token timeouts shorter than HttpClient.Timeout.
7. For streaming, use IAsyncEnumerable + ResponseHeadersRead and drain concurrently.
8. Log usage and latency; keep concurrency bounded when a fallback gateway is in play.

Follow that path and your c# async await concurrent llm calls will stay responsive under load instead of quietly exhausting the thread pool.