Building a reliable ihttpclientfactory llm api dotnet integration starts with admitting that new HttpClient() in a loop is a scalability bug, not a quick win. LLM endpoints hold connections open for seconds during generation, amplify socket exhaustion, and punish naive retry logic. IHttpClientFactory gives you pooled handlers and a clean injection surface, but only if you configure it for the specific shape of LLM traffic.
1. Register typed clients instead of named ones
Named clients (CreateClient("llm")) work, but they push configuration and HttpClient access into every consumer. Typed clients bind the HttpClient directly to an interface, making dependencies explicit and testable.
public interface ILlmClient
{
Task<ChatCompletion> CompleteAsync(ChatRequest req, CancellationToken ct);
}
public class OpenAiCompatibleClient : ILlmClient
{
private readonly HttpClient _http;
public OpenAiCompatibleClient(HttpClient http) => _http = http;
public async Task<ChatCompletion> CompleteAsync(ChatRequest req, CancellationToken ct)
{
var response = await _http.PostAsJsonAsync("chat/completions", req, ct);
response.EnsureSuccessStatusCode();
return await response.Content.ReadFromJsonAsync<ChatCompletion>(ct);
}
}Registration is a one-liner in Program.cs:
builder.Services.AddHttpClient<ILlmClient, OpenAiCompatibleClient>(client =>
{
client.BaseAddress = new Uri("https://api.example-llm.com/v1/");
client.Timeout = TimeSpan.FromSeconds(120);
});The ihttpclientfactory llm api dotnet pattern shines here because the factory manages handler lifetime; you inject ILlmClient and never touch new.
2. Tune the primary handler for long-lived generations
Default handler settings assume short HTTP calls. LLM completions can stream for minutes. Configure SocketsHttpHandler to control pooling and HTTP/2.
builder.Services.AddHttpClient<ILlmClient, OpenAiCompatibleClient>()
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromMinutes(5),
MaxConnectionsPerServer = 100,
EnableMultipleHttp2Connections = true,
KeepAlivePingPolicy = HttpKeepAlivePingPolicy.WithActiveRequests,
});PooledConnectionLifetime forces periodic reconnects so DNS changes propagate. Without it, a pod pointing at a stale IP will fail silently after a provider failover. MaxConnectionsPerServer caps concurrency per instance; set it based on your rate limits, not arbitrarily high. A too-low value causes queueing; a too-high value gets you throttled.
3. Stream responses with ResponseHeadersRead
Buffering a 2-minute completion in memory will crash your service under load. Use HttpCompletionOption.ResponseHeadersRead and parse Server-Sent Events yourself.
public async IAsyncEnumerable<string> StreamAsync(ChatRequest req,
[EnumeratorCancellation] CancellationToken ct)
{
using var httpReq = new HttpRequestMessage(HttpMethod.Post, "chat/completions")
{
Content = JsonContent.Create(req with { Stream = true })
};
using var resp = await _http.SendAsync(httpReq,
HttpCompletionOption.ResponseHeadersRead, ct);
resp.EnsureSuccessStatusCode();
await using var stream = await resp.Content.ReadAsStreamAsync(ct);
using var reader = new StreamReader(stream);
while (!reader.EndOfStream && !ct.IsCancellationRequested)
{
var line = await reader.ReadLineAsync(ct);
if (line?.StartsWith("data:") == true)
yield return line["data:".Length..].Trim();
}
}Tradeoff: you lose ReadFromJsonAsync convenience and must handle SSE framing, partial JSON, and heartbeats. The memory savings are non-negotiable at scale. If you must buffer (e.g., for a non-streaming proxy), set a hard MaxResponseContentBufferSize and fail fast.
4. Layer resilience without hiding latency
LLM providers rate-limit aggressively. Use Microsoft.Extensions.Http.Resilience (or Polly) to add retries and circuit breaking, but cap attempt counts to avoid stacking tail latency.
dotnet add package Microsoft.Extensions.Http.Resiliencebuilder.Services.AddHttpClient<ILlmClient, OpenAiCompatibleClient>()
.AddStandardResilienceHandler(options =>
{
options.Retry.MaxRetryAttempts = 2;
options.Retry.BackoffType = BackoffType.Exponential;
options.Retry.ShouldHandle = new Predicate<HttpResponseMessage>(r =>
r.StatusCode is HttpStatusCode.TooManyRequests or
HttpStatusCode.InternalServerError or
HttpStatusCode.BadGateway or
HttpStatusCode.GatewayTimeout);
options.CircuitBreaker.FailureRatio = 0.5;
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
});A common pitfall: retrying non-idempotent streaming requests. Only retry before the first byte arrives; once headers are read, a retry wastes tokens and double-sends. Gate retries on !response.Content.Headers.ContentLength.HasValue or use a custom policy that checks a flag set after SendAsync returns headers.
5. Forward routing directives and cache-control hints
Some gateways let clients pin a model or region via headers. If you call an OpenRouter-class endpoint, honor provider cache-control hints to avoid recomputing expensive prompts.
var req = new HttpRequestMessage(HttpMethod.Post, "chat/completions")
{
Content = JsonContent.Create(chatReq)
};
req.Headers.Add("Cache-Control", "max-age=3600");
req.Headers.Add("X-Route-Model", "gpt-4o-mini");The ihttpclientfactory llm api dotnet stack should place these in a DelegatingHandler so they apply uniformly:
public class RoutingHandler : DelegatingHandler
{
protected override Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken cancellationToken)
{
request.Headers.Add("X-Client-Trace", "dotnet-worker");
return base.SendAsync(request, cancellationToken);
}
}Register with .AddHttpMessageHandler<RoutingHandler>(). Keep handlers order-aware: routing before retry, metering after.
6. Capture token metering from response headers
Cost visibility starts at the HTTP boundary. If your provider returns per-token usage in headers, extract it in a handler. Gateways such as n4n.ai expose a single OpenAI-compatible endpoint with automatic fallback and per-token usage metering in response headers, which you can pipe into internal accounting.
public class UsageMeterHandler : DelegatingHandler
{
private readonly IMetrics _metrics;
public UsageMeterHandler(IMetrics metrics) => _metrics = metrics;
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request, CancellationToken ct)
{
var response = await base.SendAsync(request, ct);
if (response.Headers.TryGetValues("x-token-usage", out var vals))
{
var usage = int.Parse(vals.First());
_metrics.RecordTokens(usage);
}
return response;
}
}Do not block the response pipeline to write to a slow database; enqueue and return. Meter at the edge, aggregate later.
7. Avoid the classic dotnet HTTP footguns
- Never wrap the injected HttpClient in a
usingblock. The factory owns its lifetime. Disposing it tears down the shared handler. - Don’t set
Timeoutto 1 second “just in case”. LLM cold starts exceed that. Use per-request timeouts viaCancellationTokenif you need finer control. - Don’t log full request/response bodies in production. Prompt contents are PII and can be massive; use redacted metadata.
- Don’t create a static
HttpClient“for performance” and skip the factory. You lose DNS rotation and handler injection.
In any ihttpclientfactory llm api dotnet project, these mistakes surface only under production load. The ordered path above—typed clients, tuned handlers, streaming, resilience, routing, metering, and discipline—keeps the service stable when the model endpoint blinks.
8. Test with a mocked handler
Finally, validate behavior by swapping the primary handler in tests:
builder.Services.AddHttpClient<ILlmClient, OpenAiCompatibleClient>()
.ConfigurePrimaryHttpMessageHandler(() => new FakeHandler());This catches header propagation and streaming parsing without hitting a real provider. Ship the factory config, not guesswork.