How indie hackers pick an LLM API gateway

How indie hackers choose LLM API gateway is less about chasing the cheapest token and more about reducing operational surface area while keeping model optionality. You are one person or a two-person team; every hour spent wiring provider SDKs is an hour not spent on product. This guide gives a concrete, ordered path to pick a gateway that won’t blow up your pager or your burn rate.

Step 1: Map your real traffic before anything else

The first filter in how indie hackers choose LLM API gateway is traffic reality, not vendor marketing. You cannot evaluate coverage or fallback logic until you know what you actually send.

Log a week of production-shaped calls

If you already have users, shadow-log their requests in staging. If you are pre-launch, script your expected workloads.

import json, time, openai

client = openai.OpenAI()  # direct baseline, pre-gateway

def logged_chat(model, messages):
    t0 = time.time()
    resp = client.chat.completions.create(model=model, messages=messages)
    latency = time.time() - t0
    print(json.dumps({
        "model": model,
        "latency_ms": round(latency * 1000),
        "prompt_tokens": resp.usage.prompt_tokens,
        "completion_tokens": resp.usage.completion_tokens
    }))
    return resp

# run against representative inputs
logged_chat("gpt-4o-mini", [{"role": "user", "content": "classify: spam?"}])

Aggregate by model, token volume, and error type. Most indie apps find 80% of calls hit one or two models, with long-tail experiments elsewhere.

Write down two hard numbers

• Max acceptable p95 latency for interactive features (e.g., 800 ms).
• Max $/1M output tokens for background jobs (e.g., $10).

A gateway cannot violate physics; it only routes. If your p95 target is 300 ms, a 240-model menu that includes slow open-weight endpoints is a liability unless you pin routing.

Step 2: Require an OpenAI-compatible surface

The second axis in how indie hackers choose LLM API gateway is protocol compatibility. If the gateway forces a new SDK or request shape, you inherit a migration tax every time you tweak a call.

Minimal swap test

You should be able to change only base_url and api_key:

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.example.com/v1",
    api_key="your-key"
)

resp = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[{"role": "user", "content": "Summarize this PR"}],
    temperature=0.2
)

A gateway such as n4n.ai provides one OpenAI-compatible endpoint addressing 240+ models with automatic fallback when a provider is rate-limited or degraded, plus per-token usage metering, which lets you keep the above code unchanged while gaining redundancy and accounting.

Caveats to test on day one

• Does it forward response_format for JSON mode?
• Does streaming work with your existing stream=True loops?
• Are function_call / tools schemas passed untouched?

Some “compatible” gateways silently drop edge parameters. Write a parity test before you trust them.

Step 3: Insist on explicit fallback and routing control

Automatic fallback sounds great until your legal summarizer gets routed to a model that hallucinates citations. You need declarative routing, not a black box.

Send routing directives in the request

A sane gateway honors client routing directives and forwards provider cache-control hints. That means you can express preferences without writing retry loops.

{
  "model": "openai/gpt-4o",
  "messages": [{"role": "user", "content": "Cache this"}],
  "route": {
    "prefer": ["openai"],
    "fallback": ["azure-openai"],
    "on": [429, 503]
  },
  "cache_control": {"type": "ephemeral", "ttl": 3600}
}

If the gateway ignores route, you are back to hand-rolling fallback. That is fine for a hackathon, not for a paid product.

Verify degradation behavior

Write a chaos check that blocks your primary provider and confirms the gateway shifts traffic.

# simulate provider outage in your env
export PRIMARY_PROVIDER_BLOCKED=1
python test_routing.py  # asserts response.model != primary

Tradeoff: explicit routing gives safety but adds a small config burden. For a solo dev, a single default route with one fallback is enough.

Step 4: Meter per token, not per request

Indie budgets die on silent token spikes. Your gateway must return accurate usage objects and ideally let you query spend by model after the fact.

resp = client.chat.completions.create(
    model="meta/llama-3-70b",
    messages=[{"role": "user", "content": "Generate 10 variants"}]
)
# per-token metering means you can alert on this locally
if resp.usage.completion_tokens > 2000:
    print("unexpected generation size", resp.usage.completion_tokens)

If the gateway aggregates only at request granularity, you cannot attribute cost to a specific feature. That is a non-starter for a solo dev who needs to know why the bill doubled.

Ask the metering questions

• Does the usage object include prompt_tokens and completion_tokens from the upstream provider unmodified?
• Can you pull a daily breakdown by model via API or dashboard?
• Are cached token counts reported separately?

Without per-token fidelity, you are flying blind.

Step 5: Prototype with a thin abstraction

Do not marry the gateway. Write a 30-line wrapper that sets routing headers and logs usage. This makes future switches a one-file diff.

class GatewayClient:
    def __init__(self, base_url, key):
        self.client = OpenAI(base_url=base_url, api_key=key)

    def complete(self, model, messages, fallback_models=None):
        body = {"model": model, "messages": messages}
        if fallback_models:
            body["route"] = {
                "prefer": [model],
                "fallback": fallback_models,
                "on": [429]
            }
        return self.client.chat.completions.create(**body)

gw = GatewayClient("https://gateway.example.com/v1", "key")
gw.complete(
    "openai/gpt-4o",
    [{"role": "user", "content": "hi"}],
    fallback_models=["anthropic/claude-3.5-sonnet"]
)

This pattern isolates gateway-specific quirks. If you later move to a different vendor, only this class changes. Your product code stays on the OpenAI interface.

Common pitfalls indie hackers hit

Assuming one model fits all

You will tempt yourself to standardize on a single cheap model. Reality: classification wants small/fast, UX copy wants creative. Gateways make multi-model easy, so use it instead of forcing a square peg.

Ignoring cache-control hints

Providers like Anthropic and OpenAI support prompt caching. If your gateway strips cache_control, you pay full price on every repeated system prompt. Forwarding those hints is a technical requirement, not a nice-to-have.

Not handling 429s upstream

Even with gateway fallback, your client must respect Retry-After. The gateway reduces but does not eliminate rate limits.

import time
def safe_call(fn, retries=3):
    for i in range(retries):
        try:
            return fn()
        except openai.RateLimitError as e:
            time.sleep(2 ** i)
    raise

Forgetting streaming timeouts

Long completions over streaming can hang if the gateway does not propagate provider timeouts. Set timeout on your client and test with a 10k-token generation.

Decision checklist

Use this ordered path as your eval script:

1. List models in production today and their token volumes.
2. Confirm gateway speaks OpenAI-compatible API with your edge params (JSON mode, tools).
3. Send a routed request with fallback directive; block primary and verify switch.
4. Check usage object fidelity and spend query API for per-token breakdown.
5. Wrap in thin client; time to swap is your lock-in score.

How indie hackers choose LLM API gateway boils down to: minimize code surface, keep model optionality, and make fallback explicit. Do those three and the rest is just token math.