LLM Gateways

An LLM (large language model — a program that generates text by predicting the next token, where a token is a chunk of text roughly the size of a short word or word-piece) is offered by many providers, and each one exposes a different HTTP API, a different pricing table, a different rate limit, and a different set of quirks. The moment an application wants to use more than one model — say a small cheap model for easy work and a large model for hard work, or a second provider as a backup — it accumulates a mess: several API keys scattered through the code, several request formats, several retry strategies, and no single place to see what any of it costs.

An LLM gateway is a single service that sits between your application and all those providers and presents one API for all of them. Your code calls the gateway; the gateway calls whichever provider is right for that request. It is the messaging-broker idea from earlier in this course applied to model calls: one hop of indirection that decouples the caller from the thing that actually does the work.

An ordinary API gateway is a proxy in front of your own backend services that centralizes cross-cutting concerns — authentication, rate limiting, routing, and observability — so each service does not reimplement them. An LLM gateway is the same shape aimed at a narrower target: instead of fronting your microservices, it fronts external (and internal) model providers, and it adds concerns that only make sense for models — token accounting, semantic caching, prompt/response tracing, and model fallback. Same architectural instinct (one place for shared concerns), different domain.

The core job is a single request/response schema that maps onto every provider. Your application speaks one format — typically a list of role-tagged messages plus parameters like model, temperature, and max_tokens — and the gateway translates that into each provider's native shape and translates the reply back. Switching from one model to another becomes a one-line change of the model field, not a rewrite.

{
  "model": "vendor-neutral-model-name",
  "messages": [
    { "role": "system", "content": "You are a concise assistant." },
    { "role": "user", "content": "Summarize this in one sentence." }
  ],
  "max_tokens": 200,
  "temperature": 0.2
}

The response is normalized too, so the caller always reads the same fields regardless of which provider served it:

{
  "id": "resp_abc123",
  "model": "provider-a/model-large",
  "choices": [{ "message": { "role": "assistant", "content": "..." } }],
  "usage": { "prompt_tokens": 812, "completion_tokens": 47 },
  "provider": "provider-a",
  "cached": false
}

The usage block (token counts) and the provider/cached fields are what make everything downstream — billing, budgets, tracing — possible, because they report exactly what happened.

Once every provider speaks one schema, the gateway can choose which one handles a request, and re-choose when something fails. Three related behaviors live here.

Routing picks a target based on rules: cheapest model that meets a quality bar, a specific model for a specific customer tier, or the smallest model that can do the task. Fallback / failover means that when the chosen provider errors, times out, or is rate-limited, the gateway automatically retries the same logical request against a different provider, so a single provider outage does not become your outage. Load balancing spreads requests across several providers or several keys to stay under each one's rate limit and to smooth latency.

The load-bearing idea is that fallback turns a set of independently unreliable providers into one more-reliable virtual endpoint — the same reasoning as replicating a database, applied to model calls.

Because every call reports its token usage, the gateway is the natural place to count and to cap. Rate limiting bounds how many requests or tokens per second a caller may use, protecting both your budget and the upstream provider's limits. Quotas cap usage over a longer window (per team, per API key, per customer) so one runaway loop cannot spend the whole month's budget in an hour.

Cost tracking + budgets build directly on the usage numbers: the gateway multiplies tokens by each model's price, attributes the cost to a team or feature via the API key, and can refuse a request when a budget is exhausted. This is the single biggest operational reason teams adopt a gateway — without one, LLM spend is invisible until the invoice arrives.

Model calls are slow and paid-per-token, so avoiding a call is the cheapest optimization there is. The gateway can cache responses in two ways. An exact cache returns a stored answer when a new request is byte-for-byte identical to a past one — simple, but it misses near-duplicates. A semantic cache goes further: it converts the prompt into an embedding (a list of numbers that represents the prompt's meaning, so that texts with similar meaning have nearby number-lists) and returns a cached answer when a new prompt's embedding is close enough to a stored one — so "capital of France?" and "what's France's capital" can share a cached reply.

Semantic caching trades a small risk (two prompts judged similar that actually differ) for large savings, so you tune the similarity threshold and disable it where exactness matters.

Two more concerns fall naturally to the gateway because every request already passes through it. Observability means tracing each call — the prompt, the chosen model, token counts, latency, cost, cache hit/miss, and any fallback that fired — so you can debug quality regressions and see spend per feature. Since the gateway is the choke point, it produces a complete, uniform log across every provider, which scattered direct calls never could.

Key management + secret isolation means the provider API keys live only inside the gateway, never in application code or client devices. Applications authenticate to the gateway with their own key; the gateway holds the real provider secrets and can rotate them centrally. A leaked application key can be revoked without touching provider credentials, and no client ever sees a provider secret.

The final band of responsibilities makes calls safe and resilient. Guardrails are checks the gateway runs on the way in and out: redacting PII (personally identifiable information — names, emails, card numbers) from prompts before they reach a provider, and scanning responses for disallowed content. Centralizing guardrails means every application gets the same protection without each one reimplementing it.

On resilience, the gateway applies retries with exponential backoff and jitter for transient errors (the same discipline used for message queues), and a circuit breaker — a switch that, after a provider fails repeatedly, stops sending it traffic for a cooldown window and routes to a fallback instead, so a struggling provider is given room to recover rather than being hammered.

Gateways exist as managed services (for example Vercel AI Gateway, now generally available) and as self-hostable open-source proxies (for example LiteLLM, Portkey, and the model-router layers offered by the major clouds). They differ in packaging, not in the core idea: a normalized API with routing, fallback, budgets, caching, and tracing.

You do not need a gateway on day one. A single app calling a single model is fine talking to the provider directly. Adopt a gateway when a second provider or model enters the picture, when you need spend visibility and budgets, or when multiple teams share model access and you want one place for keys, limits, and guardrails. As with any proxy, the cost is one more hop and one more thing to run — justified once the cross-cutting concerns it centralizes would otherwise be copy-pasted everywhere.

This table collapses the day into concern-to-responsibility pairs. Each row is a job the gateway centralizes so applications do not each reimplement it.

• An LLM gateway is a unified proxy in front of many model providers, presenting one API, key, and control point — the messaging-broker idea applied to model calls.
• It is a classic API gateway aimed at model providers, adding model-specific concerns: token accounting, semantic caching, prompt/response tracing, and model fallback.
• A normalized request/response schema (with a usage token count) makes switching models a one-line change and powers billing, budgets, and tracing.
• Routing picks a model, fallback/failover retries a failed request on another provider, and load balancing spreads traffic — together turning unreliable providers into one reliable endpoint.
• Because every call reports tokens, the gateway is where rate limits, quotas, cost tracking, and hard budgets live.
• Exact and semantic caching (matching prompts by meaning via embeddings) cut latency and spend, with a tunable similarity threshold.
• The gateway centralizes observability, key/secret isolation, PII-redaction guardrails, retries, and circuit breaking, so applications inherit them for free.
• Adopt one when a second provider appears, when spend must be visible, or when many teams share model access — not before.

• [ ] I can explain why calling many providers directly accumulates keys, formats, limits, and blind spots.
• [ ] I can describe how an LLM gateway relates to, and differs from, a classic API gateway.
• [ ] I can read a normalized request/response and identify the fields that enable billing and tracing.
• [ ] I can distinguish routing, fallback/failover, and load balancing.
• [ ] I can place rate limits, quotas, and budgets and explain what each protects.
• [ ] I can explain exact vs semantic caching and the threshold tradeoff.
• [ ] I can name the resilience and safety concerns a gateway centralizes: guardrails, retries, circuit breaking, key isolation.
• [ ] I can decide when a team should adopt a gateway versus call a provider directly.