Design an LLM Gateway (rate limiting, routing, and caching for multiple LLM providers) — Meta Interview

An LLM gateway sounds like a thin proxy. Take a request, forward it to OpenAI or Anthropic, return the response. Then you realise the thing you're metering isn't requests. It's tokens. And you don't know how many tokens a response costs until after it finishes streaming. So your rate limiter has to admit traffic based on a number that doesn't exist yet.

That's what makes LLM gateway system design one of the most interesting new questions in senior interview loops. AI infrastructure questions have moved out of ML-specialist interviews and into general system design rounds, and this one sits at a sweet spot. It looks like the classic rate limiter design question, but every assumption that made that problem clean is broken here. Requests have wildly different costs. The API you depend on rate-limits you. Responses stream for thirty seconds.

Candidates who explain what to build pass. Candidates who explain why each design decision was made get offers.

That gap — between knowing the answer and explaining it clearly under pressure — is exactly what Mockingly.ai is designed to close. But first, let's build the design.

Step 1: Clarify the Scope

Interviewer: Our company has about forty product teams calling OpenAI, Anthropic, and an internal fine-tuned model directly. Costs are unpredictable, teams keep tripping provider rate limits, and nobody can tell me which team spent what last month. Design an LLM gateway that sits between our services and the providers.

Candidate: A few questions before I draw anything. How many requests a day, across how many teams? Is streaming in scope? Are budgets hard caps, or am I just smoothing traffic? How much latency can the gateway add? And is caching responses okay from a product point of view?

Interviewer: Ten million requests a day across roughly forty teams. Streaming is in scope, most of the traffic is user-facing chat. Yes to hard budgets, finance wants per-team caps. The gateway should add no more than about 20 ms of p99 overhead on top of provider latency. Caching is fine where it's safe. Single region is fine for now.

Candidate: Good. The 20 ms budget rules out synchronous billing lookups on the hot path, so admission has to come from in-memory state. Redis token buckets, exact accounting done async. And since most traffic streams, my limiter has to work off token estimates. Quick requirements and numbers, then I'll go deep on four things: token-aware limiting, fairness, routing and fallback, and caching.

What the candidate did here:

• Every question maps to a design decision, not curiosity — streaming changes failover, budgets change accounting, the latency cap kills synchronous billing checks.
• They repeated the constraints back and said out loud what those constraints eliminate.
• They committed to a plan for the next forty minutes, which tells the interviewer the candidate is driving, not drifting.
• They flagged the hardest problem (estimating tokens you can't count yet) before the interviewer had to.

Requirements

Functional

• Unified API — one endpoint and one request schema; the gateway translates to each provider's format (OpenAI, Anthropic, internal model)
• Team-level authentication — teams get gateway keys; provider keys live only inside the gateway
• Token-aware rate limiting — per-team limits on requests per minute (RPM) and tokens per minute (TPM)
• Budget enforcement — hard monthly spend caps per team
• Routing and fallback — send each request to the right provider and model; fail over when a provider errors or throttles
• Streaming passthrough — proxy server-sent events (SSE) token streams without breaking them
• Response caching — exact-match and semantic caching where the product allows it
• Usage and cost attribution — every token accounted to a team, queryable

Non-Functional

• Low overhead — ≤ 20 ms added p99 latency; the gateway must never dominate a multi-second LLM call
• High availability — the gateway fronts every AI feature in the company; it's a single point of failure we chose to build, so it must be more available than any provider behind it
• Accounting correctness — admission can run on estimates, but the ledger must end up exact
• Fairness — one team's batch job must not starve another team's interactive traffic
• Security — provider API keys never leave the gateway; cached responses stay inside the team that created them

Back-of-the-Envelope Estimates

Interviewer: What are the numbers?

Candidate:

Traffic
  10M requests/day ÷ 86,400 s  ≈ 115 RPS average
  Peak factor ~3x              ≈ 350 RPS peak
 
Tokens (assume avg 1,200 input + 400 output per request)
  10M × 1,600 tokens           = 16B tokens/day
  16B ÷ 1,440 min              ≈ 11M tokens/minute average
                               ≈ 33M tokens/minute at peak
 
Rate-limit state
  40 teams × ~10 keys × a handful of counters (RPM, TPM, budget)
  → a few thousand Redis keys, kilobytes of state
 
Redis load
  2–3 atomic operations per request
  → ~1,000 ops/s at peak. A single Redis node handles 100k+ ops/s.

Candidate: Two things jump out. The gateway itself is trivial — 350 RPS, kilobytes of state. But 33 million tokens a minute at peak is more than a single provider account typically allows. So the hard problem isn't throughput. It's a policy problem: whose quota each request burns, and which provider can absorb it.

What the candidate did here:

• The math is simple and rounded, because nobody wants to watch long division in an interview.
• Requests and tokens are estimated separately, because they stress different parts of the system.
• The punchline matters more than the numbers: gateway throughput is easy, provider quotas and cost are the real constraints.
• Calling it "a policy problem" tells the interviewer exactly where the deep dives are about to go.

Why Request-Based Rate Limiting Fails for LLM Traffic

Interviewer: Hold on. We already rate-limit at our existing API gateway. Why can't I just cap each team at some number of requests per second and be done?

Candidate: Because two requests can differ in cost by a factor of a thousand. One team summarises short comments at 100 tokens a request. Another pastes whole contracts at 80,000. Same request count, wildly different cost, and the limiter can't see it. That's why the providers enforce request and token limits side by side — OpenAI with RPM and TPM, Anthropic splitting tokens into separate input and output limits. Cross any one and you get a 429. The gateway has to mirror that.

Interviewer: Fine. But we're a client of these providers too. Doesn't their rate limiter already solve this for us?

Candidate: It solves it for them. If I let the provider 429 us, I've already paid the latency and I can't choose which team's requests fail. The gateway makes that call before the request leaves the building. It's a rate limiter sandwiched between two other rate limiters — our clients below, the provider above.

Why this answer works:

• It names the exact failure, variable cost per request, instead of vaguely saying request limits "don't scale".
• The contract-processing example gives the interviewer a concrete picture they'll remember.
• Citing how OpenAI and Anthropic actually behave shows the candidate has used these APIs, not just read about them.
• The "sandwiched between two rate limiters" line hands the interviewer a frame for the entire session.
• The candidate treats the provider's 429 as a cost to avoid, not an error to handle, which is a senior-level shift in framing.

Token-Aware Rate Limiting: Estimate, Admit, Reconcile

This is the heart of the question, and the part of the interview where you can actually stand out.

Interviewer: Your TPM bucket needs to charge tokens when the request arrives. But you just told me output tokens are unknown until the response ends, maybe thirty seconds later. How do you charge for something you can't count yet?

Candidate: Charge an estimate, fix the books later. Input tokens I can count up front. Output I can't, so I reserve the request's max_tokens as the worst case. If the bucket doesn't have room, reject right there — before we've spent a single paisa with the provider. When the response finishes, the provider reports actual usage and I credit back the difference. Reserved 4,096, generated 300? 3,796 goes back in the bucket.

Interviewer: Why does the reservation need to be atomic?

Candidate: Traffic fans across many gateway instances. Check-then-decrement as two Redis calls is a race — two instances both see room, both admit. A Lua script does it as one unit, and checks RPM, TPM, and budget in a single round trip. That's how I stay under 20 ms.

Why this answer works:

• "Charge an estimate, fix the books later" compresses the whole mechanism into one sentence the interviewer can grade.
• Using max_tokens as the reservation is the honest worst case, and the credit-back keeps it from being permanently wasteful.
• The candidate rejects before spending provider money, which shows they understand where the costs actually are.
• The atomicity answer names the concrete race, not just "for safety".
• Bundling all three checks into one Lua call ties back to the 20 ms budget from the scope conversation — constraints are still steering decisions twenty minutes later.

Spelled out, the mechanism has three phases. Estimate: count input tokens before the call (the provider's tokeniser, or a chars-divided-by-four approximation if the tokeniser is too slow for the hot path) and take max_tokens as the output worst case. Admit: reserve that estimate against the team's TPM, RPM, and budget buckets in one atomic Redis call; if any bucket is short, return a 429 with a Retry-After. Reconcile: when the response completes, credit back the gap between reserved and actual.

This estimate-and-reconcile pattern is how production systems do it. Azure API Management's llm-token-limit policy, for example, pre-estimates prompt tokens so it can reject over-limit requests before they reach the backend, then corrects against the actual usage in the response. For streamed responses it estimates on both ends, because the true count only shows up in the final stream event.

Two failure modes to raise before the interviewer does:

Over-reservation starves throughput. If clients habitually set max_tokens to the model maximum "just in case", every request reserves thousands of tokens it won't use. Between admission and reconciliation the bucket looks empty, and legitimate traffic gets 429'd. Mitigations: enforce sensible max_tokens defaults at the gateway, or reserve a discounted estimate based on each team's history of actual-versus-reserved usage, accepting a small chance of overshoot in exchange for throughput. This isn't hypothetical, by the way — OpenAI's own rate limiter counts your request as the maximum of max_tokens and its estimated size, and their docs explicitly advise setting max_tokens close to your expected response size. Providers face the same over-reservation problem you do.

Orphaned reservations leak the bucket. If a gateway instance crashes after reserving but before reconciling, those tokens are gone until the window refills. Attach a TTL to every reservation so unclaimed holds expire on their own.

Interviewer: Why token buckets and not sliding window counters?

Candidate: Buckets handle variable cost naturally — reserving 5,000 tokens is just decrementing by 5,000. They allow bursts, which is how LLM traffic actually arrives: paste a document, fire three requests, go quiet. A sliding window log at token granularity means storing every request with its weight and summing on each admission. More state, more work, no real gain. And the same bucket covers budgets — dollars instead of tokens, refilled monthly.

Why this answer works:

• It compares the options on this problem's specific needs (variable cost, bursty arrivals) rather than reciting generic algorithm trade-offs.
• It quantifies why the alternative is heavier instead of just calling it worse.
• Reusing the bucket for budgets shows the design is one mechanism applied twice, which interviewers read as taste.

If the interviewer wants precedent: Anthropic's own API documents that it rate-limits with the token bucket algorithm, continuously replenishing capacity rather than resetting at fixed intervals. Their limiter also estimates input tokens at the start of each request and adjusts to actuals as the request runs — estimate-and-reconcile, running in production at the provider you'd be calling through this gateway.

The LLM Gateway Architecture, End to End

With the core mechanism settled, here's the shape of the system:

        Clients (40 teams, gateway API keys)
                        │
                        ▼
┌───────────────────────────────────────────────────┐
│                   LLM GATEWAY                     │
│                                                   │
│  1. AuthN → resolve team, limits, budget          │
│  2. Cache lookup (exact-match → semantic) ── hit ─┼──► return cached response
│  3. Token estimator (input tokens + max_tokens)   │
│  4. Admission control                             │
│     Redis token buckets: RPM + TPM + budget ─ no ─┼──► 429 + Retry-After
│  5. Router: model map, provider health, cost      │
│  6. Provider adapter (schema translation)         │
└──────────┬──────────────┬──────────────┬──────────┘
           ▼              ▼              ▼
        OpenAI        Anthropic     Internal model
           │              │              │
           └──────────────┴──────────────┘
                          │
              7. Usage reconciliation (async)
                 → adjust buckets to actual tokens
                 → append to cost ledger
                 → metrics: TTFT, tokens/s, errors, cache hits

The request flow:

1. A client calls the gateway with its team key. The gateway resolves the team's limits, budget state, and routing policy from a local config cache, refreshed from a control plane in the background. Never fetched per request.
2. The gateway checks the cache. Exact-match first, since it's a cheap hash lookup, then optionally the semantic cache. A hit returns in milliseconds and burns no provider quota.
3. On a miss, the token estimator computes input tokens and takes max_tokens as the provisional output count.
4. Admission control reserves the estimate against the team's TPM, RPM, and budget buckets in one atomic Redis call. Any bucket empty means a 429 with a Retry-After hint.
5. The router picks a provider and model from the requested logical model, current provider health, and cost policy. The adapter translates the request into that provider's schema.
6. The response streams back through the gateway to the client.
7. After completion, reconciliation adjusts the buckets to actual usage and appends an exact record to the cost ledger.

Steps 2, 3, 4, and 5 are where the interesting decisions live. We've covered 3 and 4. The next three sections cover fairness, routing, and caching.

Multi-Tenant Fairness: The Noisy Neighbour Problem

Interviewer: Suppose every team is inside its own limits, and the provider is still throttling your account. What happened?

Candidate: The sum happened. The provider gives our account one shared quota, and my per-team limits add up to more than it. Team A kicks off a batch job — fully legal under A's limits — and Team B's chat users start watching spinners. Per-team buckets isolate teams from their own limits, not from each other. I need a second layer of admission at the shared quota.

Interviewer: And what does that layer look like?

Candidate: Cheapest version: two classes. Interactive passes straight through; batch waits in a queue that drains only when the shared quota has headroom. Users never wait behind a batch job, and batch still finishes — just slower during busy hours.

Interviewer: And if interactive traffic from two teams starts contending?

Candidate: Then weighted fair queuing. Each team gets a weight, and the scheduler serves teams in proportion to it — measured in tokens, not requests. Cheap requests get served often, 80,000-token monsters less often. Nobody starves, no capacity sits idle.

Why this answer works:

• "The sum happened" names the trap in two words: the shared provider quota is the resource nobody's individual limit protects.
• The batch-job story makes the failure concrete and shows why "everyone was within limits" isn't a defence.
• The candidate gives the cheap answer first and holds the fancy one in reserve, which is the right way to spend interview time.
• Measuring fairness in tokens, not requests, carries the article's core insight through into the scheduling layer.
• Naming a real precedent helps here: Cohere has written publicly about token-weighted, work-conserving fair queuing in their serving stack.

One more option worth naming just to dismiss it: statically splitting the provider quota, one fortieth per team. Predictable, simple, and wasteful, since quota reserved for idle teams sits unused while busy teams throttle. Say it, dismiss it, move on. The queue design itself is the same family of problem as a distributed task scheduler: priorities, retries, workers draining against a capacity signal.

Provider Routing, Fallback, and the Other Side of the 429

Interviewer: OpenAI just started returning 429s on every call. Walk me through the next sixty seconds.

Candidate: The first failing request reads Retry-After and waits at least that long — OpenAI's docs say to treat it as a minimum and add jitter on top. No header? Exponential backoff with jitter, a couple of attempts max. And failed requests still count against your request limit, so hammering retries digs the hole deeper. Meanwhile I'm counting failures per provider-model pair. Past a threshold, the circuit breaker opens: stop calling OpenAI, route straight to the fallback. Probes test the primary every few seconds; when they pass, the circuit closes.

Interviewer: Why the breaker? You already have retries.

Candidate: Retries are the problem at scale. Every gateway instance retrying into a degraded provider makes their outage worse — a retry storm. The breaker stops the herd.

Interviewer: Could you have seen it coming?

Candidate: Usually, yes. Providers report remaining quota on every successful response — OpenAI's x-ratelimit-remaining-tokens, Anthropic's equivalents. Feed those into admission and we slow ourselves down before the first 429. Reacting to 429s is table stakes. Steering by the headers is the version I'd want to run.

Why this answer works:

• The answer is ordered by time — first request, then the aggregate, then recovery — which makes a complex failure easy to follow.
• Honouring Retry-After instead of guessing shows respect for what the provider's signal actually means.
• The candidate explains why the circuit breaker exists (retry storms), not just that one should be there.
• The jump from reactive to proactive throttling is the difference between a good answer and a senior one.

The routing layer itself is simple on purpose. The unified API accepts a logical model name (chat-large, chat-fast, internal-legal), and a routing table maps each logical name to concrete provider models with an ordered fallback chain. That indirection is half the point of the gateway: when a team migrates models, it's one config change, not forty codebases.

One caveat to raise without being asked: fallback across providers is not free. Different models give different answers, tokenise differently (which shifts your cost accounting), and may have different compliance status for regulated workloads. Silent cross-provider failover is a product decision dressed up as an infrastructure feature. Make it per-route config: "this route may fall back to Anthropic; this legal-team route must fail closed."

Semantic Caching: When It Saves Money and When It Burns You

Caching LLM responses comes in two flavours, and mixing them up is a red flag.

Exact-match caching hashes the normalised request (model, system prompt, messages, temperature) and returns the stored response on an identical request. Cheap, safe, boring. Its weakness: users rarely phrase the same thing identically twice, so hit rates on free-form chat are poor.

Semantic caching embeds the incoming prompt as a vector, searches previously answered prompts for a near match, and serves the cached response when similarity clears a threshold. "How do I reset my password?" and "password reset — how?" land on nearly identical vectors. One inference call serves both.

Interviewer: Two prompts come in at 92% similarity. Same answer or different answer?

Candidate: Can't tell from the number, and that's the whole problem. "Reset my password" and "reset my password on iOS" can score that close and need different answers. So the threshold is a per-route knob, not a constant. I'd start strict, sample the hits, check whether the served answers actually fit, and loosen only where the data says it's safe. Some routes shouldn't have a semantic cache at all.

Interviewer: Which ones?

Candidate: Creative or high-temperature routes — users expect variety. Multi-turn chat — context makes every prompt unique, so hit rates go to zero anyway. And anything personalised, because the failure there isn't a wrong answer. It's showing one user something built from another user's data.

Why this answer works:

• Refusing to answer "same or different" from the score alone is the correct answer, and saying so directly builds trust.
• The password/iOS example shows why high similarity doesn't mean same intent.
• The candidate proposes measuring false positives rather than guessing a threshold, which turns a hand-wave into an operable plan.
• Listing where the cache should be off entirely shows judgement about the product, not just the machinery.
• The last line reframes personalised caching as a data-leak risk, which turns a performance discussion into a security one.

On the numbers: strict thresholds (0.95 and above) rarely misfire, but the hit rate collapses toward exact-match. Production gateways commonly default around 0.8 and tune per workload. And scope every cache key by team and system-prompt version — serving one team's cached response to another team is a data leak, not an optimisation.

Where it pays off: FAQ-style traffic, support bots, and agent workflows with repetitive tool calls, where many users ask the same things. Reported hit rates in those settings reach 40–65%, and a hit returns in milliseconds instead of seconds while burning no provider quota.

One more discipline: version the cache. A model upgrade or a prompt-template change must invalidate old entries, or you'll serve answers from a model you've retired. It's the same invalidation problem as any CDN-style caching layer, with the twist that "stale" here means semantically outdated, not byte-different.

The rule that summarises the whole section: a cache miss costs money, a false-positive hit silently gives a user the wrong answer. Bias conservative.

Streaming: The Part Everyone Forgets

Most user-facing LLM traffic streams tokens over server-sent events (SSE), the transport OpenAI, Anthropic, and Google all use. The gateway sits mid-stream, and that position creates problems a request-response proxy never sees.

Interviewer: A response is halfway streamed to the user and the provider connection drops. What do you do?

Candidate: Nothing clever, and that's deliberate. Failing over mid-answer means model B finishing model A's sentence — the user sees the seam. So failover is only allowed before the first token reaches the client. After that, end the stream cleanly with an error event and let the client retry. That's why I keep two timeouts: a short one on time-to-first-token, and a longer one for the whole response.

Interviewer: And what does a dropped stream do to your token accounting?

Candidate: The usage report arrives in the final stream event — which a truncated stream never delivers. So the gateway counts tokens as they pass through and reconciles from its own count. Idempotently, so a retried cleanup doesn't credit a team twice.

Why this answer works:

• "Nothing clever, and that's deliberate" shows restraint — the candidate knows when engineering effort makes the user experience worse.
• The first-token boundary is a crisp, defensible rule the interviewer can push on and the candidate can defend.
• Splitting TTFT from total timeout shows the candidate has actually watched LLM streams behave.
• The accounting follow-up closes the loop back to the rate limiting section, so the design holds together as one system.

Two infrastructure traps to name even if the interviewer doesn't ask:

Buffering kills streaming. Reverse proxies buffer responses by default. Nginx, the most common proxy in front of a gateway, holds several KB before flushing, so the user sees nothing for seconds and then a wall of text. Every hop on the streaming path must have buffering disabled. This is the same class of problem as real-time messaging system design: once a stream is open, every hop between model and user has to pass bytes through immediately.

Idle timeouts drop long streams. Load balancers kill connections that go quiet for 60–120 seconds, and a reasoning model can think silently for longer than that between tokens. SSE comment frames as keep-alive heartbeats hold the connection open through the quiet stretches.

Observability and Cost Attribution

Remember the original complaint in the prompt: "nobody can tell me which team spent what". This is where it gets answered. And interviewers in 2026 grade observability as a rubric item, not a bonus.

Every request emits one usage record: team, key, route, provider, model, input and output tokens, computed cost, cache hit or miss, time-to-first-token, total latency, and outcome. Records append to a durable ledger (a queue feeding a warehouse) asynchronously, never on the hot path. Reconciliation corrects the estimates, so the ledger ends up exact even though admission ran on guesses.

The metrics worth alerting on, per team and per provider-model pair: token throughput against quota, budget burn-down, 429 rates in both directions (issued to clients, received from providers), TTFT and tokens-per-second percentiles, cache hit rate with sampled false-positive rate, and circuit breaker state changes. Provider quota utilisation nearing its ceiling should page someone before users see errors. The full monitoring stack is its own interview question, covered in metrics, monitoring and alerting system design.

Cost dashboards do more than keep finance happy. Once teams see per-route spend, they fix things on their own: switching wordy routes to cheaper models, adding caching to repetitive ones. Visibility is itself a cost control.

Most of the companies, even mine right now are struggling to keep token usage in check. And with providers raising prices, it's not getting any better.

That's why companies are starting to introduce usage limits, also tiered usage to make engineers request more usage if needed. This makes engineers concious about their spends.

Common Interview Follow-ups

Q: Redis backs your admission control. What happens when Redis goes down?

Decide fail-open versus fail-closed out loud. Fail-closed turns a Redis blip into a company-wide AI outage, which is usually the wrong trade. A reasonable degraded mode: fail open with local approximate limits, where each gateway instance enforces 1/N of each team's quota from memory, accept loose enforcement for a few minutes, and reconcile the ledger after recovery. Budgets are a hard business rule, so give the budget check a stricter degraded policy than the rate limits.

Q: Why not let each team call the providers directly with their own API keys?

It works until it doesn't: no aggregate cost control, forty copies of retry and fallback logic drifting apart, provider keys scattered across codebases, no shared cache, and no way to protect the shared quota from one team's mistake. The gateway centralises exactly the things that are dangerous when scattered. The honest cost: you've built a single point of failure and must engineer its availability to match.

Q: How do you set the Retry-After on 429s you return to clients?

Compute it from bucket state: the deficit divided by the refill rate gives the earliest moment the request could pass. A precise hint teaches well-behaved clients to back off correctly; a generic "1 second" teaches them to hammer you. When the constraint is upstream, pass through the provider's own backoff hint.

Q: Could you rate-limit on cost directly instead of tokens?

Dollars are the more honest unit, since a flagship-model token and a mini-model token differ in price by an order of magnitude. But provider quotas are set in tokens, so you need token buckets anyway to protect the upstream edge. The clean design runs both: token buckets to fit inside provider quotas, a money bucket for budget truth. Same mechanism, two currencies.

Q: How would you handle a provider deprecating a model?

The logical-to-concrete model mapping absorbs it: point the logical name at the successor, canary a slice of traffic, compare quality and cost, then cut over. No client changes. This migration story is one of the strongest reasons the gateway exists, and it's worth volunteering even if nobody asks.

Q: Why build this instead of using an off-the-shelf gateway?

General-purpose API gateways handle auth, TLS, and request-count limiting well, and products like LiteLLM, Portkey, and Kong's AI plugins now cover token-aware limiting and fallback. Buying is a legitimate answer. The interview signal isn't build-versus-buy, it's knowing which parts are hard — token estimation, estimate-and-reconcile, semantic cache safety, fairness at the shared quota — whoever ends up building them.

Quick Interview Checklist

Before you wrap up in the interview, make sure you've covered:

• Clarified scale, streaming, budgets, and latency budget before designing
• Stated why request-count limiting fails — variable token cost per request
• Dual limits: RPM and TPM per team, checked atomically (Lua script in Redis)
• Estimate-admit-reconcile for output tokens, with max_tokens as the reservation
• Named over-reservation and orphaned reservations as failure modes, with mitigations
• Noisy neighbour at the shared provider quota level — priority tiers or weighted fair queuing
• Retry with backoff and jitter, honour Retry-After, circuit breaker per provider-model
• Proactive throttling from provider rate-limit headers, not just 429 reactions
• Exact-match vs semantic cache, threshold trade-off, tenant-scoped keys, versioned invalidation
• Streaming: no proxy buffering, failover only before the first flushed token, heartbeats
• Per-team cost ledger, async reconciliation, and the metrics you'd alert on
• Fail-open vs fail-closed stance for Redis loss, stated explicitly

Conclusion

The LLM gateway question looks like a proxy design and turns out to be a resource-governance problem: three rate limiters stacked on top of each other — yours, your provider's, and finance's — with the true cost of each request unknown until after you've committed to it. Estimate, admit, reconcile is the mechanism that resolves that tension, and it's the one idea to make sure your interviewer hears from you clearly.

Reading a design is one thing. Defending your semantic-cache threshold out loud while an interviewer pushes back is a different skill, and it's the one that decides offers. Mockingly.ai runs AI mock interviews on exactly these system design questions, with follow-up pressure calibrated to senior loops. Run this design against it before your interviewer runs it against you.

Frequently Asked Questions

What is an LLM gateway?

An LLM gateway is a proxy layer between an organisation's applications and one or more LLM providers (OpenAI, Anthropic, Google, self-hosted models). It exposes a single unified API and centralises rate limiting, budget enforcement, provider routing and failover, caching, and per-team cost attribution — so individual teams don't each reimplement them.

Why is request-based rate limiting not enough for LLM APIs?

LLM requests vary enormously in cost — a 50-token prompt and an 80,000-token prompt are the same "request" but differ by three orders of magnitude in tokens consumed. Rate limiting by request count alone lets a few large requests exhaust provider capacity and budget. LLM systems limit requests per minute (RPM) and tokens per minute (TPM) together.

What is the difference between RPM and TPM rate limits?

RPM (requests per minute) caps how many API calls you make regardless of size; TPM (tokens per minute) caps total tokens processed across all calls. Providers enforce both simultaneously — exceeding either one returns a 429 error — because each catches abuse patterns the other misses. OpenAI uses a combined TPM covering input and output; Anthropic splits token limits into separate input (ITPM) and output (OTPM) limits.

How does semantic caching work in an LLM gateway?

Semantic caching embeds each incoming prompt as a vector and searches previously answered prompts for a near match; if similarity exceeds a threshold, the stored response is returned without calling the model. It cuts cost and latency on repetitive workloads like FAQ and support traffic, but requires careful threshold tuning — too loose and users get answers to subtly different questions.

How do you rate limit streaming LLM responses?

You can't know a streamed response's token count at admission time, so the gateway reserves an estimate — input tokens plus the request's max_tokens — against the token bucket, then reconciles with the actual usage reported in the stream's final event. Unreconciled reservations carry a TTL so crashed requests don't permanently leak quota.

What is the difference between an LLM gateway and an API gateway?

An API gateway handles generic HTTP concerns: authentication, TLS, routing, request-count rate limiting. An LLM gateway adds model-specific layers — token estimation and token-denominated limits, estimate-and-reconcile accounting, semantic caching, model-aware routing and fallback, and per-token cost attribution. Production stacks often run both, with the LLM gateway behind the general one.

Which companies ask LLM gateway system design in interviews?

Any company running significant LLM traffic asks this or a close variant — AI-native companies (OpenAI, Anthropic-adjacent startups), big tech loops at Google, Meta, and Amazon where AI infrastructure questions now appear in general senior rounds, and infrastructure-heavy companies like Stripe, Cloudflare, and Datadog. It also appears as a follow-up escalation of the classic rate limiter question.

How long should I spend on each part in the interview?

In a 45-minute round: about 5 minutes clarifying scope, 5 on requirements and estimates, 10 on high-level architecture, and 20 on two or three deep dives — token-aware rate limiting should be one of them, since it's the heart of the question. Reserve the final 5 minutes for failure modes and operational concerns like observability and degraded-mode behaviour.