Design a RAG-based AI assistant that answers questions over a company's internal documents — OpenAI Interview

RAG System Design Interview Guide: Design an AI Assistant Over Internal Documents

Retrieval-augmented generation sounds like a weekend project. Embed your documents, store the vectors, find the nearest ones for each question, hand them to an LLM. There are a hundred tutorials that get you there in forty lines of Python. Then someone asks why the assistant confidently quoted a policy that was deleted three weeks ago, why an intern can see the CFO's board notes, and why the answer to "what is our parental leave policy" came back with a chunk about parking. And you realise the forty lines were never the system.

That's what makes RAG system design one of the most revealing questions in senior interview loops right now. It has moved out of ML-specialist rounds and into general backend loops at Google, Meta, Amazon, Microsoft, and OpenAI, usually phrased as "design an AI assistant over our internal docs" or "design a vector search service for a RAG application". It looks like a search problem wearing an LLM costume. It is actually four problems stacked on top of each other: an ingestion pipeline, a search engine, a permissions system, and a generation layer that has to be stopped from making things up.

Candidates who explain what to build pass. Candidates who explain why each design decision was made get offers.

That gap — between knowing the answer and explaining it clearly under pressure — is exactly what Mockingly.ai is designed to close. But first, let's build the design.

Step 1: Clarify the Scope

Interviewer: We have about twenty thousand employees and a few million internal documents spread across a wiki, Google Drive, a ticketing system, and some PDFs. Design an AI assistant that answers employee questions using those documents.

Candidate: A few questions before I draw anything. When you say a few million documents, is that the count of source documents or of pages? Do answers need citations back to the source, or is a plain answer fine? Does the assistant have to respect the existing permissions on those documents? How fresh does it need to be when a document changes? And is there a latency target for an answer, given the LLM call alone is going to take a couple of seconds?

Interviewer: Call it five million documents, average around 2,000 tokens each, and growing. Citations are required — legal insisted. Yes, permissions are non-negotiable; if you can't open the document, the assistant must not use it. Freshness within about fifteen minutes of an edit. Aim for a first token in under two seconds on the happy path.

Candidate: Good. Citations and permissions change the whole shape of this. Citations mean I need to retrieve passages, not documents, and carry their provenance all the way to the answer. Permissions mean the retrieval layer has to filter by access control before ranking, not after. Fifteen-minute freshness rules out nightly rebuilds, so ingestion has to be incremental and event-driven. And a two-second time-to-first-token with a multi-second LLM in the path means everything before generation — retrieval, reranking, prompt assembly — gets a budget of a few hundred milliseconds. Quick requirements and numbers, then I'll go deep on five things: chunking, the vector index, retrieval quality, permissions, and grounded generation.

What the candidate did here:

• Every question maps to a design decision. Citations force passage-level retrieval. Permissions force ACL-aware search. Freshness decides the ingestion architecture. The latency target sets the budget for everything upstream of the model.
• They said out loud what the constraints eliminate — nightly rebuilds, post-hoc permission filtering — before drawing a box.
• They named the hardest part immediately: the model is slow, so retrieval has to be fast.
• They committed to a plan for the next forty minutes.

Requirements

Functional

• Question answering — accept a natural-language question and return an answer grounded in internal documents
• Citations — every answer links to the specific passages it was drawn from
• Multi-source ingestion — wiki pages, Drive files, tickets, and PDFs, with heterogeneous formats
• Permission enforcement — only documents the asking user can already access may inform the answer
• Freshness — edits, additions, and deletions reflected within fifteen minutes
• Abstention — when the documents don't contain an answer, the assistant says so rather than guessing
• Conversation context — follow-up questions resolve against prior turns

Non-Functional

• Latency — first token in under two seconds; retrieval plus reranking under ~500 ms at p99
• Correctness over coverage — a wrong confident answer is worse than "I couldn't find that"
• Security — permission checks are enforced in the retrieval path, never left to the prompt
• Scalability — index grows with the corpus; query load grows with headcount
• Observability — retrieval quality and answer quality measured continuously, not just at launch
• Cost — per-query cost stays predictable as usage grows; generation tokens are the dominant line item

Back-of-the-Envelope Estimates

Interviewer: What are the numbers?

Candidate:

Corpus
  5M documents × ~2,000 tokens          = 10B tokens
  Chunk at ~400 tokens with overlap     → ~25M chunks
  (5M × 5 chunks, plus overlap; call it 25M)
 
Vector index
  Embedding dim 1,536, float32          = 1,536 × 4 bytes ≈ 6 KB per vector
  25M × 6 KB                            ≈ 150 GB of raw vectors
  Plus HNSW graph links                 → roughly 200 GB+ in memory
  With int8 quantisation (4x smaller)   ≈ 40 GB vectors + graph
 
Text + metadata store
  25M chunks × ~2 KB text               ≈ 50 GB (disk, not RAM)
 
Query load
  20,000 employees × ~10 questions/day  = 200K queries/day
  ÷ 86,400 s                            ≈ 2–3 QPS average
  Concentrated in work hours, 10x peak  ≈ 25 QPS peak
 
Ingestion
  ~1% of corpus changes per day         = 50K docs → ~250K chunks/day
  ÷ 86,400 s                            ≈ 3 chunks/second
  Initial backfill: 25M chunks to embed → hours, not minutes

Candidate: Two things jump out. Query throughput is tiny — 25 QPS at peak. I am not building this for scale in the usual sense. The real constraints are memory and quality. 200 GB of vectors and graph doesn't fit comfortably on one box, so I have to decide between quantisation, smaller embeddings, and sharding before I decide anything else. And at 25 QPS, I can afford expensive per-query work — reranking, multiple retrievers — that I could never afford on a consumer search engine.

What the candidate did here:

• The math is rounded and readable. Nobody wants to watch long division.
• Chunks, vectors, and queries are estimated separately because they stress different parts of the system.
• The punchline is the point: throughput is trivial, memory and quality are the problem. That tells the interviewer where the deep dives are going.
• "I can afford expensive per-query work" is a senior-level observation. Most candidates reflexively optimise for QPS they don't have.

Why "Just Put the Documents in the Prompt" Fails

Interviewer: Models have million-token context windows now. Why not skip retrieval, stuff the relevant documents in, and let the model figure it out?

Candidate: Four reasons, and any one of them kills it. Scale: the corpus is ten billion tokens, so "the relevant documents" is itself a retrieval problem — I still have to find them. Cost: every token in the prompt is paid for on every query, so a million-token prompt at 200K queries a day is a number finance will not sign. Quality: models degrade on very long contexts; a fact buried in the middle of 800K tokens is recalled less reliably than the same fact in a 2K-token prompt. And permissions: if I put a document in the prompt, the model has it, regardless of what I tell it about who's asking. Retrieval is the only place I can enforce access control.

Interviewer: So what is RAG, in one sentence?

Candidate: Retrieval-augmented generation is a search engine in front of a language model: for each question, find the small set of passages most likely to contain the answer, and give the model only those — so the answer is grounded in your data, cheap, current, and permission-scoped.

Why this answer works:

• The candidate gives reasons that compound instead of one hand-wavy "it doesn't scale".
• Permissions as an argument for retrieval is the observation most candidates miss. Retrieval isn't just about fitting in the context window; it's the enforcement point.
• The one-sentence definition is clean enough that the interviewer can move on. Knowing when to stop explaining is a signal too.

The same logic separates RAG from fine-tuning. Fine-tuning bakes knowledge into weights: it can't be updated in fifteen minutes, can't cite a source, and can't be scoped per user. Fine-tuning changes how a model behaves. Retrieval changes what it knows. For a question-answering assistant over live, permissioned documents, retrieval is the only option that satisfies the requirements.

The RAG Architecture, End to End

This is where most candidates draw one pipeline. The design is actually two systems with very different shapes, joined by a shared index.

INGESTION PATH (async, throughput-oriented)
 
  Wiki ─┐
  Drive ─┼─► Connectors ─► Change Queue ─► Parser ─► Chunker ─► Embedder ─► Index Writer
  Tickets┤    (webhooks +     (Kafka)                                            │
  PDFs ──┘     polling)                                                          ▼
                                                              ┌─────────────────────────────┐
                                                              │ Vector index (ANN, sharded)  │
                                                              │ Lexical index (BM25)         │
                                                              │ Chunk store (text + meta)    │
                                                              │ ACL store (doc → principals) │
                                                              └─────────────────────────────┘
                                                                              ▲
QUERY PATH (sync, latency-oriented)                                           │
                                                                              │
  User ─► API ─► Query Rewriter ─► Permission ─► Hybrid Retrieval ─► Reranker ─► Prompt ─► LLM ─► Stream
                 (history +        Resolver       (vector + BM25,      (cross-      Builder   (via     to user
                  rewrite)         (user →         ACL-filtered,        encoder)    (top-k +   gateway)  with
                                   principals)     fused)                            citations)          citations

The ingestion path is a data pipeline. It cares about throughput, idempotency, and never silently dropping a document. It's the same class of system as a distributed task scheduler: a queue of work, stateless workers, retries, and a durable record of what has been processed. Latency is measured in minutes.

The query path is a request-response service. It cares about p99 latency and getting the right five passages out of twenty-five million. Latency is measured in milliseconds, right up until the LLM call, which is measured in seconds.

The two paths share the index and nothing else. Keeping them separate means a burst of document updates can never slow down a user's question, and a spike in questions never delays indexing.

Walking the query path once, end to end:

1. The user asks "how many days of parental leave do we get in the Netherlands?" with two prior turns of conversation.
2. The query rewriter resolves the conversation into a standalone question, expands acronyms, and may generate one or two paraphrases. This is a small, fast model call or a rules layer — not the big model.
3. The permission resolver expands the user into their set of principals: user ID, groups, org units, regions. This is a cached lookup, not a live call to the identity provider.
4. Hybrid retrieval runs a vector search and a lexical search in parallel, each pre-filtered to chunks the user's principals can access, and fuses the two ranked lists.
5. The reranker scores the fused top ~50 candidates against the question with a cross-encoder and keeps the top 5–8.
6. The prompt builder assembles the question, the selected passages with their source IDs, and instructions to cite and to abstain when unsure.
7. The LLM call goes through an LLM gateway for rate limiting, model routing, and cost attribution, and streams tokens back.
8. The response is post-processed: citation markers are mapped to document links and verified to exist in the retrieved set.

Every one of those steps is a deep dive. The interviewer will pick two or three. Let me take them in the order that matters most.

Ingestion: Parsing and Chunking Decide Your Recall Before You've Searched Anything

Interviewer: Let's start at the top. A PDF arrives. What happens?

Candidate: The connector emits a change event onto the queue: source, document ID, version, and the operation — create, update, or delete. A parser worker fetches the content and converts it to a normalised structure: text with headings, paragraphs, tables, and code blocks preserved as elements, not flattened to a string. Then the chunker splits that structure into passages. Each chunk gets a deterministic ID derived from the document ID and its position, is embedded, and is written to the vector index, the lexical index, and the chunk store — with the document's ACL and version attached.

Interviewer: Why does the structure matter? Why not split every 500 tokens?

Candidate: Because a fixed-size splitter cuts tables in half, separates a heading from the paragraph that explains it, and slices a code block mid-function. Then retrieval finds the half-table, the model reads a row with no column headers, and the answer is confidently wrong. Chunk boundaries are where recall is won or lost — before a single query has been run. I split on structural boundaries first, fall back to size limits within a section, and keep a heading trail on every chunk so a passage says "Benefits › Parental Leave › Netherlands" even when the words "parental leave" don't appear in it.

Why this answer works:

• It names the failure mode concretely: half a table, a heading orphaned from its body. Interviewers remember examples, not adjectives.
• "Recall is won or lost before a query runs" reframes chunking from a preprocessing detail to a ranking decision. That's the insight most candidates lack.
• The heading trail is a small, practical trick that shows the candidate has actually debugged a RAG system.

A few chunking decisions worth stating explicitly, because each one is a follow-up waiting to happen:

• Size. Somewhere in the range of 200–500 tokens is the common sweet spot. Smaller chunks are more precise but lose context; larger chunks carry context but dilute the embedding, because one vector has to represent several ideas. The right number depends on the content and is found by evaluation, not by picking a default.
• Overlap. A 10–20% overlap between adjacent chunks stops a sentence that straddles a boundary from being lost to both. It costs storage and slightly more index size. Cheap insurance.
• Small-to-big retrieval. Index small chunks for precise matching, but when a chunk is selected, hand the model its parent section for context. You search with a sentence and read with a page. This separates the granularity of matching from the granularity of reading, which are different problems.
• Tables. Serialise each row with its column headers, or keep the whole table as one chunk with a generated summary. Never let a row float free of its header.
• Deterministic chunk IDs. hash(doc_id, version, chunk_index). This is what makes re-ingestion idempotent: if a worker retries, it writes the same IDs and nothing duplicates.

Embeddings and the Vector Index: The Memory Problem

Interviewer: You said 200 GB in memory. Walk me through the index.

Candidate: Each chunk becomes a dense vector — with a 1,536-dimension model that's about 6 KB in float32. Exact nearest-neighbour search over 25 million of those means comparing the query against all 25 million on every request, which is a scan of 150 GB. Too slow. So I use an approximate nearest-neighbour index. HNSW is the usual choice: a layered graph where search hops from a coarse layer down to fine neighbours in logarithmic time. The catch is that HNSW keeps the vectors and the graph in RAM, so memory is the constraint, not CPU.

Interviewer: And how do you get 200 GB down to something sane?

Candidate: Three levers, and I'd pull them in order. First, quantisation: storing vectors as int8 instead of float32 cuts memory roughly 4x with a small recall loss; more aggressive schemes like product quantisation go further at a higher quality cost. Second, dimension reduction: several current embedding models are trained so you can truncate to 512 or 768 dimensions and keep most of the quality, which is another 2–3x. Third, sharding: split the index by document source or tenant across nodes and fan out queries. With the first two levers, 200 GB becomes something like 20–30 GB, which fits on one well-provisioned node with a replica. I'd still shard, because I want the growth path and I want to isolate a rebuild to one shard at a time.

Why this answer works:

• The candidate explains why exact search fails before naming HNSW. Why before what, every time.
• "Memory is the constraint, not CPU" is the correct diagnosis for graph indexes, and it's what lets the candidate justify quantisation as the first lever.
• The three levers come with rough multipliers, so the interviewer can see the 200 GB actually shrink. Vague "we'd optimise it" answers don't survive follow-ups.
• Choosing to shard anyway for operational reasons — rebuilds, growth — shows production thinking beyond the immediate numbers.

Two more things to say about the index before the interviewer has to ask:

HNSW vs IVF. HNSW gives the best recall-latency trade-off for in-memory indexes but is memory-hungry and slow to build. IVF-style indexes cluster vectors and only scan a few clusters per query; they're cheaper on memory and friendlier to disk, at some recall cost. For 25 million vectors and a strict latency target, HNSW with quantisation is the default; if the corpus grows by 10x, a disk-backed IVF or a hybrid becomes the conversation. Name the trade-off; don't just name the acronym.

Where the text lives. The vector index stores vectors and IDs. The chunk text, heading trail, source URL, version, and ACL live in a separate chunk store — a document database or even Postgres, keyed by chunk ID. Retrieval returns IDs; a batched lookup hydrates them. Keeping text out of the vector index keeps the index small and lets the two scale independently.

Retrieval: Why Pure Vector Search Fails the Interview

This is the section that separates people who have shipped RAG from people who have read about it.

Interviewer: You have a vector index. Query comes in, embed it, take the top ten nearest chunks. Done?

Candidate: That's the demo version, and it fails on the questions employees actually ask. Someone searches "ticket INC-48213" or "the Q3 SOC 2 audit" or "error code E_QUOTA_EXCEEDED". Dense embeddings are good at meaning and bad at exact tokens — an ID, a product name, a code — because those strings are rare and the model has no semantic handle on them. Lexical search, BM25, is the opposite: exact tokens are its whole job, and it has no idea that "time off for a new baby" means parental leave. So I run both, in parallel, and fuse the results.

Interviewer: Fuse how? The scores aren't comparable.

Candidate: Right — a cosine similarity of 0.82 and a BM25 score of 14.3 mean nothing to each other. So I fuse on rank, not score. Reciprocal rank fusion: each result gets 1/(k + rank) from each list, with k around 60, and I sum. A chunk ranked highly by both retrievers floats to the top; one ranked highly by only one still survives. It's crude and it works well enough that it's the standard baseline. Then I hand the fused top 50 to a reranker.

Interviewer: Why do you need a reranker if you've already ranked twice?

Candidate: Because both retrievers scored the question and the chunk separately — the chunk was embedded months ago, with no idea what question would be asked. A cross-encoder reads the question and the chunk together and scores how well this specific passage answers this specific question. It's far more accurate and far too slow to run against 25 million chunks, so it only runs against the 50 candidates the cheap retrievers surfaced. Cheap and wide, then expensive and narrow. At 25 QPS I can afford it easily.

Why this answer works:

• The failure case is specific and believable: ticket IDs, error codes, product names. Every internal corpus is full of them.
• The candidate identifies why scores can't be mixed and chooses rank fusion for that reason. This is the kind of detail that shows the design was reasoned, not copied.
• "Cheap and wide, then expensive and narrow" is a frame the interviewer can carry to every later question about latency.
• The candidate ties the reranker back to the estimates: low QPS is what makes an expensive per-query step affordable. The numbers earned their place.

The retrieval funnel, with rough latency at each stage on a warm system:

Query embedding             ~10–30 ms   (small model, or cached)
Vector search  (top 100)    ~10–50 ms   (HNSW, ACL pre-filtered)
BM25 search    (top 100)    ~10–50 ms   (in parallel with vector)
Rank fusion                 < 1 ms
Cross-encoder rerank (50)   ~50–150 ms  (GPU-backed, batched)
Chunk hydration             ~5–20 ms    (batched key lookup)
                            ─────────
                            ~100–300 ms  before the LLM sees anything

That fits the 500 ms budget with headroom, and leaves the rest of the two seconds for the model's time-to-first-token.

Two refinements to raise if there's time:

Query rewriting. "What about the UK?" as a follow-up is unanswerable in isolation. Before retrieval, a small model rewrites it against the conversation into "how many days of parental leave do employees get in the UK?" The same step can generate a hypothetical answer and embed that — the idea being that an answer-shaped passage is closer in embedding space to other answers than a question is. Both are cheap and both measurably lift recall on conversational traffic.

Metadata filters as a first-class retriever. "Docs updated this quarter", "only from the engineering wiki", "only tickets". Structured filters applied inside the index narrow the candidate set before ranking. They're also how permissions work, which is the next section.

Permissions: The Part That Gets Systems Pulled From Production

Interviewer: Legal's requirement. An engineer asks a question, and somewhere in the corpus there's a board deck they can't open. How do you make sure it never influences the answer?

Candidate: The filter has to be applied before ranking, inside the retrieval step, using the same ACLs the source systems use. Every chunk carries its document's allowed principals — users, groups, org units. Every query carries the asker's principals. Vector and lexical search both run with a filter: only chunks whose allowed set intersects the user's set. The board deck is never a candidate, so it can never be reranked, never enter the prompt, never leak into a sentence.

Interviewer: Why not retrieve the top 50 and then drop the ones the user can't see? Simpler.

Candidate: Two reasons, and the second is the one that gets you paged. Recall: if the user can see 5% of the corpus and I post-filter the top 50, most of the 50 get dropped and I'm generating an answer from three chunks, or zero. I'd have to over-fetch by 20x to compensate, and the ratio changes per user. Leakage: post-filtering means the restricted chunk was retrieved, scored, and present in the process. A logging bug, a cache keyed on the query alone, or a "similar questions" feature that reuses retrieved sets, and it's exposed. Pre-filtering means the system structurally cannot see what the user cannot see.

Why this answer works:

• The candidate separates a quality argument (recall collapses) from a security argument (structural leakage), and ranks them. Interviewers want to see that ordering.
• "Structurally cannot see" is the right standard for a security control. Prompt instructions like "don't reveal restricted documents" are not a control; the model is not the enforcement point.
• The over-fetch ratio observation shows the candidate has thought about what post-filtering actually costs in practice, not just in principle.

The mechanics worth spelling out:

• ACL storage. Each chunk stores a compact list of allowed principal IDs (or a document ID that joins to a per-document ACL). Most vector databases support filtered search natively; the filter is evaluated during graph traversal, not as a post-pass. Filtered HNSW search has its own performance quirks — a very restrictive filter can force the graph to traverse far more nodes — so for users with narrow access, a brute-force scan over their small allowed set can be faster. Say that; it's a real trade-off.
• Group expansion. Users belong to groups, groups nest. Resolve the user to a flat set of principals once per query, from a cache with a short TTL, so retrieval does a set intersection instead of a graph walk.
• Permission changes. When a document's sharing changes, the ACL store updates for every chunk of that document. That's a metadata write, not a re-embedding, so it's fast — but it must go through the same change queue so it's never lost. Removing someone's access has to propagate within the same fifteen-minute freshness window as content edits. Ideally faster.
• Caching. Any cache in the query path — semantic cache, retrieved-set cache — must be keyed on the user's permission set as well as the query. Two users asking the same question can legitimately get different answers. The gateway's semantic cache, covered in the LLM gateway design, has the same tenant-scoping rule for the same reason.

Grounded Generation: Citations, Confidence Gating, and Saying "I Don't Know"

Interviewer: You've got eight good passages. Now the model writes the answer. How do you stop it from adding things that aren't in the passages?

Candidate: I can't fully stop it, so I design for the fact that it will sometimes happen. Three layers. The prompt: each passage is labelled with an ID, and the instructions say to answer only from the passages, cite the ID after each claim, and say explicitly if the passages don't contain the answer. The verification: after generation, I parse the citation markers, check each one maps to a passage that was actually in the prompt, and strip or flag any that don't. And the gate: if the reranker's top score is below a threshold — meaning nothing retrieved was a confident match — I don't generate at all. I return "I couldn't find this in our documents" with the nearest passages as suggestions. An abstention is a correct answer. A fabrication is an incident.

Interviewer: What does the user actually see?

Candidate: A streamed answer with inline citation markers that resolve to the source document, the section, and ideally the highlighted passage. The stream matters: the model takes seconds, and a user watching tokens arrive waits far more patiently than one staring at a spinner. But citations are only trustworthy once the response is complete, so markers render as placeholders while streaming and resolve at the end.

Why this answer works:

• "I can't fully stop it, so I design for it" is the honest senior answer. Claiming a prompt eliminates hallucination is a red flag.
• Three layers — prompt, verification, gate — each with a clear mechanism. The interviewer can push on any one.
• Confidence gating on the reranker score is a concrete, measurable trigger, not "if the model isn't sure".
• The streaming detail shows the candidate has watched real users wait for real LLM calls. Streaming plumbing has its own traps — proxy buffering, idle timeouts — the same ones that come up in chat system design.

A few generation details that tend to come up:

• Passage order. Models attend more reliably to the start and end of a prompt than the middle. Put the highest-ranked passages first or last, not buried in position four of eight.
• Context budget. Eight passages at 400 tokens is about 3,200 tokens plus the question, instructions, and history. Cap the total, and when the budget is tight, drop the lowest-ranked passage rather than truncating all of them.
• Two models, not one. Query rewriting, and possibly the abstention decision, run on a small fast model. Answer generation runs on the capable one. Routing between them is exactly what the gateway is for.
• Conflicting sources. When two passages disagree — the 2024 policy and the 2026 policy — the model should be told to prefer the newer document and say that it did. Version and date are in the metadata; put them in the prompt.

Freshness: Incremental Indexing, Deletes, and Changing the Embedding Model

Interviewer: Someone edits a wiki page. Fifteen minutes later, the assistant should reflect it. What happens in between?

Candidate: The wiki fires a webhook, or the connector's poll notices a new version. A change event lands on the queue. A worker fetches the new content, re-chunks it, and computes the chunk IDs. Then it diffs against the previous version's chunk set: unchanged chunks — same content hash — are skipped, changed chunks are re-embedded and upserted, and chunks that no longer exist are deleted from all three stores. Finally the document's version pointer advances. That last write is what makes the update atomic from the reader's side: until it flips, queries see the old version consistently.

Interviewer: And a delete?

Candidate: A delete is the operation people forget to test, and it's the one legal cares about most. The event carries the document ID; the worker looks up every chunk ID for it and removes them from the vector index, the lexical index, and the chunk store. I'd also run a periodic reconciliation job that compares the source system's document list against the index and removes anything orphaned, because webhooks get dropped and a document that was deleted at the source but still lives in the index is a compliance problem waiting to be discovered.

Why this answer works:

• The candidate describes a diff, not a full re-index. Re-embedding an entire document because one paragraph changed is the naïve version, and it's expensive at scale.
• The version pointer as the atomic commit is a clean way to avoid readers seeing half-updated documents.
• Deletes get their own answer, including the reconciliation job. Volunteering "webhooks get dropped" shows the candidate expects the unhappy path.

The follow-up that separates senior from staff:

Interviewer: You want to switch to a better embedding model. What happens to the index?

Candidate: Everything has to be re-embedded, because vectors from two different models live in different spaces — a query embedded with the new model is meaningless against chunks embedded with the old one. There's no shortcut. So it's a migration: build a second index with the new model in the background, backfilling all 25 million chunks over hours, while the old index keeps serving. Dual-write new changes to both during the backfill. When the new index is complete and evaluation shows it's better, flip the query path to it, keep the old one for a rollback window, then drop it. The index needs to be versioned by embedding model from day one, or this migration is a nightmare.

That answer is worth practising until it's fluent. The "embedding model version" requirement is invisible until the day you need it, and it's the kind of forward-looking constraint interviewers use to gauge whether a candidate has operated a system past its launch.

Evaluation: Retrieval Metrics and Answer Metrics Are Different Problems

Interviewers in 2026 grade this as a rubric item. Not a bonus.

Interviewer: How do you know the system is good? And when you change the chunk size, how do you know you didn't make it worse?

Candidate: I evaluate the two halves separately, because they fail separately. For retrieval, I maintain a golden set of a few hundred questions, each labelled with the chunks that actually contain the answer. Every change to chunking, embeddings, fusion, or reranking is scored against it: recall@k — did the right chunk make it into the top k — and a rank-aware metric like MRR or nDCG, since position in the prompt matters. For generation, I hold retrieval fixed and score the answers: faithfulness — is every claim supported by a retrieved passage — and correctness against a reference answer. Faithfulness can be scored by a strong model acting as judge, with a human-reviewed sample to keep the judge honest.

Interviewer: Why separate them?

Candidate: Because if the answer is wrong, I need to know which half to fix. If recall@10 is high but faithfulness is low, retrieval is fine and the prompt or model is the problem. If recall@10 is low, no amount of prompt engineering will help — the right passage was never in the prompt. Blending them into one "answer quality" score hides the diagnosis.

Why this answer works:

• Two metric families, two failure modes, one diagnostic rule. Simple and defensible.
• "Hold retrieval fixed" is how you get a controlled experiment. It signals the candidate has actually run evaluations, not just listed metrics.
• The LLM-as-judge with a human sample is the pragmatic 2026 answer. Pure human evaluation doesn't scale; pure model evaluation drifts.

What this looks like in production, beyond the offline golden set:

• Online signals. Thumbs up/down, whether the user clicked a citation, whether they rephrased the question immediately (a strong signal the first answer missed), and abstention rate. Abstention rate rising is either a retrieval regression or a corpus gap; both are worth knowing.
• Retrieval health. Distribution of top reranker scores over time. If the median drops, something upstream changed — a connector broke, a source went stale, a chunking deploy went wrong.
• Index freshness. Lag between source-of-truth version and indexed version, per source. This is the fifteen-minute SLO, measured directly.
• Alerting. Index lag over the SLO, abstention rate spike, faithfulness sample dropping, ingestion queue depth growing. The alerting stack itself is the subject of metrics, monitoring and alerting system design; the point here is that a RAG system without these numbers is a system you can't defend in a postmortem.

This is also a natural place to connect to the broader machine learning system design interview framing: offline metrics, online metrics, and a feedback loop are the same discipline whether the model is a ranker or a language model.

Cost and Latency: Where the Money Goes

Interviewer: What does a query cost, and what would you cut first?

Candidate: Roughly in order of expense: generation tokens dominate — a few thousand prompt tokens plus a few hundred output tokens on a capable model is the bulk of the bill. Reranking is next, but it's a small model on a GPU I'm already running, so it's more a fixed cost than a per-query one. Query embedding and the two searches are negligible. Ingestion is a separate line: the initial backfill of 25 million chunks is a large one-off, and steady-state re-embedding of 250K chunks a day is small.

Interviewer: So what do you cut?

Candidate: Prompt tokens, in two ways. Better reranking so I can send five passages instead of eight without losing recall — every passage I don't send is tokens I don't pay for, on every query. And caching at the gateway: exact-match for repeated questions, semantic for near-duplicates, scoped by permission set. A question like "how do I set up VPN" is asked hundreds of times a week in the same words. I'd also route by difficulty: a small model handles queries where the reranker score is very high and the passage is short, the big model handles the rest.

Why this answer works:

• The candidate ranks costs before proposing cuts, and the ranking is right: tokens on the capable model are the dominant cost in almost every RAG deployment.
• Better retrieval is a cost optimisation. Connecting quality work to the bill is the kind of thing that gets noticed.
• Caching is scoped by permissions, which shows the security section wasn't forgotten the moment money came up.

The latency budget, summarised, for the interviewer who asks where the two seconds go:

Permission resolution (cached)          ~5 ms
Query rewrite (small model)             ~100–300 ms   ← often the surprise
Hybrid retrieval + fusion               ~50–100 ms
Rerank                                  ~50–150 ms
Hydration + prompt assembly             ~20 ms
LLM time-to-first-token                 ~500–1,500 ms
                                        ──────────
                                        ~0.8–2.0 s to first token

The query rewrite is the step that quietly eats the budget. It's a model call, and model calls are slow. Run it on the smallest model that works, skip it when there's no conversation history, and consider running the rewrite and a retrieval on the raw query in parallel so the common case doesn't pay for it.

Common Interview Follow-ups

Q: Why not just use the vector database's built-in full-text search and skip the separate BM25 index?

If the vector store offers real lexical scoring with proper tokenisation, stemming, and BM25-style term weighting, that's fine and it simplifies operations. Many "full-text" features in vector databases are basic keyword filters, not ranked lexical search. Ask what it actually does. The requirement is two independent retrievers with different strengths, fused on rank; where they live is an implementation detail.

Q: How do you handle a question that spans multiple documents — "compare our UK and Netherlands leave policies"?

Decompose it. The query rewriter splits it into two sub-questions, each retrieves independently, and the prompt builder assembles both sets with clear labels. Without decomposition, a single embedding of the combined question lands somewhere between the two topics and retrieves neither well. This is the simplest form of the agentic, multi-step retrieval that shows up in harder variants of this question.

Q: The vector index node goes down. What happens?

Each shard has a replica; the query router fails over. If a whole shard is unavailable, decide fail-open or fail-closed out loud: serving answers from a partial corpus risks confidently wrong answers with missing context, so the safer default is to return "search is degraded" for queries that would have hit the missing shard, or to fall back to lexical-only retrieval with a visible warning. Index rebuild from the chunk store is the recovery path, which is another reason the chunk store is the source of truth and the vector index is derived.

Q: How do you stop someone from extracting the whole corpus through the assistant?

Rate limiting per user at the API, using the same rate limiter design as any other service, plus per-user quotas on tokens through the gateway. Retrieval only ever exposes documents the user can already open, so the assistant can't exceed their existing access — but it can make bulk reading faster, which is why the limits exist. Log every query with the user and the cited documents for audit.

Q: Would you fine-tune the embedding model on our corpus?

Not first. Off-the-shelf embeddings with good chunking, hybrid retrieval, and a reranker cover most of the gap. Fine-tuning the embedder or the reranker on in-domain question-passage pairs — which the golden set and click logs provide — is a real lever once the basics are measured and stable, and it comes with the migration cost from the freshness section. Say it's a later optimisation with a known cost.

Q: How would this change for a customer-facing support assistant instead of an internal one?

The corpus is smaller and public, so permissions simplify to product tiers or regions. Query volume rises by orders of magnitude, so caching and cheap-model routing move from nice-to-have to essential. The cost of a wrong answer rises too, so the abstention threshold gets stricter and there's a hand-off path to a human. Same architecture, different tuning — and saying that shows the design is principled rather than memorised.

Quick Interview Checklist

Before you wrap up in the interview, make sure you've covered:

• Clarified corpus size, citation requirement, permissions, freshness, and latency target before designing
• Explained why long context and fine-tuning don't satisfy the requirements — cost, freshness, citations, permissions
• Drawn two paths: async ingestion and sync query, sharing only the index
• Structure-aware chunking with heading trails, overlap, and deterministic chunk IDs
• Sized the vector index and named the levers: quantisation, dimension reduction, sharding
• Hybrid retrieval — dense plus BM25 — fused on rank, then a cross-encoder reranker on a narrow candidate set
• ACL pre-filtering inside retrieval, with the recall and leakage arguments against post-filtering
• Grounded prompt, citation verification, and a confidence gate that abstains
• Incremental diff-based re-indexing, deletes with reconciliation, and an embedding-model migration plan
• Retrieval metrics (recall@k, nDCG) separate from answer metrics (faithfulness, correctness)
• Per-query cost ranking, with prompt tokens as the dominant cost and better retrieval as the cut
• Latency budget with the query rewrite called out, and streaming for the answer

Conclusion

The RAG question looks like a pipeline and turns out to be a search engine with a permissions model and a very expensive, slightly unreliable final step. The demo version — embed, search, generate — is where every candidate starts. The design that gets offers is the one that explains why chunk boundaries decide recall, why two retrievers beat one, why permissions belong inside retrieval and not in the prompt, and why the two halves of the system are evaluated separately. Cheap and wide, then expensive and narrow, and never let the model see what the user can't.

Reading a design is one thing. Defending your abstention threshold while an interviewer asks why you didn't just post-filter is a different skill, and it's the one that decides offers. Mockingly.ai runs AI mock interviews on exactly these system design questions, with follow-up pressure calibrated to senior loops. Run this design against it before your interviewer runs it against you.

Frequently Asked Questions

What is a RAG system in a system design interview?

A RAG (retrieval-augmented generation) system is a search engine placed in front of a language model. For each question it retrieves the small set of passages most likely to contain the answer from a document corpus, and passes only those passages to the model, so the answer is grounded in the organisation's data, current, citable, and scoped to what the user is permitted to see. In interviews it's usually framed as "design an AI assistant over internal documents".

How does retrieval-augmented generation work?

Documents are parsed, split into chunks, converted to embedding vectors, and stored in a vector index alongside a lexical index. At query time the question is embedded, the nearest chunks are retrieved (often combined with keyword search), a reranker selects the best few, and those passages are placed in the LLM prompt with instructions to answer from them and cite sources. The retrieved passages are what "augment" the generation.

Why use hybrid search instead of pure vector search in RAG?

Dense vector search captures meaning but handles exact tokens poorly — ticket IDs, error codes, product names, and acronyms are rare strings with weak semantic signal. Lexical search (BM25) is the reverse: precise on exact terms, blind to paraphrase. Running both and fusing the ranked lists with reciprocal rank fusion recovers the failures of each, and a cross-encoder reranker on the fused candidates then produces the final ordering.

How do you reduce hallucinations in a RAG system?

Three layers: a prompt that labels each passage and instructs the model to answer only from them and cite by label; post-generation verification that every citation maps to a passage that was actually in the prompt; and a confidence gate that abstains — returning "not found in our documents" — when the reranker's top score is below a threshold. Hallucination can't be eliminated by prompting alone, so the design assumes it will occur and catches it.

How do you evaluate a RAG system?

Evaluate retrieval and generation separately. Retrieval is scored against a golden set of questions with labelled relevant chunks using recall@k and rank-aware metrics like MRR or nDCG. Generation is scored with retrieval held fixed, on faithfulness (every claim supported by a retrieved passage) and correctness against reference answers, often using a strong model as judge with a human-reviewed sample. Separating them tells you which half to fix when answers are wrong.

How do you handle document permissions in RAG?

Attach each document's access control list to its chunks at ingestion, resolve the asking user to their set of principals at query time, and apply the permission filter inside the retrieval step so restricted chunks are never candidates. Post-filtering retrieved results is both a recall problem (most candidates get dropped for users with narrow access) and a leakage risk (restricted content has already entered the process). Caches must be keyed on the permission set as well as the query.

Which companies ask RAG system design in interviews?

RAG and "design an AI assistant" questions appear in senior and staff loops at OpenAI, Google, Meta, Amazon, Microsoft, and Anthropic, and increasingly at any company shipping an AI feature over proprietary data — Notion, Atlassian, Salesforce, and fintech and healthcare companies with strict permission requirements. Since 2025 the question has shown up in general backend system design rounds, not only ML-engineering interviews.

How long should I spend on RAG in a system design interview?

In a 45-minute round: about five minutes clarifying scope and estimating, five on the two-path architecture, and the remaining thirty on two or three deep dives the interviewer steers toward — most often retrieval quality, permissions, and freshness or evaluation. Don't try to cover everything; go deep on the parts you can defend under follow-up questions.