Interview Questions

RAG Interview Question — PolicyBazaar (AI Engineer)

A policy document got updated this morning, but your assistant keeps serving the old answer.

The question: How will you fix the issue in the current RAG system ?

Here's how I'd answer:

First, name the real problem. There are two independent staleness bugs here. Both the vector index and the response cache is stale. They fail separately, so they need separate invalidation. Fixing retrieval but leaving a memoized answer in the cache still serves the wrong answer to the customer.

1. Version every document. Each doc carries a content_hash and updated_at. On ingestion, compare hashes and only re-embed the chunks that actually changed, not the whole corpus. This is delta re-indexing, and …

You're in an AI Engineer interview at Anthropic and the interviewer asks:

"We're training a new model for the legal and medical domain. What's the production risk of just using a standard, pre-trained Llama 3 tokenizer, and what's your fix?"

Most candidates say: "It's not optimal, but the tokenizer will just use subwords for the OOV (out-of-vocabulary) terms like 'aneurysm'. The model will eventually learn the combinations during fine-tuning."

Wrong . This answer ignores the real production bottleneck: compute cost.

The reality is that tokenization is a 𝐥𝐞𝐚𝐤𝐲 𝐚𝐛𝐬𝐭𝐫𝐚𝐜𝐭𝐢𝐨𝐧.

Using a general tokenizer on a specialized corpus is like trying to write a legal brief using only the 1000 most common English words. It's possible, but painfully inefficient.
Here's the …

LLM interview question (level - Architect/Tech Lead/Senior Engineer):

You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on long traces.

So you add KV cache compression and evict 90% of the cached tokens.

VRAM usage stays the same, and the GPU still runs out of memory.

Why?

Detailed answer:

Evicting 90% of the KV cache can free almost none of the memory it was using.

This sounds counterintuitive, but it follows directly from how production servers store the cache today.

The KV cache grows with every token a model generates. Each token appends its key and value vectors across every layer, and nothing is freed while generation continues. This is the dominant memory cost …