Try it
THE QUESTION“How many casual leaves do I get, and do unused ones carry over?”
Ten chunks. Top-k = 3. No filters, no reranker, no hybrid — the pipeline most demos ship with.
You already know what RAG is — search your documents, paste the relevant part into the prompt. This week is the engineering. One HR handbook, one question, six places it breaks. Everything below recomputes live.
Ten chunks from a company handbook. One question an employee actually asks. Run the naive RAG pipeline — top-k vector search, stuff the results in the prompt — and watch it answer confidently from the wrong policy. This is a retrieval failure, not a model failure.
A chunk is the unit of what can be retrieved. Split this handbook page yourself and watch two facts that belong together get separated — or a chunk grow so broad it stops being about anything. Parent-document retrieval and structure-aware splitting are the ways out.
Every chunk becomes an embedding — a point in space. Near means “worded alike”, which is not the same as “correct”. Click a chunk to see its nearest neighbours by cosine similarity, then move top-k and watch which ones make the cut.
“Casual leave” is an exact phrase the employee typed — exactly what BM25 is good at and dense retrieval smears over. Fuse the two lists, then let a reranker read each chunk against the question. Watch the correct chunk go 7 → 3 → 1.
Retrieval worked. The right chunk is in the prompt — fifth of nine. Models attend best to the start and the end of a long context and genuinely miss what sits between: lost in the middle, Liu et al., 2023. Reorder the context and watch what the model grounds its answer in.
Five real handbook questions with known correct sources — a golden set. Toggle everything you just built and watch two numbers move independently: recall@k says the right chunk arrived, faithfulness says the answer actually used it. One of these states scores perfect recall and still answers badly.