You changed the chunk size from 500 to 1000 characters. Did that help? "The answers feel a bit better" isn't an answer you can trust, and it definitely isn't one you can repeat every time you touch chunking, the embedding model, or top-k. RAG evaluation exists to replace that feeling with a number.
Retrieval and generation fail differently - evaluate them separately
A RAG system has two places to go wrong, and conflating them makes debugging much harder than it needs to be. Retrieval can fail by not finding the right chunks at all - the answer exists in your documents, but the search never surfaced it. Generation can fail even when retrieval succeeds - the right chunks got retrieved, and the model still ignored them, misread them, or added something they didn't say. Evaluating the whole pipeline as one black box tells you something is wrong; evaluating the two stages separately tells you what to actually go fix.
Retrieval metrics: did we find the right chunks?
Recall@k asks: of the chunks that were actually relevant to this question, what fraction showed up in your top-k results? Precision@k asks the inverse: of the chunks you retrieved, what fraction were actually relevant? A system that returns 10 chunks to catch the 2 relevant ones has high recall but poor precision, and precision matters because irrelevant chunks aren't free - they cost tokens, dilute the model's attention, and increase the odds it latches onto the wrong passage.
Generation metrics: did the model use what it was given, honestly?
Faithfulness (also called groundedness) measures whether every claim in the answer is actually supported by the retrieved context - it's the metric that catches hallucination even when retrieval worked perfectly. Answer relevance measures whether the response actually addresses the question asked, independent of whether it's faithful - a perfectly grounded answer to the wrong question still isn't useful. Both are typically scored with an LLM-as-judge: a second model call, given the question, the retrieved context, and the generated answer, scores the answer against a rubric.
# Rough shape of a faithfulness check via LLM-as-judge
judge_prompt = f"""
Question: {question}
Context: {retrieved_chunks}
Answer: {generated_answer}
Does every claim in the Answer appear in or follow directly
from the Context? Respond with a score from 1-5 and why.
"""
score = judge_model.generate(judge_prompt)
Where the test questions come from
You don't need a large, perfectly curated evaluation set to start being useful. A handful of real questions pulled from actual usage - especially the ones where something visibly went wrong - is worth more than a large synthetic set that doesn't reflect what people actually ask. As your system runs, a natural, ongoing source of new evaluation examples is simply promoting real queries, especially the failures, into your test set.
| Question | What it measures | Catches |
|---|---|---|
| Did we retrieve the right chunks? | Recall@k, Precision@k | Bad chunking, wrong top-k, a retrieval bug |
| Does the answer address the question? | Answer relevance | The model going off-topic or answering a different question |
| Is every claim actually supported? | Faithfulness | Hallucination, even with correct retrieval |
That's the series
Across these five guides you've gone from what RAG actually is and why it beats fine-tuning for knowledge, through the six-stage pipeline every system is built from, the chunking decisions that determine what's even retrievable, the vector database that stores the result, and now how to tell, with numbers instead of vibes, whether any of it is actually working. That's the full loop - and it's the same loop regardless of which framework or platform you build it in.