RAG testing that shows whether the answer really came from the source
A citation can be real while the answer says something the document does not. We test retrieval and generation separately, so you know which one to fix.
The whole story, short
The problem
Answers cite a real document that does not actually support the claim.
What we did
- Build a labeled question set
- Separate retrieval from generation
- Score faithfulness and correctness
- Verify citations
- Evaluate chunking and embeddings
The outcome
Retrieval + generation scored separately
What goes wrong
- Answers cite a real document that does not actually support the claim.
- The wrong chunks are retrieved, so even a good model answers from bad context.
- Chunks split a rule from its exception, so answers miss important conditions.
- Nobody can tell whether a failure started in search or in generation.
Why the usual fix falls short
Testing only the final answer hides where the failure began. Without a labeled question set, every fix is a guess between retrieval and the model.
How we tested it
Build a labeled question set
Write questions with reference answers and supporting documents alongside subject experts, including questions the documents cannot answer.
Separate retrieval from generation
Measure context precision and recall on their own, so a failure can be traced to the step where it began.
Score faithfulness and correctness
Check each claim against the retrieved context and compare answers with the reference set, with experts reviewing a sample.
Verify citations
Confirm each citation exists, supports the claim and points to the current version of the document.
Evaluate chunking and embeddings
Compare chunking configurations to find which ones keep related content together.
Questions buyers ask
What is RAG testing?
RAG testing checks both halves of a retrieval-augmented generation system: whether it finds the right documents, and whether its answer stays faithful to them. Measuring the two separately shows where a failure begins, so teams can fix the retrieval or the generation step instead of guessing between them.
How do you measure hallucination in a RAG system?
Compare each claim in an answer with the retrieved source text and count the claims the source does not support. Run this across a representative question set, then track the rate over time. Human review of a sample keeps the automated scoring honest and catches subtle errors.
Which RAG evaluation metrics should we track?
Track context precision and recall for the search step, then faithfulness and answer correctness for the generated response. Citation accuracy matters when users rely on sources. Together, these metrics show whether the system found the right content, used it properly, and gave the user a correct answer.