← All field notes

Research Watch · Retrieval · Evaluation

RAG evaluation: measuring whether an answer is actually grounded

RAGAS introduces reference-free metrics that separate retrieval quality from generation quality, so a team can tell whether a RAG failure came from search or from the model — without hand-labeled ground truth.

AI-assisted / research-based

This field note was drafted with AI assistance and synthesizes publicly available research papers and disclosed industry practice on an emerging AI technique. It is not based on confidential deployment data, is not investment, legal, medical, or security advice, and every primary claim links directly to its source so you can verify it yourself.

Retrieval-augmented generation is easy to demo and hard to evaluate rigorously. A response can read fluently while contradicting the very sources it cites, or cite sources that never supported the claim at all — and standard accuracy metrics do not catch either failure without exhaustive, expensive human review of every answer.

A RAG system can fail in the retriever, the generator, or both. Most evaluation setups can only tell you that something failed.

Four separable questions, no ground truth required

RAGAS: Automated Evaluation of Retrieval Augmented Generation ↗, by Es, James, Espinosa-Anke, and Schockaert, proposes a suite of metrics that score a RAG pipeline without requiring hand-labeled reference answers for every test question — a reference-free framework, in the paper's own terms.

Evaluation / RAG scoring framework

Four questions, no labeled answer key

FIG 01
RAGAS reference-free evaluation dimensions A RAG pipeline is scored on four separate dimensions without human-labeled ground truth: whether retrieved passages are relevant, whether all necessary passages were retrieved, whether the answer is faithful to those passages, and whether the answer actually addresses the question. RETRIEVAL SCOREDGENERATION SCORED CONTEXT RELEVANCEAre retrieved passagesactually relevant to the question? CONTEXT RECALLWas everything neededto answer actually retrieved? FAITHFULNESSAre claims supportedby the retrieved context? RELEVANCEDoes the answeraddress the question? NO GROUND TRUTH REQUIREDEvery metric is scored without hand-labeled reference answers— the point is faster evaluation cycles as RAG systems change
Splitting retrieval quality from generation quality tells you which half of the pipeline actually broke — a single end-to-end accuracy score cannot.

The framework separates retrieval quality from generation quality along four dimensions: context relevance (are the retrieved passages actually relevant to the question), context recall (was everything needed to answer the question actually retrieved), faithfulness (are the claims in the generated answer supported by the retrieved context), and answer relevance (does the response actually address the question that was asked, independent of whether it is grounded). The paper frames the motivation directly: accelerating evaluation cycles as RAG architectures proliferate, without paying the cost of building a labeled ground-truth set for every new configuration tested.

Why separating the four dimensions is the actual contribution

A single end-to-end accuracy score cannot distinguish a retrieval failure from a generation failure — both produce the same symptom, a wrong or unhelpful answer. Splitting the score changes what a team can do with a failing evaluation result:

  • Low context relevance, high faithfulness points at the retriever surfacing irrelevant passages that the generator then faithfully — and unhelpfully — summarized.
  • High context relevance, low faithfulness points at the generator fabricating or embellishing beyond what the retrieved passages actually support, even though the search results were good.
  • Low context recall points at an indexing or chunking problem — the needed information was never retrievable in the first place, regardless of how the generator behaved.
Diagnosis / component isolation

Same symptom, different fix

FIG 02
Diagnosing a RAG failure by which score dropped Low context relevance combined with high faithfulness points to a retrieval problem, while high context relevance combined with low faithfulness points to a generation problem, letting a team fix the actual failing component instead of guessing. LOW RELEVANCE + HIGH FAITHFULNESSRetriever surfaced the wrong passagesfix the retriever or the index HIGH RELEVANCE + LOW FAITHFULNESSGenerator fabricated beyond sourcesfix the prompt or the model USEa single end-to-end accuracy score cannot separate these two — the four RAGAS dimensions can
Cross-referencing the four RAGAS dimensions turns 'the answer was wrong' into a specific, actionable diagnosis of which pipeline component actually failed.

This is a debugging capability, not just a scoring capability: it tells a team which component to fix.

The metrics need their own validation

RAGAS's metrics are themselves computed by prompting a language model to judge another language model's output — faithfulness is typically assessed by decomposing an answer into individual claims and checking each against the source passages using an LLM judge. That means the metrics' reliability is bounded by the judge model's own competence on the specific domain being evaluated, and a metric validated on general-domain question answering is not automatically trustworthy on a specialized domain with different failure patterns. A high faithfulness score also only confirms the answer is grounded in what was retrieved — it says nothing about whether the retrieved source itself was correct or current.

What this changes in production

Pre-release regression gates

A RAG system's retriever, prompt, or underlying model changes constantly. Running the RAGAS dimensions as an automated gate before each release catches a silent grounding regression — a faithfulness score that quietly drops — that a spot-check of a handful of example outputs would likely miss.

Continuous production monitoring

Sampling live traffic and tracking faithfulness and relevance trends over time surfaces drift that only appears under real usage patterns, as opposed to the fixed evaluation set a team tested against at release time.

Root-causing user-facing quality complaints

When a support or product team receives a complaint about a wrong RAG answer, scoring that specific interaction across all four dimensions localizes the failure to retrieval or generation before deciding what to fix — rather than guessing.

A production checklist

Before relying on reference-free RAG metrics as a release gate, the team should be able to answer:

  • Has the LLM judge computing these metrics been validated against a human-labeled sample from this specific domain, not just a general benchmark?
  • Are context and faithfulness scores tracked separately, or collapsed into one aggregate "RAG quality" number that hides which component failed?
  • Does a high faithfulness score get treated as proof of factual correctness, or correctly understood as only proof of grounding to whatever was retrieved?
  • Is the evaluation re-run after every retriever, index, prompt, or model change, or only at infrequent, scheduled intervals?
  • What is the sampling strategy for continuous production monitoring, and does it cover the traffic segments most likely to fail?
  • Is there a human escalation path when automated metrics disagree with a user-reported quality issue?

Research referenced

Continue readingReturn to field notes →