RAG Evaluation Metrics: Faithfulness, Relevance, Recall

By NexGen QA TeamPublished November 3, 2026 10 min read

RAG evaluation measures the two halves of a retrieval-augmented generation system separately: retrieval (did it fetch the right passages? context relevance, precision and recall) and generation (is the answer supported by those passages and does it address the question? faithfulness and answer relevance). Scoring them apart tells you which half to fix.

A RAG assistant can fail in two different places: the retriever brings back the wrong documents, or the generator ignores, misreads or embellishes the right ones. A single "answer quality" score cannot tell those apart, which is why the standard RAG metrics split the pipeline. This article explains each metric, how it is computed, and how to read them together. It belongs to the LLM evaluation guide.

The core metrics

Metric	Half	Question	Typical computation
Faithfulness (groundedness)	Generation	Is every claim in the answer supported by the retrieved context?	Split answer into claims; judge labels each as supported or not; supported / total
Answer relevance	Generation	Does the answer address the question asked?	Judge rating, or similarity between the question and questions generated from the answer
Context relevance (precision)	Retrieval	How much of what was retrieved is relevant to the question?	Judge labels each passage or sentence as relevant or not
Context recall	Retrieval	Did retrieval find everything needed to answer?	Needs a reference answer or labelled relevant passages
Hallucination rate	Generation	How often does an answer contain an unsupported claim?	Share of answers with at least one unsupported claim

The RAGAS paper introduced reference-free versions of faithfulness, answer relevance and context relevance, which is why they are now common across tools (Es et al., 2023). Terminology varies between frameworks, so check how a tool defines a metric before comparing numbers across tools.

Reading the metrics together

Pattern	Likely cause	Fix to try first
High faithfulness, low answer relevance	Generator sticks to context that does not answer the question	Improve retrieval or query rewriting
Low faithfulness, high context relevance	Generator embellishes good context	Tighten the prompt: answer only from context, cite passages
Low context relevance	Retriever returns noise	Chunking, embedding model, filters, reranking
Low context recall	Relevant documents missing or not found	Index coverage, hybrid search, top-k
High scores, users unhappy	Dataset does not reflect real questions	Add production questions to the dataset

How to run a RAG evaluation, step by step

Collect 50 to 200 real questions, including some whose answer is not in your documents (the correct behaviour is to say so).
For a subset, write reference answers or mark the passages that should be retrieved, so you can measure recall.
Log for every run: question, retrieved passages with ids, final answer, model and prompt version.
Score faithfulness and answer relevance on every case with a judge model that is not the generator.
Score context relevance per passage, and recall where you have labels.
Read the lowest-scoring cases and classify them as retrieval or generation failures before changing anything.
Re-run on the same dataset version after each change to chunking, embeddings, top-k, prompt or model.

Pitfalls

Counting word overlap as faithfulness. Lexical overlap is a cheap signal, but paraphrases score low and negations score high.
Letting the generator grade itself. A model is lenient with its own claims; use an independent judge model.
Ignoring "not in context" questions. A RAG system that answers everything is hallucinating some of the time.
Evaluating with a context window that differs from production (different top-k, filters or reranker).
Treating a citation as proof. Check that the cited passage actually says what the answer claims.

RAG testing in NexGen QA

The RAG Testing tab in AI Analysis takes your context and questions, splits each answer into claims and has a judge model you choose check every claim against the context, reporting faithfulness, answer relevance, context relevance and hallucination rate, with per-claim results and source passages. Lexical overlap is shown separately and is never counted as faithfulness, and the UI warns when the judge is the same model as the target. The same RAG metrics are available as scorers in the Evaluation Kit for dataset runs from your code or CI. RAG Testing is on Pro; Evaluation Kit scorers are on Free within its caps.

Frequently asked questions

What is faithfulness in RAG evaluation?

Faithfulness is the share of claims in an answer that are supported by the retrieved context. It is usually computed by splitting the answer into claims and having a judge model verify each one.

What is the difference between answer relevance and context relevance?

Answer relevance measures whether the answer addresses the question. Context relevance measures whether the retrieved passages are relevant to the question. The first is about generation, the second about retrieval.

Why do my RAG scores look good when users are unhappy?

Usually because the test questions do not match what users ask. Add real production questions, including ones the documents cannot answer, to the dataset.

Sources

NexGen QA puts AI evaluation (tracing, datasets and scorers, red teaming, human review, release gates and evidence) next to 35+ software testing modules, on your own model keys.

See Free and Pro plans Read the LLM evaluation guide