LLM Evaluation Metrics: Which to Use and When

By NexGen QA TeamPublished November 3, 2026Updated November 4, 2026 9 min read

LLM evaluation metrics fall into four families: deterministic checks that code can verify (format, schema, required content, latency, cost), reference-based similarity (exact match, BLEU, ROUGE, embedding similarity), judge-based scores where a second model applies a rubric (correctness, relevance, faithfulness), and human ratings. Pick per criterion, cheapest reliable first.

There is no single score for "is my LLM app good". Each metric measures one property, has a cost, and fails in a predictable way. This article maps the common metrics to the questions they answer, as part of the LLM evaluation guide.

Family 1: deterministic checks

If code can decide it, let code decide it. Deterministic checks are free, instant and never disagree with themselves, which makes them ideal for CI.

Format: output parses as JSON; output validates against a JSON Schema; output matches a regex.
Content: contains or excludes a required phrase, a link, a disclaimer, a citation marker.
Tool use: the expected tool was called with the expected arguments (for agents).
Budgets: latency, input and output tokens, estimated cost per case.
Safety patterns: email addresses, phone numbers, card numbers and similar patterns in the output.

Family 2: reference-based similarity

When you have a reference answer, you can measure overlap. Exact match works for labels and numbers. BLEU (n-gram precision) comes from machine translation and ROUGE (recall-oriented) from summarisation; both reward word overlap, so they penalise valid paraphrases. Embedding similarity, such as BERTScore or cosine similarity between sentence embeddings, credits paraphrase but can score a fluent wrong answer close to a right one. Use these as cheap trend signals, not as release criteria, unless you have checked them against human judgment on your data.

Family 3: judge-based metrics

A judge model reads the input, the output and optionally a reference or context, and answers rubric questions. This is how most teams measure correctness, relevance, completeness, tone, faithfulness to context, refusal correctness and safety. G-Eval is a well-known variant that has the judge write evaluation steps before scoring (Liu et al., 2023). Judges are flexible but biased in measurable ways, so calibrate them against human labels; see LLM-as-a-judge.

Family 4: human ratings and user feedback

People define the ground truth that every other metric approximates. Use expert review to write rubrics and label calibration sets, and collect user feedback (thumbs, corrections, escalations) in production as a lagging but honest signal. Measure reviewer agreement; if two experts disagree often, the rubric needs work before any automated metric can match it.

Which metric for which task

Task	Start with	Add
Classification or extraction	Exact match, JSON Schema, per-field accuracy	Confusion matrix per class
Question answering with references	Judge correctness against reference	Embedding similarity as a trend
RAG	Faithfulness, answer relevance, context relevance	Retrieval precision and recall if you label chunks
Summarisation	Faithfulness to source, coverage of key points (judge)	ROUGE as a cheap trend
Agents and tool use	Task success, tool-call checks, budgets	Trajectory judge
Chat and multi-turn	Per-turn checks plus conversation-level judge (role adherence, knowledge retention)	User feedback
Safety	Attack success rate per category, refusal correctness	Toxicity and bias judges

Turning scores into decisions

Convert each metric to pass or fail per case with a threshold you have checked by reading examples near the line.
Report the pass rate with a confidence interval, not just the mean score.
Track cost and latency next to quality, so a quality gain that doubles cost is visible.
Gate releases on regressions against a baseline as well as on absolute thresholds.

Metrics in NexGen QA

The Evaluation Kit scorer library includes deterministic scorers (exact, contains, regex, JSON, JSON Schema, similarity, latency, tokens, cost, PII patterns, tool calls), an LLM-judge scorer with your own criteria, judge presets for hallucination, toxicity, bias, PII leakage, refusal, relevance, correctness and summary, RAG metrics, conversation-level presets, and word and character error rate for speech transcripts. Custom logic runs as an expression, a webhook to your endpoint, or a local scorer in your own process; on Pro it can also be a JavaScript function that runs in a WebAssembly sandbox with no network or file access. Pro adds scorer calibration against human labels with accuracy, precision, recall, F1 and Cohen's kappa.

Frequently asked questions

Are BLEU and ROUGE good metrics for LLM apps?

They measure word overlap with a reference, so they penalise valid paraphrases. They are cheap trend signals for translation and summarisation, but rarely good release criteria for LLM applications.

What is G-Eval?

G-Eval is an LLM-as-a-judge method in which the judge model writes evaluation steps for a criterion and then scores the output, introduced by Liu et al. in 2023.

Which metrics should I use for a RAG application?

Faithfulness (groundedness), answer relevance and context relevance, plus context recall where you have labelled relevant passages.

Sources

NexGen QA puts AI evaluation (tracing, datasets and scorers, red teaming, human review, release gates and evidence) next to 35+ software testing modules, on your own model keys.

See Free and Pro plans Read the LLM evaluation guide