How to Evaluate an LLM Application, Step by Step

This is the practical, step-by-step version of our LLM evaluation guide. It assumes you have a working prototype, such as a support assistant, an extraction pipeline or an agent, and want to know whether it is good enough to ship and whether the next change makes it better or worse.

Step 1: write the success criteria first

Before any tooling, write three to six criteria in plain language, each one testable. For a support assistant: answers only from the policy text; cites the section it used; asks for the order ID before discussing a refund; refuses legal advice; replies in under four seconds at p95; never shows another customer's data. Vague goals such as "helpful" become rubric questions a person or judge can answer consistently: does the answer resolve the user's question without asking for information already given?

Step 2: look at real outputs before you automate

Read 50 to 100 outputs from your prototype and note every failure in a spreadsheet with a short label: wrong policy, missing citation, too long, ignored the format. Group the labels. This error analysis tells you which criteria matter in practice and which scorers to build first; teams that skip it tend to automate the metrics that are easy to compute rather than the ones that reflect user pain.

Step 3: build a versioned test set

Turn the inputs you read into test cases, each with the expected behaviour: a reference answer, a list of required facts, a required tool call or "should refuse". Cover the common paths, the edge cases from your error analysis, and adversarial inputs. Version the set so a score change can be traced. See how to build an LLM evaluation dataset.

Step 4: score with the cheapest reliable method

LLM evaluation metrics explains the main metric families, and LLM-as-a-judge explains how to make judge scores trustworthy.

Step 5: run, then read the failures

Run the whole set against the current version and record the pass rate per scorer, with a confidence interval, plus latency and cost. Then read every failure. Some will be real bugs, some will be wrong expectations in the dataset, and some will be judge mistakes. Fix the dataset and the judge before you start changing prompts, or you will optimise against noise.

Step 6: compare changes against a baseline

Save the run as the baseline. For every change to prompt, model, retrieval or tools, run the same dataset version and compare case by case: which cases regressed, which improved, how the pass rate moved and whether that move is larger than chance. A significance test such as McNemar's on paired pass/fail results, or bootstrap intervals, keeps you from shipping a "4-point improvement" that is noise. Automate this in CI; see regression testing in CI.

# Minimal case-by-case comparison between two runs
def compare(base: dict[str, bool], head: dict[str, bool]):
    shared = base.keys() & head.keys()
    regressed = sorted(k for k in shared if base[k] and not head[k])
    improved = sorted(k for k in shared if not base[k] and head[k])
    rate = lambda run: sum(run[k] for k in shared) / len(shared)
    return {"delta_points": round((rate(head) - rate(base)) * 100, 1),
            "regressed": regressed, "improved": improved}

Step 7: add specialised suites

• If you retrieve documents: RAG evaluation metrics.
• If the model calls tools or acts: AI agent evaluation.
• If users can send it arbitrary text or it reads web pages and files: AI red teaming.

Step 8: keep evaluating in production

Trace production calls, score a sample with the same judges (online evaluation), alert when quality drops, and feed new failures back into the dataset. Without this loop, the test set slowly stops representing what users actually send. See LLM observability with OpenTelemetry.

Doing this in NexGen QA

The Evaluation Kit in NexGen QA holds versioned datasets (import JSON, JSONL or CSV, or build from traces and reviewed annotations), a scorer library with deterministic checks, judge presets and RAG metrics, and scored runs you start from your code or CI over a REST API with an OpenAPI document. Runs show the pass rate with a 95% interval, per-scorer results and per-item failures, and can be compared with another run. On Pro, release gates turn the comparison into a CI decision. Every judge call uses your own provider key. See pricing for the Free caps.