LLM Evaluation: A Practical Guide for AI Applications

A benchmark tells you how a model does on someone else's test. An evaluation tells you how your product does on yours. Public leaderboards are useful when you choose a model, but they say little about whether your support assistant quotes the right refund policy or whether your agent calls the payment tool without approval. This guide walks through the whole practice, from a first smoke test to a release gate, and links to a deeper article for each step.

The evaluation lifecycle at a glance

Why LLM applications need their own kind of testing

Three properties make LLM features different from ordinary code. Outputs vary: the same input can give several acceptable answers, so exact-match assertions fail correct behaviour. Behaviour drifts: providers update models, and your prompts, retrieval index and tools change, so last month's pass says little about today. And failures are quiet: a hallucinated answer returns HTTP 200 with fluent text. Evaluation answers all three with statistics over many cases instead of single assertions, continuous re-runs instead of a launch review, and scorers that read the content instead of the status code.

Step 1: smoke-test the model

Start by sending ten or twenty realistic prompts to one or two candidate models and reading the answers yourself. Record tokens, latency and cost for each call. This is not evaluation yet, but it surfaces obvious problems (the model ignores your format, refuses benign requests, is too slow) before you invest in a dataset. Keep the prompts you used: they are the seed of your test set.

Step 2: put prompts under version control

An evaluation result is only meaningful if you know exactly which prompt produced it. Store prompts as immutable versions, point "staging" and "production" labels at versions, and have your application resolve the label at runtime. Rolling back then means moving a label, not redeploying. Prompt versioning and management covers the pattern.

Step 3: build an evaluation dataset

A dataset is a fixed list of inputs, each with the behaviour you expect: a reference answer, required facts, a required tool call, or simply "should refuse". Start small (30 to 100 cases you would be embarrassed to get wrong), version it, and grow it from production failures. Split off a holdout set you do not tune against. The dataset guide covers sources, size and splits.

Step 4: choose scorers

Use the cheapest scorer that measures what you care about. Deterministic checks (valid JSON, matches a schema, contains a required phrase, latency under a budget, no personal data patterns) are fast, free and exact; use them wherever the requirement is checkable by code. For qualities that need reading, such as correctness against a reference, relevance, tone or groundedness, use a judge model with a written rubric, and check the judge against human labels before you trust it. LLM evaluation metrics and LLM-as-a-judge go deeper.

Step 5: specialise for your architecture

• Retrieval-augmented generation: score retrieval and generation separately, with faithfulness, answer relevance and context relevance. See RAG evaluation metrics.
• Agents: check the path as well as the answer: which tools were called, with which arguments, in what order, with what authorization and within what budget. See AI agent evaluation.
• Safety and security: send adversarial probes and map the results to the OWASP Top 10 for LLM Applications. See AI red teaming.
• Hallucination: check claims against context and references. See hallucination testing.

Step 6: gate releases on regressions

Once scores exist, use them to decide. Save a known-good run as the baseline for each suite, run the suite on every change to prompt, model, retrieval or tools, and fail the build when the pass rate drops below the baseline or a threshold, or when attack success rises. Make "could not decide" (a missing baseline, an incomplete run) a failure too. Regression testing LLM apps in CI includes a GitHub Actions example.

Step 7: watch production

Offline datasets never cover everything users do. Instrument your application with OpenTelemetry using the GenAI semantic conventions, so every model call, tool call and agent step becomes a span with model, tokens and timing. Score a sample of live spans with the same judges you use offline (online evaluation), alert when the average drops below a threshold, collect user feedback, and route doubtful cases to human review. Turn the interesting failures into new dataset cases. LLM observability with OpenTelemetry explains the setup.

Human review still matters

Judges and code checks scale, but people define what good means. Review a sample of outputs by hand early on to write the rubric, label a few hundred cases to calibrate each judge (agreement, precision, recall, Cohen's kappa), and keep a review queue for production traces the judges are unsure about. Hamel Husain's widely shared essay describes the same three levels: unit-test-style assertions, human and model review of traces, and A/B tests for mature products (Husain, 2024).

Common mistakes

• Evaluating once at launch and never again.
• Using the model under test as its own judge.
• Tuning prompts against the whole dataset, so the score measures memorisation.
• Reporting a single average with no interval and no per-case failures.
• Treating benchmark scores as product quality.
• Testing only the final answer of an agent and ignoring the tool calls.

Choosing tools

You can do all of the above with open-source libraries and your own CI, and many teams start that way. Platforms add shared datasets, dashboards, review queues and governance on top. Our comparison of LLM evaluation and observability tools describes the main options fairly, including where each is the better choice.

Where NexGen QA fits

NexGen QA covers this lifecycle in one workspace on your own model keys: a model playground, prompt management with versions and labels, the Evaluation Kit (versioned datasets, scorers, scored runs from your code or CI, feedback, monitors and calibration), RAG metrics, red-team campaigns mapped to the OWASP LLM Top 10, agent execution against your own endpoint, LLM tracing over OpenTelemetry with online evaluations, human review queues, release gates, and exportable evidence. Free includes capped evaluation; Pro removes most caps and adds online evaluations, release gates and evidence. See pricing and LLM testing.