How to Build an LLM Evaluation Dataset

Scores are only as meaningful as the cases behind them. This article covers what to put in an evaluation dataset, how big it needs to be, and how to keep it honest over time. It belongs to the LLM evaluation guide.

What a test case contains

{
  "id": "refund-45-days",
  "input": "Can I get a refund 45 days after buying?",
  "context": "Policy 4.2: refunds within 30 days; account credit after that.",
  "expected": {
    "reference": "No refund after 30 days; account credit is offered instead.",
    "must_include": ["30 days", "credit"],
    "should_refuse": false
  },
  "tags": ["refunds", "edge-case"],
  "source": "production-trace"
}

Keep the expected behaviour as specific as the case allows: an exact label, required facts, a required or forbidden tool call, or "should refuse". Tags let you report pass rates per slice (refunds, multilingual, long inputs), which is where regressions usually hide.

Where cases come from

• Product knowledge: the questions your support, sales or domain experts say matter most.
• Error analysis: failures you found by reading outputs.
• Production traces: real inputs, especially ones users rated badly or escalated.
• Reviewed annotations: traces a person has labelled in a review queue.
• Adversarial cases: injections, out-of-scope requests, malformed input, very long input.
• Synthetic generation: a model drafts variations of real cases. Review every draft before it enters the set; synthetic data inherits the generator's blind spots.

How many cases you need

There is no universal number. Published guidance ranges from a few dozen cases to start, to several hundred for a stable test set, to 500 or more when you are calibrating a judge and need a holdout. The useful way to think about size is statistical: with n cases, the uncertainty of a pass rate shrinks roughly with the square root of n. At 25 cases, a 92% pass rate has a 95% interval of about 75% to 98%; at 400 cases, about 89% to 94%. Size the set so that the smallest regression you care about is larger than that interval, and grow it where the stakes are highest.

Splits and versioning

1. Assign each case to a split (for example dev and holdout) with a deterministic hash of its id, so the split never changes by accident.
2. Tune prompts against dev only. Report holdout separately; if dev improves and holdout does not, you are overfitting.
3. Make versions immutable. Adding, editing or removing a case creates a new version, and every run records the version it used.
4. Compare runs only on the same dataset version, or flag the comparison as not like-for-like.

Keeping it representative

Datasets drift away from reality. Review a sample of production traffic every few weeks, add the patterns you were not covering, and retire cases that no longer reflect the product. Keep personal data out: redact or synthesise identifiers before a trace becomes a test case.

Datasets in NexGen QA

Evaluation Kit datasets have immutable versions with diffs, import and export in JSON, JSONL and CSV (with spreadsheet-safe CSV), deterministic hash splits, and builders that turn production traces or reviewed annotations into items. A dataset version can be published to Experiments as name@version. Synthetic generation on Pro produces drafts that are not saved until you commit them. Free allows 5 datasets with up to 500 items per version.