Practical guides to evaluating LLM applications (datasets, metrics, judges, RAG, agents, red teaming, CI gates and observability) and to choosing and running a QA platform.
Start hereWhat LLM evaluation is and how to do it: datasets, scorers, LLM-as-a-judge, RAG and agent metrics, red teaming, CI release gates and production monitoring.
Read the guideAn eight-step process for evaluating LLM apps: success criteria, error analysis, a versioned test set, scorers, baselines, specialised suites and production checks.
Read articleThe four families of LLM evaluation metrics (deterministic checks, reference similarity, judge-based scores and human ratings) and which to use for each task.
Read articleWhat goes in a golden dataset for LLM evaluation, where cases come from, how many you need (with confidence intervals), and how to version and split it.
Read articleHow LLM-as-a-judge works, how to write judge prompts, calibrate against human labels with kappa, and mitigate position, length and self-preference bias.
Read articleHow to evaluate retrieval-augmented generation: faithfulness, answer relevance, context relevance and recall, how to read them together, and common pitfalls.
Read articleEvaluate AI agents on task success and trajectory: tool selection, arguments, order, authorization and budgets, plus multi-turn scenarios and MCP server checks.
Read articleWhat AI red teaming is, the OWASP Top 10 for LLM Applications 2025 explained, how to run a campaign step by step, and how to report attack success honestly.
Read articlePrompt versioning with immutable versions and staging/production labels: how to set it up, promote on evaluation results, review diffs and roll back safely.
Read articleRun LLM evals in CI: what triggers a run, baseline comparison, threshold policies, exit codes, a GitHub Actions example and how to read a failed gate.
Read articleTrace LLM calls, tools and agents with OpenTelemetry GenAI semantic conventions: what to capture, setup steps, privacy, online evaluation and alerts.
Read articleA fair comparison of Promptfoo, DeepEval, Ragas, Langfuse, LangSmith, Arize Phoenix, Braintrust and NexGen QA, including when each is the better choice.
Read articleHow to evaluate AI testing platforms in 2026: the capability checklist that matters, where traditional automation stops, and what to ask before you buy.
Read articleNexGen QA vs Katalon on test authoring, AI capabilities, module breadth, compliance and deployment, with guidance on which fits which team.
Read articleWhat federal teams need from a QA platform: NIST 800-53 control mapping, POA&M tracking, ATO evidence, data residency answers and precise authorization language.
Read articleTest an LLM for hallucinations: groundedness vs factuality, claim-level checks, reference answers, SelfCheckGPT, a pytest example and continuous runs.
Read article