AI Evaluation: From Traces to Release Gates

An LLM feature can pass every unit test and still give a wrong answer, drift after a provider update or fall apart on the fifth turn of a conversation. This page covers the tools NexGen QA gives you for each of those, what each one is limited to, and which plan includes it.

Trace a framework app in a few lines

NexGen QA accepts OpenTelemetry traces over OTLP/HTTP with a write-only ingest key, and reads OpenTelemetry GenAI attributes first, then OpenInference and OpenLLMetry attributes. In Python, one call to tracing.configure in the nexgen_eval SDK adds the NexGen exporter and the OpenInference instrumentor for your framework, keeping any OpenTelemetry provider your app already has. In TypeScript, configureTracing registers the exporter, and you instrument OpenAI and LangChain.js with the OpenInference packages. The SDKs are provided to customers and are not yet published to PyPI or npm; any OpenTelemetry SDK can send to the same endpoint.

• Python: LangChain, LlamaIndex, the OpenAI Agents SDK, OpenAI and Anthropic.
• TypeScript: OpenAI and LangChain.js.
• Model, provider, tokens, estimated cost, tool calls and agent names are read from the span attributes.

Privacy controls apply at ingest. Workspace admins choose whether prompts and responses are stored; with content capture off, only an allowlist of metadata such as model, token counts, operation and status is kept. PII redaction, on by default, removes emails, phone numbers, SSNs, card numbers, IP addresses and API keys before storage, and a session id that itself looks like personal data is stored as a keyed hash so the conversation still groups.

Read a conversation turn by turn

Spans that carry the same session id are grouped into a session. The sessions list shows recent conversations with their totals, and each session opens as a turn-by-turn timeline with the input and output of every turn. A conversation can also become dataset items for later evaluation. On Pro, session-level online evaluation scores a whole conversation once it has gone quiet for an idle window you choose, with a judge rubric over the recent transcript and checks on turn count, total cost, total latency, errors and PII. If the conversation continues, it is scored again and the new verdicts replace the old ones.

Score with datasets, scorers and public benchmarks

Versioned datasets hold your test cases, built by hand, imported from JSON, JSONL or CSV, or taken from production traces and reviewed annotations. The scorer library covers deterministic checks, LLM-judge scorers with your own criteria, safety and quality judge presets, RAG metrics and conversation-level presets.

• Public benchmark library: import a seeded sample of up to 500 rows from one of 10 openly licensed benchmarks (GSM8K, TruthfulQA, SQuAD, MMLU, ARC-Challenge, HumanEval, MBPP, IFEval, Do-Not-Answer and RealToxicityPrompts) into a versioned dataset, with its license and citation. Rows are fetched on demand from the Hugging Face dataset viewer. Sampled scores are for comparing your own models and prompts and are not comparable to official leaderboards.
• Speech transcript accuracy: word error rate and character error rate scorers with configurable normalization and thresholds. Optionally, audio stored through multimodal evaluation (Pro) is transcribed first on your own OpenAI or Google key.
• Code scorers (Pro): write a score function in JavaScript. It runs in a WebAssembly sandbox with no network, file or host access, a fresh runtime per call, a 32 MB memory limit and a time limit of up to 2 seconds. Errors and timeouts are never a pass.

Improve a prompt, with a person in charge (Pro)

1. Pick a saved prompt version and a dataset. NexGen QA quotes the run and refuses it if it would exceed your credit or spend cap, before any model call.
2. An optimizer model you choose reads the failing cases and proposes up to 5 rewrites. A rewrite that drops or adds a template variable is rejected.
3. The original and the candidates run as one experiment over the same cases, scorers and judge.
4. Candidates are ranked by pass rate, each compared with the original by a paired McNemar test. One is recommended only when the evidence supports it.
5. You accept a candidate, which becomes one new prompt version. Labels such as production stay where they are until you move them.

Watch production for drift (Pro)

Evaluation monitors compare a current window of live traffic with a baseline window and alert through email, Slack or a webhook when a monitor starts or stops firing. A quiet hour or a missing key never triggers a false recovery.

• Score drift: population stability index (PSI) of online evaluation scores, for spans and sessions, with a minimum of 30 scores per window.
• Pass-rate drop: the fall in pass rate, in percentage points.
• Latency PSI, plus monitors on errors, cost, tokens, volume and negative feedback.
• Embedding drift (opt-in): a seeded sample of up to 500 prompts or responses per window, embedded with your own OpenAI or Google key and compared by energy distance. It needs content capture on, runs hourly with call and character caps, and stores only aggregate numbers, never the sampled text or embeddings.

Gate releases in your CI

The nexgen-eval CLI in both SDKs scores your outputs as a run and exits 0 when the gate passes, 1 when it fails and 2 on a usage, API or connection error, which you should treat as a failure. It fails closed: with no threshold set, any failed or not-evaluated case, or a run with no results, fails the job. With --format junit it writes one test case per eval case plus a release gate decision, so results show in your CI's test report; --format json writes a machine-readable gate result. Templates for GitLab CI, Jenkins, Azure DevOps and GitHub Actions publish the report even when the gate fails and read the API key from the CI system's secret store. Comparing a run with an approved baseline, case by case, is part of release gates on Pro. NexGen QA does not start your pipelines; your CI runs the CLI.

What is on Free and what needs Pro

Current limits

• The Python and TypeScript SDKs and CLI are not yet published to PyPI or npm. You install them from the SDK package NexGen provides.
• Code scorers are JavaScript only and run inside the hosted NexGen service, not in a separate container per customer.
• The prompt optimizer runs one round per request and scores with case expectations and the judge, not Evaluation Kit custom scorers.
• Embedding drift needs stored content and your own OpenAI or Google key; without them the monitor shows as unavailable.
• There is no model gateway or proxy: tracing works by instrumenting your app, and NexGen QA is hosted only.