LLM Regression Testing in CI: Baselines and Gates

Standard CI catches code that does not compile or tests that throw. It does not catch a prompt edit that makes answers 8% less accurate, or a provider model update that starts leaking the system prompt. An evaluation gate closes that gap. This article shows the pieces and a working GitHub Actions example. It belongs to the LLM evaluation guide.

What triggers a run

• Changes to prompt files or prompt versions.
• Changes to model name, parameters or provider.
• Changes to retrieval: chunking, embeddings, top-k, filters, reranker.
• Changes to tools or agent logic.
• A schedule, because providers change models without telling you.

The gate logic

1. Run the suite on a pinned dataset version.
2. If any case was not evaluated (judge error, timeout, missing item), the run is incomplete and cannot pass.
3. Compare with the suite's baseline run on the same dataset version: pass rate change and the list of regressed cases.
4. Apply policies: for example, pass rate at least 90%, no more than 2 points below baseline, attack success rate at most 5%.
5. Exit 0 to pass, 1 to block, and 2 when no decision could be made (no baseline, incomplete run, API error). Treat 2 as a failure.
6. When a run is accepted as the new normal, save it as the baseline explicitly; never move the baseline automatically.

A GitHub Actions example

This workflow runs only when prompt, eval or model configuration files change, runs a pytest-based eval suite, and fails the job on regressions. The --baseline option is one you add yourself in conftest.py (pytest_addoption) so tests can load the last accepted results; the provider key comes from repository secrets.

name: llm-evals
on:
  pull_request:
    paths: ["prompts/**", "evals/**", "app/llm/**"]
  schedule:
    - cron: "0 6 * * 1"   # weekly, to catch provider-side model changes
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r evals/requirements.txt
      - name: Run evaluation suite and compare with baseline
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest evals/ -q --baseline evals/baseline.json

Keeping CI evals fast and affordable

• Use a small, high-signal smoke suite on every pull request and the full suite nightly.
• Run deterministic scorers on everything and judges only where needed.
• Cache outputs for unchanged inputs and prompt versions.
• Report cost per run so the suite's own spend is visible.
• Post a short summary on the pull request: pass rate, delta, regressed case ids, a link to details.

Reading a failed gate

Open the regressed cases first. A real regression shows a pattern (all refund questions, all long inputs). Scattered single-case flips on a small dataset are often noise; check the confidence interval and, for variant comparisons, a significance test before rejecting a change. If the dataset expectation was wrong, fix the dataset in a new version and re-baseline deliberately.

Release gates in NexGen QA

On Pro, NexGen QA release gates compare a run with a saved baseline for its suite, case by case, and apply threshold policies such as minimum pass rate and maximum attack success rate. Gates work for experiments, agent runs, red-team campaigns, multimodal runs, evaluation jobs, imported agent evaluations and Evaluation Kit runs, and fail closed. From any CI system, a gate check over the REST API returns a decision; the bundled CLI exits 0 to pass, 1 when blocked and 2 when no decision could be made. With --format junit the CLI writes one test case per eval case plus the gate decision, and templates for GitLab CI, Jenkins, Azure DevOps and GitHub Actions publish that report even when the gate fails; --format json writes a machine-readable result. With the NexGen GitHub App, eval suites can run in a Continuous Testing pipeline that posts a GitHub check run. The Python and TypeScript SDKs and CLI are provided by NexGen and are not yet published to PyPI or npm. See AI evaluation in NexGen QA for plan details.