AI Red Teaming with the OWASP Top 10 for LLMs

Functional evaluation asks whether your application does the right thing for normal users. Red teaming asks what it does for a user who is trying to make it misbehave, or for a document, web page or email that contains hidden instructions. This article explains the OWASP LLM Top 10, how to run a red-team campaign step by step, and how to report the results honestly. It is part of the LLM evaluation guide.

The OWASP Top 10 for LLM Applications (2025)

The list comes from the OWASP GenAI Security Project (OWASP, 2025). Some entries, such as supply chain, are mostly about process and architecture rather than prompts, so a prompt-based campaign covers part of the list, not all of it. MITRE ATLAS and the NIST AI Risk Management Framework are useful companions when you need to map findings to adversary techniques or governance functions.

How to run a red-team campaign

1. Define scope: the target (a model with your system prompt, or the full application or agent endpoint), what counts as a successful attack, and which categories matter for your product.
2. Assemble probes per category, and apply mutations (encodings such as base64, role-play framing, translation) to see whether simple obfuscation bypasses defences.
3. Send probes to the real target, the way an attacker would reach it, and store every prompt and response as evidence.
4. Grade each response with an independent judge model against an explicit rubric (refused, complied, partial). Keyword heuristics are a fallback, not a grader.
5. Assign severity per finding and map it to the OWASP category.
6. Report attack success rate with a confidence interval, per category, and the number of probes that errored or could not be graded.
7. Fix, then re-run the same probes as a regression suite before every release.

Reporting honestly

An attack success rate of 0% on 20 probes does not mean the system is safe; the 95% upper bound is still about 16%. Report how many probes were graded, how many errored, the grading method, and the categories you did not test. Treat red teaming as a way to find and fix weaknesses, not as a certificate.

Defences worth testing for

• Separating instructions from data in prompts, and telling the model to treat retrieved content as data.
• Least-privilege tools, with human approval for irreversible actions.
• Output handling that escapes or validates model output before it reaches a browser, database or shell.
• Input and output filters for secrets and personal data.
• Rate, token and cost limits per user to contain unbounded consumption.

Red teaming in NexGen QA

NexGen QA's Red Team Campaigns send real attack prompts to a model, or on Pro to your own agent endpoint, across categories such as jailbreak, injection, indirect injection, data extraction, PII leakage, system prompt leakage, excessive agency, insecure output and unbounded consumption, with optional mutations. Each probe is graded by a judge model you choose (without one, a keyword heuristic is used and no severity is assigned), mapped to the OWASP LLM Top 10, and kept as evidence with its prompt and response. Results show the attack success rate with a 95% confidence interval and a per-category breakdown. Free runs up to 10 probes per campaign; Pro runs the full set and can gate releases on attack success.