How to Evaluate AI Agents: Outcomes and Tool Calls

An agent is an LLM that decides which actions to take: look up an order, query a database, send an email, issue a refund. Evaluating it like a chatbot, by reading the final message, misses most of what can go wrong. This article lays out a layered approach that starts with deterministic checks on tool calls and adds judges only where needed. It is part of the LLM evaluation guide.

What to measure

Layer 1: deterministic trajectory checks

Have the agent report its trajectory (each tool call with arguments, status and whether it was authorized) along with its output. Most agent requirements can then be checked with code, which is exact and free. Trajectory matching can be strict (same calls in the same order), unordered, or a subset or superset of a reference, depending on how much freedom the agent should have.

{
  "id": "refund-over-limit",
  "input": "Refund my order A-1042, it arrived broken.",
  "expect": {
    "requiredTools": ["lookup_order"],
    "forbiddenTools": ["delete_account"],
    "toolOrder": ["lookup_order", "issue_refund"],
    "requireAuthorization": true,
    "outputIncludes": ["manager approval"],
    "budgets": { "maxSteps": 6, "maxLatencyMs": 15000, "maxCostUsd": 0.05 }
  }
}

Layer 2: judge the trajectory where code cannot

Some questions need reading: was the plan sensible, did the agent recover well from a tool error, did it ask a clarifying question when the request was ambiguous? Give a judge model the user input, the tool definitions and the ordered tool calls, and ask narrow questions with binary answers. Calibrate it against human labels as you would any judge (see LLM-as-a-judge).

Layer 3: multi-turn scenarios and simulated users

Many agent failures only appear over several turns: forgetting a constraint the user gave earlier, drifting out of role, looping. Script multi-turn scenarios with assertions per turn and for the whole conversation, and for broader coverage let a model play a user persona with a goal, then score the transcript. Simulated users are useful for breadth, but they are not real users; keep a scripted core you can reproduce exactly.

Evaluating tools themselves, including MCP servers

If your agent uses Model Context Protocol (MCP) servers, test those too: that the server handshakes and lists its tools, that tool descriptions are clear and do not contain instructions aimed at the model (a "tool poisoning" pattern), and that tool calls return what the agent expects. Skip destructive tools in automated runs unless you have a sandbox.

Security belongs in agent evaluation

Agents widen the attack surface: an injected instruction in a retrieved page can now trigger an action, not just a bad sentence. Include red-team scenarios for goal hijacking, tool misuse and privilege abuse; the OWASP Top 10 for LLM Applications lists Excessive Agency (LLM06) as its own risk, and OWASP has published a separate Top 10 for agentic applications. See AI red teaming.

A practical rollout

1. Make the agent emit a structured trajectory for every run, in tests and in production traces.
2. Write 20 to 50 scenarios with deterministic expectations for the tasks that matter most.
3. Add budgets so a looping or expensive agent fails visibly.
4. Add a small number of judge checks for plan quality and recovery.
5. Add multi-turn and simulated-user scenarios for conversational agents.
6. Gate releases on the scenario pass rate against a baseline, and trace production to find new scenarios.

Agent evaluation in NexGen QA

On Pro, NexGen QA's agent execution sends scenarios to your own HTTPS agent endpoint (credentials are encrypted) and expects an output plus an optional trajectory of tool calls with arguments, status and authorization. Scenario assertions include requiredTools, forbiddenTools, toolOrder, expectedToolCalls, outputIncludes, outputJsonSchema and budgets, and a run can stop at a cost budget. The Evaluation Kit adds scripted multi-turn scenarios, simulated users and MCP server checks over Streamable HTTP (stdio servers are not supported). Agent runs can be gated in CI and traced in production with OpenTelemetry. Free includes Agent Evaluation of imported traces within its caps.