Evaluating Agents

Time: ~12 minutes. Tangible win: write one eval with a grader, run it several times, and interpret the result as a pass rate rather than a single pass or fail.

An agent that worked once has not proven much. Agents take many actions across many turns, and the same input can lead to different paths. Anthropic's position is that evals let a team catch behavior problems before users do, and that agents need evals more than single-turn models because the failure modes are harder to see.

Primary source Demystifying evals for AI agents, Anthropic. Read it before you design your first eval. It covers graders, capability and regression evals, and metrics for non-deterministic runs.

An eval is an input plus a grader

An eval gives the agent a task and applies grading logic to what it produces. The grading logic is the part people underestimate. Anthropic describes three kinds of grader, and each suits different outputs.

GraderHow it worksBest for
Code-basedA script checks the result, such as a test suite or a query comparisonOutputs with a verifiable answer
Model-basedA model scores the output against a written rubricOpen-ended outputs such as summaries
HumanA person reviews the outputCalibrating the other two graders, and spot checks

Prefer code-based graders wherever the answer is checkable. They are cheap, repeatable, and do not drift. Use a model-based rubric only where code cannot judge the quality, and check it against human review on a sample.

Two kinds of eval

A capability eval asks what the agent can do now, and is expected to pass only some of the time at first. A regression eval protects behavior that already works, and should stay near full pass. Keep both. A capability eval shows progress, and a regression eval shows that a change did not break anything.

Run each task more than once

Because runs are non-deterministic, one result proves little. Run each task several times and report the rate. Anthropic discusses metrics for this, including pass@k, which asks whether at least one of k attempts succeeds, and pass^k, which asks whether all k attempts succeed. For a customer-facing agent, pass^k is usually the stricter and more honest measure.

Check the definitions before you rely on them The exact grader and metric definitions in the Anthropic article should be read in the original before you use them in a report. Confirm the wording against the page.

Practice

  1. Write five tasks your agent should handle, drawn from real failures you have seen. For each, write the expected outcome in one sentence.
  2. For one task with a checkable answer, write a code-based grader. It should return pass or fail with a reason.
  3. Run that task three times. Record the pass rate. Then decide whether the rate is good enough to ship, and what you would need to see to change that decision.

Retrieval check

An agent passed a task on one run. Does that show it is reliable?

Correct. Agent runs vary, so you need several runs to estimate a rate. One pass could be luck.

Not quite. Agent output varies from run to run. A single pass says little about the rate, and a model grader does not fix that.

An agent writes a SQL query. Which grader fits best?

Correct. The result is checkable, so a script can verify it cheaply and repeatedly. Tone is not the thing to measure here.

Not quite. The query result can be checked by code. A human on every run is slow, and tone does not measure whether the query is correct.

Ask the agent: "Help me write a code-based grader for this task: [describe the task and expected output]." Next lesson covers workflow patterns, the reusable shapes that many agent systems are built from.