toolcall() ← all concepts

// concept · evals

Your agent said it worked. Check the database.

The run ends with "refund processed" and your eval goes green. Then you open the table and nothing happened. The reply and the result are two different facts, and most evals only ever read the first one.

// two different facts

Agent evaluation has two objects, and they are easy to collapse into one. Anthropic's engineering write-up names them separately:

transcript the complete record of a trial — outputs, tool calls, reasoning, intermediate results outcome the final state in the environment at the end of the trial

An eval that reads the transcript is grading what the agent said it did. An eval that reads the outcome is grading what happened. Those diverge more often than anyone is comfortable with, because a language model is extremely good at producing the sentence that should follow a success — whether or not the success occurred.

The flight that was never booked

The canonical example, near-verbatim from the source, is worth keeping in your head as an image:

A flight-booking agent might say "Your flight has been booked" at the end of the transcript — but the outcome is whether a reservation exists in the environment's SQL database.
transcript: "Your flight has been booked" ✓ reads great database: 0 rows ✗ nothing happened

Same for support: an agent can claim success verbally while never processing the refund. Grading on conversational quality misses genuine task completion — it scores the story, not the fact.

Make the test go and look

The fix is not a better rubric. It is a different assertion. Your test case ends by querying the environment it was supposed to change.

# not this assert "booked" in reply # this row = db.query("SELECT * FROM reservations WHERE ...") assert row # missing row = FAIL, however good the answer sounded

The rule that follows: a missing row is a failure no matter how confident the reply was. That sounds obvious written down, and it is the single most common thing missing from agent evals in the wild, because asserting on text is so much easier to write than standing up an environment you can inspect afterwards.

Don't score pass or fail

The second half, and the part people skip. A binary score throws away the information you most need while debugging.

Build in partial credit. A support agent that correctly identifies the problem and verifies the customer but fails to process a refund is meaningfully better than one that fails immediately.
✓ identified the problem ✓ verified the customer ✗ processed the refund score 2/3, not "fail"

Both of those runs are red under pass/fail, and one of them is nearly working. Partial credit is what lets you see a regression move.

What this doesn't say

It does not say transcripts are useless. The transcript is how you find out why it failed — which tool errored, what it did instead, where the reasoning went sideways. It is just not the proof that it succeeded. Keep it beside the row, not instead of it. Assessing the trajectory has its own value too: it can reveal unsafe reasoning even on runs where the outcome came out fine.

Related: an eval turns "seems fine" into a number is the prerequisite — this page assumes you already have one. Your eval set is too small to trust covers how many cases you need; this covers what a single case should assert. And grading with a model is for the genuinely subjective calls — note that the check on this page isn't one. A row exists or it does not.

Source: Anthropic — Demystifying evals for AI agents. Verified 2026-08-30.

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click