toolcall() ← all concepts

// concept · evals

A vibe can't tell you. A number can.

You changed the prompt — better or worse? Eyeballing a few outputs can't answer that. An eval turns "seems fine" into a score you can actually move.

// what an eval is

Two parts: a set of test cases that look like your real traffic (plus the tricky edge cases), and a grader that scores each output. Run it, get a number. Now a prompt tweak is "78% → 84%," not a shrug.

# the loop test cases + grader -> score # "did it get better?" becomes measurable

How to grade

Pick the cheapest grader that works:

deterministic (prefer this) # fast, reliable exact match · a rule · "contains X" model as judge (open-ended) # for free-form answers grades to a rubric — but calibrate it against human labels first
LLM-as-judge has biases. It can favor the first answer (position), longer answers (verbosity), and sometimes its own outputs. Strong judges hit ~80% agreement with humans (MT-Bench) — but only once you've checked the judge against human labels and use a fixed rubric. Grade with a different model than the one being tested.

The workflow

You don't need thousands of cases. Start with a few dozen good ones. Then every bug you find becomes a new case — so it can never come back quietly. Run the cheap graders on every change and track the score over time; a dip is a regression you caught before shipping.

# your test set grows from real failures ✓ refund question ✓ empty input ✓ wrong language + the bug from prod ← add it the moment you see it
Have an eval already? The rest of this lane is four separate ways one can be green and still be lying to you. It measures a metric you guessed — the list was written before anyone read a failure. It reads the reply, not the row — scoring the story instead of the fact. It ran each case once — one run is a sample, not a result. It has too few cases — the score carries more noise than the change you are measuring. And grading with a model covers the judge above in depth.

Sources: Hamel Husain — Your AI product needs evals · Anthropic — build evaluations · OpenAI — eval best practices · Zheng et al. — LLM-as-a-judge (MT-Bench)

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click