toolcall() ← all concepts

// concept · evals

A vibe can't tell you. A number can.

You changed the prompt — better or worse? Eyeballing a few outputs can't answer that. An eval turns "seems fine" into a score you can actually move.

// what an eval is

Two parts: a set of test cases that look like your real traffic (plus the tricky edge cases), and a grader that scores each output. Run it, get a number. Now a prompt tweak is "78% → 84%," not a shrug.

# the loop test cases + grader -> score # "did it get better?" becomes measurable

How to grade

Pick the cheapest grader that works:

deterministic (prefer this) # fast, reliable exact match · a rule · "contains X" model as judge (open-ended) # for free-form answers grades to a rubric — but calibrate it against human labels first
LLM-as-judge has biases. It can favor the first answer (position), longer answers (verbosity), and sometimes its own outputs. Strong judges hit ~80% agreement with humans (MT-Bench) — but only once you've checked the judge against human labels and use a fixed rubric. Grade with a different model than the one being tested.

The workflow

You don't need thousands of cases. Start with a few dozen good ones. Then every bug you find becomes a new case — so it can never come back quietly. Run the cheap graders on every change and track the score over time; a dip is a regression you caught before shipping.

# your test set grows from real failures ✓ refund question ✓ empty input ✓ wrong language + the bug from prod ← add it the moment you see it

Sources: Hamel Husain — Your AI product needs evals · Anthropic — build evaluations · OpenAI — eval best practices · Zheng et al. — LLM-as-a-judge (MT-Bench)

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click