toolcall() ← all concepts

// concept · llm as judge

Grade your AI with another AI.

You can't read ten thousand model outputs by hand, so you use an AI to grade them. It scales — but if you don't swap the order, your judge is lying to you.

// the idea

Grading open-ended outputs (chat, summaries, code) by hand doesn't scale, and exact-match metrics don't fit. So you hand a strong model a rubric and let it score each answer — like a tireless teaching assistant with an answer key. It can score one answer against the rubric, or compare two and pick the better. Strong judges have agreed with human preferences over 80% of the time on MT-Bench — about the rate humans agree with each other.

Bias one: position

Show a judge answer A then B and it leans toward the first. Swap them and it leans first again. Same two answers — the order changed the winner, not the quality. In pairwise code judging, flipping the order has moved accuracy by more than ten points.

[ A , B ] -> picks A (the first) [ B , A ] -> picks B (the first) order alone flipped the winner

Bias two: length & self-love

Judges reward longer answers — a padded reply with the same facts often out-scores a tight, correct one. And they quietly prefer writing that sounds like their own (self-preference).

short + correct -> 4 / 5 long + padded -> 5 / 5 ★ # same facts, more words

De-bias it, then check it

None of this makes LLM-as-judge useless — it makes it something you calibrate:

· swap the order and average both runs · a strict, explicit rubric ("don't prefer longer") · use a different model as the judge · check its scores against a few real human ratings first
It's a proxy, not ground truth. Treat the judge like an instrument you calibrate against humans before you trust its numbers — not an oracle. This is the technique layer on top of the broader eval discipline.

Sources: Zheng et al. — Judging LLM-as-a-Judge (MT-Bench), NeurIPS 2023 · Justice or Prejudice? position bias · Self-Preference Bias

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click