Your eval set is too small to trust

// the assumption

You check your AI by running it on a set of test questions and counting the passes. Sixteen out of twenty. You change the prompt, run it again, seventeen pass. That reads like an improvement you can act on — the number went up, and the number is the evidence.

# what we think we measured before 16 / 20 = 80% after 17 / 20 = 85% # "+5 points. ship it."

A score is a sample, not a measurement

Those twenty questions are a sample of every question your system could face. So the score is a sample statistic, and it carries sampling error — the same reason a poll of twenty people comes with a margin either side. Scoring cases pass/fail is a binomial proportion, and the uncertainty in that proportion shrinks only as the set grows.

At 80% with twenty cases, the 95% band is roughly ±17 points. Your true score could be anywhere from about 13 to 19 out of 20. One extra pass disappears inside that.

# 16/20 = 80%, 95% band ≈ ±17pp |←—————— the band ——————→| 13 15 17 19 / 20 ▲ ▲ before after ← both live in here

How wide, by size

The band narrows with the square root of the set size, which is a punishing curve: to halve your uncertainty you need four times the questions.

# 95% band at a score near 80% 20 questions ≈ ±17 points ████████████████ 100 questions ≈ ±8 points ███████ 400 questions ≈ ±4 points ███ # band = 1.96 * sqrt( p(1-p) / N )

Reproduce it in one line — nothing here is cited, it's arithmetic:

python3 -c " import math def band(p,n): return 1.96*math.sqrt(p*(1-p)/n)*100 print(band(.8,20), band(.8,100), band(.8,400))" # 17.5 7.8 3.9

These are the normal approximation at p=0.8. They're the standard figures and directionally right, but not exact. The Wilson interval is more accurate at small N and it's asymmetric — at N=20 it runs −21.6 / +11.9, not a clean ±17.5. The band also depends on the score itself: widest near 50% (N=20 → ±21.9), narrower toward the extremes (95% → ±9.5). Treat ±17 / ±8 / ±4 as a sense of scale, never as a threshold you passed.

Volume alone won't save you

Here's the part that catches people out. A bigger set only helps if the cases are hard. Four hundred easy questions all pass — the score pins at 100%, the band collapses, and you have a confident measurement of nothing. You've narrowed the error bar around the wrong quantity.

# 400 easy cases ████████████████████████████████████████ 400/400 # narrow band, zero information # the cases worth adding ███ the ones it currently gets wrong ███ the ones you argued about in review ███ the ones that broke in production

Composition matters as much as N. Mine your failures, your bug reports and your edge cases — the questions where you genuinely don't know which way it'll go. Those are the ones that move the score when the system actually changes.

The fix

Check the range, not the score. Before you call a change a win, ask whether it cleared the band. If it didn't, you have no evidence — collect more cases or accept you can't tell yet. And freeze the set: if the questions change between runs, the two scores aren't comparable at all, and no amount of arithmetic recovers that.

1 quote the band with every score # "80% ± 17" 2 a win must clear the band # else: not measurable 3 grow the set with HARD cases # failures, not filler 4 freeze it # or scores aren't comparable

Comparing two systems is a different test. The band above is the confidence interval on one score. "Must clear the band" is the conservative version. If you can run both systems on the same cases, a paired test (compare per-case outcomes — e.g. McNemar) is far more sensitive and can legitimately detect a smaller difference at the same N. And none of this covers graded or continuous scores, which need a different interval — the binomial framing is for pass/fail.

This is the sequel to evals, not vibes: that one argues for having a number at all. This one is about how much the number is allowed to tell you.

Method: textbook binomial confidence interval, computed above — no external source required. Standard references if you want one: Wilson (1927), "Probable Inference, the Law of Succession, and Statistical Inference" · Brown, Cai & DasGupta (2001), "Interval Estimation for a Binomial Proportion," Statistical Science 16(2) — on why the normal approximation is unreliable at small N.

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click