toolcall() ← all concepts

// concept · hallucination

Why AI confidently makes things up.

It doesn't lie to you. It was trained to never say "I don't know" — because on nearly every benchmark, a confident guess scores higher than a blank.

// the mechanism

An LLM generates by predicting the next likely token, not by checking a fact store. It's producing what sounds right. For anything with no learnable pattern — an arbitrary date, a specific citation — a confident, wrong answer is statistically almost unavoidable.

"the capital of France is ___" -> Paris # easy, patterned "the title of X's 2019 paper is ___" -> a plausible-looking invention

The exam analogy

Picture a multiple-choice exam. A blank scores zero. A wrong guess scores zero. But a lucky guess scores one. The rational move is to always guess, never leave it blank. We trained the model the exact same way.

blank -> 0 wrong guess -> 0 lucky guess -> +1 # so: always guess

It's baked into training

OpenAI's 2025 analysis puts it plainly: hallucinations persist because models are "optimized to be good test-takers, and guessing when uncertain improves test performance." Most benchmarks grade right-or-wrong with no credit for admitting uncertainty — so saying "I'm not sure" scores the same as a lie. Honesty is penalized.

It's not "lying." There's no intent — it's a statistical artifact of the objective, not deception. The framing is about incentives (binary grading), which is more general than any one "next-token" story.

What reduces it

You can lower the rate — you can't delete it:

· ground the answer in real sources (RAG) · make it cite them · let it say "not sure" / abstain · lower temperature for factual tasks

But it never fully goes away. Grounding helps a lot, yet retrieved context can still be misread or ignored — it's baked into predicting the next word. So trust, but verify: treat a confident answer as a claim, not a fact.

Sources: OpenAI — Why language models hallucinate (2025) · Kalai et al. (arXiv 2509.04664) · grounding with RAG

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click