toolcall() ← all concepts

// concept · context rot

Your prompt's middle gets ignored.

You buried a rule in paragraph nine and the agent dropped it. That's not a bug in your prompt — it's how attention falls off across a long context.

// the assumption

We hand the model a long prompt or a long pasted document and assume it reads the whole thing with even attention — top, middle, bottom, all the same. So it shouldn't matter where the key instruction sits.

# what we picture: uniform attention [ start ████ middle ████ end ████ ] # every line weighted the same

The reality is a U

Recall isn't flat. It follows a U shape: strong at the very start (primacy) and the very end (recency), and weakest right in the middle. The paper that named it — "Lost in the Middle" — puts it plainly: performance is highest when the relevant info is at the beginning or end, and "significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."

# recall vs. position (the U) high │█ █ │ █ █ │ █ █ │ █ █ low │ █ █ █ └────────────────────────────── start middle end

The evidence

In that study, moving the single relevant document from the ends to the middle dropped multi-document QA accuracy by more than twenty points (roughly mid-seventies at the ends down to mid-fifties in the middle). In the worst cases — long stuffed contexts — the model scored worse than getting no documents at all (its closed-book baseline was 56.1%). More context actively backfired.

Read this as an illustration, not a law. Those exact numbers are GPT-3.5-Turbo in one study (Liu et al.). The U shape is a strong general pattern; the precise size of the dip varies by model. Don't quote "20 points" as a universal constant.

Why it happens

Two forces stack. Position bias pulls attention toward the ends, and sheer length wears the model down — the longer the input, the more the middle gets lost. Chroma's context rot study tested eighteen current models and found performance "varies significantly as input length changes, even on simple tasks," with all of them degrading as input grew — and they saw "significantly higher performance on focused prompts compared to full prompts."

# two forces, one rotten middle position bias -> attention favors the ends input length -> recall decays as tokens pile up net: the bigger the pile, the deeper the burial

The fix

You don't need a clever prompt trick. Put the most important instructions or facts first or last, and keep the context lean. A bigger pile of context isn't a smarter agent — it's a deeper hole to bury your point in. Don't put the lede in the middle.

first # the rule that must not be dropped ... # supporting context, trimmed last # the instruction / the question, restated
This is not the token limit. Context rot happens below the model's max window — it's about attention, not running out of room. The hard cap is a separate thing (see context window limits). And it varies by model: newer long-context models are better, but none fully solve it — so place things on purpose.

Sources: Liu et al. — Lost in the Middle (arXiv 2307.03172) · TACL vol. 12 (2024) · Chroma — Context Rot: How Increasing Input Tokens Impacts LLM Performance

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click