What temperature actually changes

// what it actually does

At each step the model emits logits — raw scores over the whole vocabulary. Temperature T divides every logit by T before the softmax turns them into probabilities. That's the entire mechanism. It adds no facts and no creativity; it only changes how peaky or flat the next-token distribution is.

# temperature scales logits before softmax p_i = exp(z_i / T) / Σ_j exp(z_j / T) T < 1 sharpens # gaps exaggerated → top token dominates T > 1 flattens # gaps shrink → mass spreads across tokens T = 0 ≈ greedy # collapses to argmax (the single top token)

Peaky vs flat

Same prompt, same logits, two temperatures. Watch the next-token distribution for "The capital of France is ___":

# T = 0 (sharpened → greedy) Paris ████████████████████ 0.99 Lyon ▏ 0.004 Nice ▏ 0.003 # picks Paris every time # T = 1.5 (flattened) Paris ████████ 0.41 Lyon ████ 0.19 Nice ███ 0.14 # junk tokens get real odds

That's why low temperature wins for facts, code, extraction, and classification — you want the single most-likely, repeatable token at every step. Crank it up and you sample worse answers. If your model returns JSON but sometimes drifts off-schema, you probably left temperature high.

The rule + top_p

Low for anything that has a right answer; higher for ideation, brainstorming, and varied copy. And tune temperature or top_p — not both. They're different levers: temperature reshapes the whole distribution, top_p (nucleus sampling) truncates its tail, keeping the smallest set of top tokens whose probability sums to top_p. Move both and the effect gets impossible to reason about. OpenAI says this outright; Anthropic flags top_p as advanced-use-only.

# pick a lane extraction / code / facts → temperature low (≈0) ideation / brainstorming → temperature higher tune temperature OR top_p — never both at once

The honest caveats. Low temperature gives you the most likely token — not a true one, so it does NOT fix hallucination. Vendor ranges differ: OpenAI is 0–2 (default 1), Anthropic is 0–1 (default 1) — a "temperature of 1.5" only makes sense on OpenAI. And temperature 0 is not perfectly deterministic in production (batching, hardware, floating point); Anthropic says so directly. For reproducibility you need a seed, not temperature alone.

Sources: Anthropic — Messages API (temperature, top_p, determinism) · OpenAI dev community — temperature & top_p cheat sheet · MachineLearningMastery — logits, softmax, sampling · Softmax temperature explained

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click