Temperature and Sampling: Why the Same Prompt Gives Different Answers

Temperature controls how much randomness a model uses when picking its next word - low for precise, high for creative.

The basic idea

When a model generates text, it doesn't just pick the single most likely next word every time - it picks from a range of likely candidates, weighted by probability. Temperature controls how sharply that weighting favors the top candidates. At temperature 0, the model almost always picks the single most probable word, so answers become very consistent, close to deterministic. At a higher temperature, less likely words get a real chance too, making output more varied and sometimes more creative, but also more prone to going off track.

Top_p: a different knob for the same job

Top_p (nucleus sampling) works differently. Instead of flattening the whole probability curve, it narrows the pool of candidate words to the smallest set whose combined probability crosses a threshold - say, the top 90%. When the model is confident about the next word, that pool stays small. When it's uncertain, the pool grows. Most APIs recommend tuning only one of temperature or top_p at a time, not both together.

Why coding tools pick low temperature

Code has to be exactly right - a variable name, a bracket, a function signature can't be creatively wrong. AI coding tools default to low temperature, at or near 0, so the same prompt reliably produces similar, predictable code, instead of stylistic risk-taking on a task where consistency matters far more than variety.

EXAMPLE

Set temperature to 0 when asking for code: 'Write a function that reverses a string.' Run it five times - at temperature 0 you'll get nearly identical code every time; at temperature 1 you'd likely see several different valid approaches.

🛠️ EXERCISE — TRY IT YOURSELF

Run the same prompt at low and high temperature and compare the results.

Pick an open-ended prompt, e.g. 'suggest a name for a small bakery.'
Run it several times at temperature 0 (or the lowest your tool allows) and note how similar the answers are.
Run it several times at temperature close to 1 and note how much they vary.
Now try a coding prompt (e.g. 'write a function that adds two numbers') at both settings and compare how much the variation actually matters there.

✅ SELF-CHECK

☐ Could you observe a visible difference between low and high temperature?
☐ Do you understand why top-p and temperature usually shouldn't be changed at the same time?
☐ Can you explain why coding agents typically use low temperature?

QUICK QUIZ

Why do AI coding tools like Claude Code typically use a low temperature by default?