Temperature: A Dice-Rolling Adventure

Have you ever sent the exact same prompt to an AI coding tool twice and got two different answers? Or wondered why a model sometimes writes a boring, safe function and sometimes a creative (and slightly wrong) one? The answer is not magic, it is a setting called temperature, and a process called sampling. In this post, we will understand how a model actually picks its next word, why that makes answers change between runs, and which settings to use for which task.

How does a model pick the next word?

Let's start with an example before any definitions. Imagine I say to you:

"The puppy was very ___"

You have a few good options in your head: happy, playful, tired, hungry. Some of them feel more likely than others. You would probably not say purple. An LLM does something very similar, but with numbers. For every position in the text, the model looks at everything before it and gives every token in its vocabulary a score. Then those scores are turned into probabilities that add up to 100%.

So for our sentence, the model might end up with something like:

📝Markdown
Copy
1
2
3
4
5
6
happy    45%
playful  25%
tired    15%
hungry   10%
purple    0.01%
...

This happens again and again, one token at a time, until the model decides to stop.

What is sampling?

Sampling is the rule that picks one token from the menu. There are two simple ways to do it:

• Greedy: always take the most likely token. In our example, always happy. Same prompt, same output, every time.
• Random sampling: roll a dice that is weighted by the probabilities. happy wins 45% of the time, playful 25% of the time, and very rarely you get something strange.

You might wonder, why not always use greedy? Great question! Greedy sounds safe, but it has a known problem: it tends to repeat itself, because it never takes a small risk that would lead to a better sentence later. Pure random sampling has the opposite problem, it can wander off into nonsense. We want something in between, and that is where temperature comes in.

What is temperature?

Temperature is a single number that reshapes the menu before the dice is rolled. It does not change what the model knows, only how confident the dice is.

• Low temperature (close to 0): the probabilities get sharper. The top option becomes even more dominant, the rare options almost disappear. At temperature 0 the sampling becomes the same as greedy: pick the top token.
• High temperature (above 1): the probabilities get flatter. The gap between happy and tired shrinks, so the dice lands on less likely tokens more often.

Here is the same menu at two temperatures, roughly:

📝Markdown
Copy
1
2
3
4
5
6
Temperature 0.2         Temperature 1.5
happy    85%            happy    30%
playful  10%            playful  25%
tired     4%            tired    20%
hungry    1%            hungry   15%
purple    0.0%          purple    2%

What about top-p and top-k?

Next to temperature, you will often see two more settings. They do not reshape the menu, they cut it.

• Top-k: keep only the k most likely tokens and throw away the rest. With k = 3, our menu is only happy, playful, tired. Then the dice is rolled among those three.
• Top-p (also called nucleus sampling): keep the smallest group of tokens whose probabilities add up to p. With p = 0.9, the model keeps adding tokens from the top until it reaches 90%, then stops. When the model is very sure, this group is tiny. When it is unsure, the group grows. That is why top-p is considered more flexible than top-k.

In most setups, temperature reshapes the probabilities first, and then top-k or top-p trims the candidates. The important thing to remember: these knobs only control the picking step. They are applied after the model already did its thinking.

So why did I get two different answers?

Now it should be clear. If the temperature is above zero, then every time the model faces a close call between two tokens, the dice can land differently, and from that point the whole answer takes a different path. It is not that the model "changed its mind". The first fork went a different way, and everything after it followed.

What should I actually set?

Here is the practical takeaway. Match the temperature to the job:

• Temperature 0 or very low: data extraction, classification, converting formats (JSON, CSV), code refactoring, answering from a document, anything where there is one correct answer and you want it every time. This is the right choice when a model runs inside an automated pipeline.
• Medium (around the default): everyday coding help, writing explanations, brainstorming a few options for a design.
• High: naming things, marketing copy, generating many diverse ideas to pick from later. Expect to filter the results.

Example: if you build a small tool that reads a user story and returns a JSON list of tasks, set temperature to 0. You want stability, and you want the output to parse every time. If you ask the model "give me 10 different ways to name this feature", raise it and enjoy the variety.

Conclusion

An LLM does not "decide" its answer in one shot. It builds a menu of next tokens with probabilities, and sampling picks from that menu, one token at a time. Temperature sharpens or flattens the menu, top-k and top-p trim it. That is the whole reason the same prompt can give you different answers, and it is also a knob you control. Next time we will look at a related question that this post leads to: why models sometimes state things that are simply not true, and what hallucination really is.