Tokens, context, and temperature
This toy model turns a short support prompt into visible token chunks, then shows how the next-token distribution changes when the prompt carries policy context. Temperature only changes how strongly the sampler follows the distribution it already has.
Token chunks
38 chunksNext token
sampler view- refund43%
Base: 38%. Name the likely outcome directly.
- approve37%
Base: 34%. Start an answer that approves the request.
- ask16%
Base: 19%. Ask for missing facts before deciding.
- explain2%
Base: 5%. Explain the relevant policy.
- escalate1%
Base: 2%. Send the decision to a manager.
- deny1%
Base: 2%. Start an answer that refuses the request.
Better context changes the model's belief. Lower temperature only changes the sampler.
In this prompt, policy context moves the probability mass toward the right business action by 5%. Setting temperature to zero just locks in the current top token, whether the prompt is under-specified or well-grounded.
The sampler is confident because it has to choose one token, not because the prompt contains enough evidence.
The top token changes because the prompt now says what decision rule applies.