Why the same prompt gives you a different answer

You ran the prompt yesterday and it produced exactly what you wanted. You ran it again this morning — same words, same model in the dropdown — and got something noticeably worse. The reasonable conclusion is that you changed something and cannot remember what.

Usually you did not. The belief underneath that conclusion is that a language model is a function: same input, same output, so a different output must mean a different input. Half of that is right. The wrong half is the assumption that "the same prompt" and "the same input" are the same thing.

Every vendor documents every feature of these products in exhaustive detail. Almost none documents why identical text produces different answers on Tuesday and Wednesday. What follows is not a checklist of causes. It is the machine, in the order the parts matter.

The answer is sampled, not retrieved

Start at the smallest scale, because everything else follows from it.

A model does not look up a response. For each next token it produces a ranking of the possible options with probabilities attached, and then one of them is picked. That picking is a real step with real settings. Anthropic documents top_k as "Only sample from the top K options for each subsequent token", and top_p as nucleus sampling: "In nucleus sampling, we compute the cumulative distribution over all the options for each subsequent token in decreasing probability order and cut it off once it reaches a particular probability specified by top_p."

The word to hold onto is cumulative. The choice happens once per token, and each choice conditions every choice after it. One different token near the start of an answer does not produce a slightly different answer — it produces a different answer, because the model is now continuing from a different sentence. That is why the variation you see is rarely a synonym swap and is often a completely different structure.

OpenAI states the consequence for the whole system in one line: "Because the content generated from a model is non-deterministic, prompting to get your desired output is a mix of art and science."

So variation is not a fault condition. It is the normal operating mode, and the useful question is never "why did it vary" but "why did it vary that much".

Temperature narrows the spread; it does not remove it

The obvious next move is to turn the randomness off, and the obvious next disappointment is that you cannot.

Temperature is documented as the "Amount of randomness injected into the response", and lowering it does what you would hope — it concentrates the choice on the highest-probability options, which is why the guidance is to use it "closer to 0.0 for analytical / multiple choice, and closer to 1.0 for creative and generative tasks". What it does not do is make the system a lookup table. The API reference says so directly: "Note that even with temperature of 0.0, the results will not be fully deterministic."

That line is the most useful sentence in this article, because of what it forbids: the debugging move where you set temperature to zero, run the prompt twice, get two different answers, and conclude something is broken. Nothing is broken. You have hit a documented limit.

Low temperature is still worth using — it turns an unreliable prompt into a mostly-reliable one. It never turns a prompt into a guarantee, and any workflow that needs one has to get it elsewhere: a schema, a validator, a second pass that checks the first.

"The same prompt" is almost never the same input

Sampling is the smaller half of the story. The model is not sent your sentence; it is sent a request, and the request is an array. OpenAI's documentation is blunt about what that implies: "While each text generation request is independent and stateless, you can still implement multi-turn conversations by providing additional messages as parameters to your text generation request."

Read that as a description of the chat window you are typing into. The window has no memory of its own. Every time you press enter, the application rebuilds a request containing the whole conversation and sends it fresh. What feels like an ongoing relationship is a sequence of unrelated requests that each happen to carry a transcript of the ones before.

So the input to your "same prompt" includes everything above it in the thread. Two identical sentences typed into two different threads are two different inputs.

Every turn you take changes the input for the next one

The accumulation is the mechanism. Anthropic describes it plainly: "As the conversation advances through turns, each user message and assistant response accumulates within the context window, and previous turns are preserved completely."

Preserved completely — including the model's own earlier answers. That is the part people miss. A slightly off interpretation in message three is part of the input for messages four through twenty, read as established fact rather than as a guess. Threads do not just get longer; they get more committed.

That retires two mysteries. "It was great at the start of the session and got worse" is not fatigue; it is a growing input that increasingly contains its own output. And "it worked in a new chat" is not a fix; it is a different experiment.

The same mechanism is also what a long conversation costs you, since every one of those preserved turns is re-sent and re-charged on the next one — that side of it is here.

There is text in the request that you did not write

This is the part that makes "I changed nothing" wrong even when it is sincere.

A consumer chat application does not send your message alone; it prepends a system prompt of its own. Anthropic publishes theirs, and publishes this alongside it: "The system prompt also encourages certain behaviors, such as always providing code snippets in Markdown. This prompt is periodically updated to improve Claude's responses."

Sit with periodically updated. Part of your input is written by the vendor, changes on a schedule you are not consulted about, and is not announced in your interface. A prompt that has been stable for six weeks can produce a different style of answer on a morning when you did nothing at all, because the paragraph in front of your paragraph is not the one that was there last week.

The same page adds the corollary: "These system prompt updates do not apply to the Claude API." So the chat app and the API are not two doors into one room. They are different inputs, and a prompt tuned in one behaves differently in the other — not because of a hidden quality setting, but because the text being sent is not the same text.

Memory makes a "fresh chat" a false control

The standard way to isolate a problem is to open a new conversation and run the prompt clean. Memory features break that test.

Anthropic's memory tool "lets Claude store and retrieve information across conversations in a directory of memory files", with files "that persist between sessions, building up knowledge over time without keeping everything in the context window".

The consequence for debugging is specific: a new thread is no longer an empty input. It can carry forward preferences and facts inferred from work you did days ago, and they arrive invisibly — nothing in the transcript shows them. When a prompt behaves differently for you than for a colleague running identical text, the difference may be a stored fact about you that neither of you can read. A genuine control needs memory off, or an account that has none.

None of which is an argument against starting fresh chats. For everything other than isolating a variable, a new conversation is the single most useful habit you have — it is what stops a thread's accumulated history from being re-sent, and re-charged, on every turn (what actually uses up your Claude usage limits). The two purposes just want different things from the same action: spending less is served by a new chat, and proving something needs one with nothing carried into it.

A tool call makes the answer depend on the world

If the model can search, run code or read a file, the input has grown a part that changes without anybody editing anything.

Web search exists precisely to do this: it "gives Claude direct access to real-time web content, allowing it to answer questions with up-to-date information beyond its knowledge cutoff". Which means the request sent this morning contains pages that did not exist or did not rank yesterday.

Two things follow. Identical prompts over changing sources produce changing answers, correctly. And a difference in output may not be a difference in the model's reasoning at all — it may be a difference in what came back. Before you rewrite a prompt that searched, look at what it retrieved. You may be editing the wrong half.

The thing behind the name is not a constant either

Last layer — the one people assume is solid ground.

Model identifiers on the Claude API are pinned snapshots, and for a pinned ID the weights do not move. But weights are not the whole system. Anthropic is candid: "Occasionally, infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed." The docs name what sits around the model — the request router, safety classifiers, sampling logic — and note that those change over time.

In a chat product the ground is softer still: you have a product name, not a pinned identifier, and the mapping between them is the vendor's to change.

So "nothing changed on my side" can be entirely true while the answer moves. That is not a reason for paranoia. It is a reason to pin what you can — an explicit model ID rather than a friendly alias — so one variable is nailed down when you go looking for the others.

What this means the next time you debug a prompt

The habit worth building is to stop reasoning from single runs.

One bad output is not evidence that a prompt is bad, and one good output is not evidence that a fix worked. Both are single samples from a distribution you cannot see. If you change a prompt and it works once, you have learned almost nothing; if it works five times out of five where the old one worked twice, you have learned something real.

That also tells you when to stop editing. A prompt that succeeds nine times in ten is not missing a magic word. Its failure mode needs catching downstream, because no amount of rewording converts a distribution into a certainty.

So what do you actually do

In the order that removes the most variance for the least effort.

1. Test in a fresh thread, deliberately. Not because new threads are better, but because a thread you have been working in is a different input and a dishonest control.
2. Pin the model where you can. An explicit version identifier removes one moving part; a product name does not.
3. Put context in the prompt, not in the history. Anything the model must know belongs in a message you can re-send verbatim, not four turns up where you cannot reproduce it.
4. Constrain the shape of the output. A fixed structure — fields, a schema, a format — narrows what sampling can do to you far better than asking politely for consistency.
5. Judge changes over several runs. Three to five is usually enough to tell a real improvement from a lucky sample.
6. Validate instead of trusting. For anything automated, check the output's shape and retry on failure. It is the only item here that turns a probability into a guarantee.

None of that makes the system deterministic, and nothing available to you does. What it does is shrink the range of things that can vary until the rest is small enough to design around.