Why it confidently used an API that no longer exists

You asked for a snippet. It came back instantly and looked right — the right package imported, a function that exists, arguments in a sensible order. It ran. Three days later you found that the function's default changed two releases ago, and everything you wrote had gone to the wrong place since.

Nothing flagged it. No hedge, no "check this against your version", no difference in tone between that line and the twenty around it that were perfectly correct. That flatness is the whole problem, and it is not a bug. It is a consequence of where the answer came from.

Here is the machine, and then the habit.

Two kinds of knowing, and only one of them is in your request

Every answer a model gives you is assembled from exactly two sources.

The first is what you supplied: your message, the attached files, the system prompt, the results a tool handed back. That is text, it is present in the request, and you can go and read it.

The second is what is already in the weights — the compressed residue of a training corpus, with a date on it. That is where the model's knowledge of your language, your framework and your library lives, and it is not text anywhere. It cannot be inspected, diffed or pointed at. It is the shape the network was left in by everything that had already been written.

Almost every piece of advice about getting better answers is about the first source. This one is about the second, because when the model answers about a library you did not paste, the first source is empty and the second is doing all the work.

The two feel identical, from the inside and from your chair

The uncomfortable part is that the model does not experience these as two different things, and neither do you.

Generation produces one token after another from the same distribution regardless of provenance. Nothing marks a token as recalled from a document I saw rather than reconstructed from the shape of a thousand similar documents, so there is no mechanism by which the answer could sound less certain about the second kind — and it does not.

From your side the loss is worse. A tool result sits in the transcript and you can scroll up and read it. When the answer came from weights there is nothing to scroll to: the sentence arrives finished, in the same voice as everything else. GitHub's application card for inline suggestions names the general risk as output that "sounds plausible but is factually incorrect, unsupported by the provided context, or entirely fabricated". Sounds plausible is not a stylistic note. It is what a fluent guess is made of.

The cutoff is a gradient, not a wall

Everyone knows models have a training cutoff. Almost nobody has been told that the published date is the optimistic end of a slope.

Anthropic publishes two dates per model rather than one, and defines them apart: "Reliable knowledge cutoff indicates the date through which a model's knowledge is most extensive and reliable. Training data cutoff is the broader date range of training data used."

Read that twice. The training data runs later than the knowledge you should rely on, and on the current table that gap reaches six months. (Checked 23 August 2026 — the dates and the gap move with every release.)

The mechanism generalises to every vendor, whether they publish two dates or one. Writing about a thing accumulates for years after the thing ships. A release from three years before the cutoff has blog posts, Stack Overflow answers, tutorials and arguments behind it. A release from three weeks before it has a changelog. Both are "in the training data". Only one was learned.

So the practical cutoff is not a date after which the model knows nothing. It is a band, months wide, in which it knows thinly — enough to produce confident-sounding output, not enough for that output to be right.

"As of my last update" is a generated sentence, not a lookup

This is the first fingerprint, and it fools careful people, because it looks like the model being careful.

When a model writes "as of my last update in early 2025" it is not consulting a manifest. There is no field in the weights holding the cutoff date and no code path that reads one. The phrase is output like any other: the most plausible continuation, given that this looks like a question where a well-behaved assistant hedges. The date inside it came off the same machinery that produced the wrong function signature — so it can be wrong, and often is, off by months or naming a different model's cutoff entirely, because the corpus is full of other assistants' hedges with other dates in them.

What the hedge does tell you reliably is that the answer came from weights rather than from a search. Unreliable timestamp, reliable provenance marker — and the provenance is the useful half.

Fingerprint two: an answer that is right, for a version you never named

If you did not state a version, the model answered about the version its corpus is heaviest in — the older, more written-about one, not the one in your lockfile.

GitHub says it directly about code suggestions: "For each language, the quality of suggestions depends on the volume and diversity of training data for that language." Substitute version for language and the sentence still holds, which is why a model is weakest exactly where a library changed most. The v2 rewrite has one migration guide behind it. The v1 API it replaced has five years of the internet.

The fingerprint: the code is idiomatic, well-structured and coherent — coherent with itself, following the conventions of a version you stopped using. Staleness does not look like confusion. It looks like a competent developer who has been offline for eighteen months.

Fingerprint three: the quiet change, not the missing function

Here is the case that costs you a day, and it is not the one people brace for.

A removed function is the harmless failure. You get an AttributeError, look it up, fix it in four minutes. The expensive failure is a function still there, still named the same, still accepting your arguments — and behaving differently.

That is not an edge case, it is the documented norm. Anthropic's API versioning policy promises that "Generally, if you are using the API as documented in this reference, Anthropic will not break your usage" — and then reserves, within one unchanged version, the right to add optional inputs, add values to the output, change the conditions under which specific errors are raised, and add new variants to enum-like output values. Every one of those is a change a model trained before it gets wrong while producing code that runs.

The fingerprint: no exception, no warning, a green test suite, a result that is subtly wrong in production. New default, wider return shape, an error arriving as a different type. You find it by reading the changelog and by nothing else.

Fingerprint four: a search tool being available is not a search having run

The last one is a category error about your own setup. You enabled web search, or connected a docs server, and concluded the model is now current. It is not: whether the tool runs is decided per request.

All three major vendors document this in the same terms. Anthropic: "Claude determines when to search based on the prompt" — and the same page lists what makes Claude answer directly instead, namely requests drawing on "stable knowledge", coding concepts included. Google describes the identical step: "The model analyzes the prompt and determines if a Google Search can improve the answer." OpenAI notes that with tool_choice left on auto, searching is optional.

Now look at where that leaves you. "Which arguments does this function take?" reads exactly like stable coding knowledge, so it is the class of question most likely to be answered from weights — and the class most likely to have changed. The tool is switched on, the answer is stale, and nothing in the interface distinguishes the run that searched from the run that did not.

How to tell whether a search actually happened

Stop inferring; go and look. Every vendor emits an artefact when a search runs, and its absence is your answer.

In a chat interface it is the citation chips and the "searched for…" line — no chips, no search. Through an API it is structural: a web_search_call item in the output, url_citation annotations on the message, a google_search_call step in the response. OpenAI also exposes a sources field listing every URL consulted rather than only the ones cited — the difference between what the model looked at and what it showed you.

And if you need a search rather than hoping for one, OpenAI is explicit about the control: with tool_choice on auto search is optional, and setting it to required — or naming the web search tool specifically — is what makes it run. In a chat window the equivalent is a sentence, not a setting: search the current docs before answering, and quote the page.

A search that did run is a different problem, because the retrieval chose what reached the model; that half is a separate article. This one is about the case where nothing ran at all.

Name the version in the prompt, every single time

The habit that fixes most of this costs eight words at the top of the message.

State the version. "React 19", "Pydantic v2", "the Responses API, not Chat Completions". You are not being pedantic — you are moving the fact from the second source to the first, where it can outweigh a corpus that disagrees.

Then go further where it matters. Paste the changelog entry or the current signature into the prompt for the one call you care about; a hundred tokens of actual docs beats any amount of remembered docs. And when an answer surprises you, ask directly: which version did you assume, and what in my prompt told you that? An answer of "the latest" means it assumed, because nothing in the request said so.

If you are hitting this daily on one project, stop retyping it. Claude Code's guidance on project memory is to "Treat CLAUDE.md as the place you write down what you'd otherwise re-explain", and a pinned block of framework versions is exactly that — which layer it belongs in is a decision in its own right.

When you cannot search, fence the model in instead

Sometimes no tool is available, or the docs are internal. The move is the same one: make the supplied source outrank the parametric one, explicitly.

Anthropic's guidance on reducing hallucinations names it as a technique — "External knowledge restriction: Explicitly instruct Claude to only use information from provided documents and not its general knowledge." Paste the reference, then say answer only from this; if it is not here, say so. The same page recommends giving the model explicit permission to admit uncertainty, which matters here precisely because the default behaviour is a fluent guess rather than a refusal.

There is a cheap detector too. Asking the same question several times and comparing outputs is standard hallucination triage, and it catches staleness for the same reason: recalled facts are stable across samples, reconstructed ones drift. Three runs, three signatures, none of them memory — why the same prompt gives you a different answer is the mechanism underneath that test.

What no amount of thinking will fix

One thing worth ruling out before you spend on it: turning the reasoning dial up does not make a model's knowledge newer.

Extra deliberation improves how well the model uses what it has. It cannot introduce a fact that was never in the corpus, and applied to a stale premise it produces a longer, better-argued, more convincing version of the same wrong answer. That distinction — a missing fact versus a mishandled step — is the whole of when reasoning is worth paying for. Staleness is an input problem.

The rule, applied to something this article never mentioned

Next time an answer about anything versioned looks confident, work down this list:

1. Did I name the version? If not, assume it answered about the most-written-about one, which is rarely yours.
2. Is there evidence a search ran? Citations, a search-call item, a sources list. No artefact, no search — regardless of what is enabled.
3. Does this sound stable but move? Signatures, defaults, config keys and flags all read as settled knowledge and all change. That combination is where the tool declines to search and the answer is wrong.
4. Would this fail loudly or quietly? An unknown name errors and costs minutes. A same-name behaviour change ships. Spend the checking time on the second.
5. Did it hedge with a date? Take the provenance seriously and the date not at all.
6. Can I move the fact into the request? Paste the signature, the changelog, the schema. Supplied outranks remembered.
7. Does it drift across runs? Ask three times. Stability is weak evidence of memory; drift is strong evidence of invention.

None of this is specific to code. A model's knowledge of a pricing page, a regulation, a rate limit or a job title has the same shape and the same flat confidence, and each changes faster than a corpus can be rebuilt. The question to carry is not "does this model know?" but "was this in the request, or is it being remembered?" — because only one of the two can be checked.