An LLM is not looking anything up

You type a question. A moment later an answer appears. The box looks like a search box, the answer arrives like a search result, and so almost everyone builds the same intuition: somewhere behind this, the thing is looking something up.

It isn’t. There is no database, no index, and no lookup step. And once you replace that intuition, most of the surprising things about these systems — hallucination, confident wrongness, knowledge cutoffs, why obscure questions go badly — stop being separate mysteries and become one consequence.

What actually happens

The model is a large set of fixed numbers — the weights. Your text is converted to numbers, those numbers flow through the weights in a single pass of arithmetic, and out comes a probability distribution over what token should come next. One is picked. The whole thing repeats for the next token.

That is the entire mechanism. No part of it consults a store of facts, because there is no store of facts. Nothing is retrieved because there is nothing to retrieve from.

Where did the knowledge go, then? Into the weights. Training adjusted billions of numbers so that plausible continuations of text became likely, and facts that recurred throughout the training corpus ended up encoded in those adjustments — distributed across the weights rather than filed anywhere. There is no row for the capital of France. There is a configuration that makes “Paris” the overwhelming continuation of “the capital of France is”.

It is closer to something a person knows than to something a person can look up — though even that comparison has a hard boundary, and it is worth naming: people usually have some sense of whether they actually know a thing. That signal is exactly what is missing here.

Everything else follows

Hallucination is structural. The model produces a plausible continuation. When it has strongly encoded a fact, the plausible continuation is the true one. When it has not — an obscure detail, a specific citation, a version number, a person who is not famous — it still produces a plausible continuation, because that is the only thing it does. A fabricated reference and a real one are the same kind of object from the inside: both are text that fits.

This is why “stop making things up” does not work as an instruction. There is no separate making-things-up mode to switch off.

There is no not-found. A search engine can return zero results. That is a real state it can be in. A model has no equivalent — every prompt produces a continuation, and there is no point in the process where a lookup fails and something else happens instead. Expressions of uncertainty are themselves generated text, produced because uncertain-sounding text was the plausible continuation, not because a confidence value was consulted.

Knowledge has a cutoff and cannot be patched. The weights were set during training and do not change while you talk to it. Nothing that happened afterwards is in there. And you cannot correct a fact the way you would edit a row, because the fact is not in a row.

The rare things are the least reliable. Something mentioned a million times in training is deeply encoded. Something mentioned twice is barely there — but it is still something, so the model will still produce a fluent answer. This is precisely backwards from what you want: reliability is lowest exactly where you are most likely to be asking, because you are asking about the obscure thing.

Citations are the worst case of all. A specific author, year, title and page is a long, exact string that appeared rarely. It is maximally fabricable and minimally checkable at a glance, which is why fake references keep turning up in places where somebody should have looked.

What retrieval actually adds

This is what RAG is for, and the framing makes it obvious rather than clever: it bolts on the lookup step that was never there.

Fetch relevant documents first, put them in the prompt, then generate. The model is no longer producing an answer from its weights; it is summarising text placed directly in front of it. Web search in a chat interface is the same idea with a different retriever.

Note what this changes. The failure mode moves — the model can still misread what it retrieved, or lean on its weights when the documents are thin — but it gains something it structurally could not have before: a real empty result. If nothing was retrieved, there is genuinely nothing there, and the system can be built to say so.

The practical reading

Treat unverified model output as recall from somebody who has read an enormous amount and remembers imperfectly, cannot tell which parts they are sure of, and will never say they don’t know.

That reading gets the risk profile right. It predicts that common knowledge will be reliable, that specifics will be shakier than they sound, and that the tone will be identical either way — because the tone is generated too.

If you want to see it directly, ask about something that does not exist. Invent a plausible library and ask for its API, or ask for the main argument of a paper you have made up. A search engine returns nothing. You will usually get a confident, well-structured, entirely fictional answer — and that is not the system malfunctioning. That is the clearest possible demonstration of what it was doing all along.