What is actually happening
The model does not know the answer; it predicts what probably comes next. Given enough to go on, the most likely continuation happens to be true. Given nothing to go on, it still puts down the most likely continuation — because it has no other mode of operation.
So hallucination is not the system breaking. It is the system doing exactly what it was built to do, in a situation where that produces a wrong result. How a Machine Guesses the Next Word covers the mechanism in more detail.
This is why it cannot be switched off. The same capability that lets it solve problems nobody trained it for is responsible for confidently inventing a regulation that does not exist. The two cannot be separated.
Why does it sound so confident?
Because confidence belongs to the form of the text, not to the knowledge behind it. The model learned what a competent answer looks like — and a competent answer does not begin "I vaguely recall that". The tone comes from the texts it learned on, not from how sure it is.
There is a more uncomfortable consequence: the model's internal uncertainty does not show up in the writing. An accurate sentence and an invented one sound equally assured. You cannot tell them apart by looking — which is why "I'll notice if it says something silly" is not a strategy.
When is it most likely?
It does not arrive at random. Five situations raise the odds sharply:
- Specific numbers, dates, names, references. This is the most dangerous category: precisely the kind of detail we do not check, because it sounds so definite.
- Rare or narrow subject areas. Where there was little training material there is less to go on, and a higher proportion of guesswork.
- Recent events. The model's training data stops at a given date. After that it only knows something if it searches, or if you supply it.
- When you suggest the answer yourself. "It's true the company was founded in 2019, isn't it?" — the model tends to agree. The shape of the question shapes the answer.
- Towards the end of a long conversation. Once the beginning has slipped out of the context window, the model builds on its own earlier claims without being able to see them precisely any more.
Five things that genuinely reduce it
These are not tricks, they are working practices.
1. Give it the material. There is an enormous difference between "what does the contract say about the notice period" and "here is the contract, what does it say about the notice period". In the first it works from memory; in the second it reads.
2. Ask for a reference into the material you supplied. Not a general citation — it can invent those too — but which paragraph it is basing the claim on. If it cannot point at one, that is an answer in itself.
3. Allow it not to know. Write it into the instruction: "if the material does not contain the answer, say that it is not in there." Without that, the model has no acceptable output for not knowing, so it guesses.
4. Check back in a separate conversation. Ask "are you sure?" in the same thread and the model is looking at its own earlier claim, and inclined to defend it. In a fresh conversation, with the original material, it is far more honest.
5. Verify what matters. Not everything — the numbers, names, dates, regulations and references. That list is short, and it is exactly the risky part.
What does not help
Telling it "don't hallucinate". The model does not know when it is doing it. If it knew, it would not.
Asking it to "be very accurate". At best this changes the tone, not the facts.
Asking whether it is sure. You are running the same mechanism a second time. "I am completely certain" is worth precisely as much as the original claim.
An example worth remembering
Ask a model which page of a well-known textbook in your field a particular concept appears on. You will often get a page number. A round, plausible page number.
Now ask the same thing after giving it the book's table of contents. The answer will either be accurate, or it will be that the contents do not show it.
Same model, same question, two entirely different levels of reliability. The difference is not in the model but in whether you gave it something to stand on.
What to take away
Hallucination is not a disease of the system, it is an operating characteristic. You do not cure it — you design your work around it: supply material, ask for references, allow it to say no, and check the handful of things that actually matter.