Start with something you already know
Type "Good morning" into your phone and it offers "everyone". It does not understand what you are writing. It only knows that in the messages it has seen from you, "Good morning" is often followed by that word.
A language model does exactly the same thing. The difference lies in three things, all of them differences of degree — and yet together they produce a difference in kind.
1. It learned from far more text
Your phone learns from your messages. The model learned from a substantial slice of the internet: books, documentation, forums, code repositories. Not because more data is better in itself, but because it sees enough patterns to have a guess even for rare cases. If someone asks for the first time in history how to recalibrate a 1978 sewing machine, the model does not remember that sentence — but it has seen enough sewing-machine manuals and enough calibration instructions to put the two together.
2. It counts in tokens, not words
The model does not see words, it sees tokens — fragments of words. "Unbelievable" splits into three tokens; a long German or Hungarian compound splits into many more. This is not merely a technical detail: it determines what a conversation costs and how much fits into one. Tokens and the Context Window covers it.
3. It sees the whole text at once
This is the important one, and in 2017 it was the breakthrough. Older systems crawled through a sentence from left to right, and by the time they reached the end they had forgotten the beginning.
The transformer architecture solved that: inside it, every word can look at every other word and decide which ones matter to it. This is called attention.
An example: "I put the pan on the stove because it was cold." Was the pan cold, or the stove? To fill that in correctly the model has to look back at "pan", five words earlier, and work out what the coldness refers to. Older systems could not. Today's can — and they hold an instruction given at the start of a text in view in exactly the same way twenty paragraphs later.
How this turns into an answer
When you ask a question, the model does not "look up" the answer. It lays down tokens one at a time, each based on the text so far. At every step it computes how likely each possible next token is, picks one, appends it, and starts again.
So the answer does not exist first and get printed afterwards. It comes into being as it is written. That is why asking the same question twice gives two different answers — the pick does not always land on the most likely token. How far it wanders is governed by a setting, and that is what Prompt, System Prompt, Temperature is about.
What follows from this, and surprises most people
The model does not know in advance where it will end up. Ask it how many words the answer will be and it cannot tell you, because it has not written it yet. That is why it is bad at counting words and why it cannot reliably answer "in exactly three paragraphs" on the first attempt.
It does not take back what it has already written. If the first sentence set off in the wrong direction, the rest gets fitted to it, because every subsequent token is chosen on the basis of the text so far. That is why correcting a bad answer works better if you say what is wrong with it than if you simply ask again.
And that is why making it think step by step helps. If you ask for the working as well, the model can lean on its own earlier sentences when it writes the conclusion. It did not get smarter — it just laid down more footholds for itself.
And that is why it hallucinates. With no good guess available it still puts down the most likely token rather than saying "I don't know". Why AI Makes Things Up goes into this properly.
What to take away
A language model is not a knowledge base you look things up in. It is a text-continuation machine that guesses very well. Every strength and every weakness it has follows from that one property — and if you keep it in mind, nothing it does or fails to do will ever surprise you.