An LLM does exactly one thing: it predicts the most likely next piece of the text, and then repeats that over and over. The autocorrect on your phone does the same, only far more stupidly and with a far shorter memory. The difference is not in the principle, it is in the size and in how much material it was trained on.
Everything important about it follows from that mechanism. It can summarise, translate and rephrase because those are all "continuation" jobs. It hallucinates because the likely continuation is not always the true one. And it is bad at mental arithmetic because arithmetic is not a statistical continuation.
The most common misconception is that an LLM can be queried like a database. It cannot: it does not store its training material in any searchable form, and there is no true-or-false switch inside it. If you need dependable facts, you have to hand it the facts — that is RAG.
The mechanism, in plain language, is the subject of How a Machine Guesses the Next Word, a post of its own. Directly related terms: token, transformer and foundation model.