What a Language Model Is Actually Doing

Most confusion about language models is downstream of one thing: not knowing what the model is computing. The mechanism is not complicated, and knowing it settles a surprising number of arguments.

The mechanism#

Text is split into tokens — roughly, common word fragments. Each token becomes a vector. Those vectors pass through a stack of layers, and at each layer an attention step lets every position look at every other position and pull in what is relevant.

At the end, the model produces a probability distribution over the entire vocabulary for the next token. One token is sampled. It is appended to the input. The whole thing runs again.

That is the loop. Training adjusted several billion weights so that, across an enormous corpus, the distribution puts high probability on continuations that actually occurred.

What the mechanism explains#

Several behaviours that get discussed as mysteries follow directly.

Confident wrong answers. The model is producing a likely continuation, and false statements are frequently likely-looking. There is no internal check against a knowledge base, because there is no knowledge base. Nothing has gone wrong when this happens; the machine is doing the only thing it does.

Different answers to the same question. Sampling is stochastic. Set the temperature to zero and you narrow it, but any change to context, tokenisation or model version moves the distribution. Reproducibility is not a property you get for free.

Poor arithmetic on unusual numbers. Digits are tokens like any other. The model is predicting plausible digit sequences, not executing addition. Common sums appear often enough in training to be memorised; unusual ones are pattern matching that occasionally lands.

Sensitivity to phrasing. A rephrased prompt is a different point in input space and can yield a different distribution. There is no canonical representation of the question underneath.

Where it is genuinely strong#

None of that makes it weak. It makes it specific.

Language models are very good where the task is language-shaped, where the space of valid answers is large, and where the output will be checked by something — a person, a compiler, a test suite.

• Rewriting, summarising, changing register
• Drafting from a specification, where the draft is reviewed
• Translating between structured formats
• Explaining unfamiliar code or prose
• Extracting structure from messy text, with validation on the way out

The unifying property is that a plausible answer, verified, beats no answer.

Where a smaller system wins#

This is the part that is under-published, because it is not the exciting answer.

Exact retrieval. If the answer exists in a document, find the document. BM25 is decades old, runs in microseconds, costs nothing, and returns the actual text rather than a paraphrase of it.

Deterministic scoring and matching. Deduplication, similarity, ranking against fixed criteria. A classical approach gives the same answer every time, runs on a laptop, and can be explained line by line to an auditor. Benchmark it before assuming a model is better — sometimes it is not, and the gap is often smaller than expected.

Anything with a rule. If the decision is "reject the claim when the policy lapsed before the incident date", that is an if statement. Encoding it in a prompt makes a reliable rule unreliable.

Anything requiring an audit trail. "The model said so" is not a justification. Where a decision must be defended, the decision procedure has to be inspectable.

The honest summary#

A language model is a very strong general-purpose text transformer with no notion of truth and no guarantee of consistency. Used where those properties are acceptable, it is the best tool available and the difference is not close. Used where exactness, reproducibility or auditability matter, it is a worse version of something that already exists.

Most bad AI engineering is not a modelling failure. It is applying it to the second category and then trying to prompt the properties back in.