Quantization: How Big Models Fit on Small Machines

What is quantization, in one paragraph?

A model's weights are numbers. By default they are stored on 16 bits. Quantization cuts that down — to 8, 5, 4, sometimes 3 bits — and in exchange the file gets smaller, is read faster, and fits where it did not fit before.

The stored information is less, though, so the model loses something. The only question is how much — and when.

The naming follows the pattern Q<bits>_<variant>, where _K marks the more advanced "k-quant" family and _S / _M / _L the size within that family. The actual bits per weight is rarely a round number:

(From the three-part labels: Q3_K_S is 3.41 bits/weight.)

How much does it degrade? The measurement

The best available data comes from a unified January 2026 study: one model, 13 quantization configurations, run through a public evaluation framework.

Perplexity first — the classic metric for language models:

So far, tidy and dull. Task accuracy tells a different story:

Four findings, and every one of them changes something:

1. Four-bit k-quants are essentially free. Across the study's full benchmark set the average loss is 0.43 percentage points, for a 71% cut in size.

2. Three bits already hurts — but not evenly. On maths problems it drops 9.3 points, on common-sense tasks barely anything. Multi-step reasoning breaks first.

3. Perplexity is a poor predictor. In the author's own words: several five-bit variants have practically identical perplexity while their task accuracy differs significantly. Which means perplexity does not tell you whether the model breaks on the thing you use it for.

4. The name is not the bit count. Within the same nominal bit width, Q3_K_L recovers almost all of the loss Q3_K_S takes. According to the author, the details of the format matter at least as much as the bit-width label.

(The limits of the measurement, which have to be spelled out: one model, one size, CPU inference, one author. The quantized models are public and the methodology is reproducible.)

And the table that circulates on every blog

There is a 2023 perplexity table that nearly every summary quotes. It is worth knowing that it is outdated, and not mildly so: for the same level, the measurement of the day showed a +0.05 perplexity increase, while the 2026 one shows +0.24.

The reason is interesting: newer, more "saturated" models are more sensitive to quantization. As models pack more into fewer parameters, there is less slack left in the weights.

Where it degrades disproportionately — the non-English part

This is the chapter that gets left out of nearly every summary of the topic.

A peer-reviewed 2025 study measured specifically what happens on long input (above 64k tokens) and in languages other than English. The sample: 9,700 test cases, 5 quantization methods, 5 models, across five benchmarks — including a retrieval task in 26 languages.

Average degradation by method:

And the largest individual collapses:

The language finding, and the reason this article does not end where the English-language write-ups end:

According to the study's own summary, degradation outside English runs up to five times the English figure — and Hungarian was one of the 26 languages measured.

The authors' own summary, word for word: "eight-bit quantization preserves accuracy (~0.8% drop), while four-bit methods lead to significant losses, particularly on long-context inputs (with drops of up to 59%). This degradation typically worsens when the input is not in English."

Two measurements, two opposite lessons — and they do not contradict each other. The difference is the length and the language of the input:

SHORT, ENGLISH INPUT
-> 4-bit quantization is essentially free
   (0.43 points lost, 71% smaller)

LONG (64k+) OR NON-ENGLISH INPUT
-> 4-bit is not the same bargain
   (non-English degradation up to fivefold)

So on a local model that works in a language other than English and chews through long documents, it is worth stepping up to five or eight bits. That is the most concrete piece of advice in this article — and it is in none of the standard guides.

The payback curve

[Own calculation] Read the size table and the quality table together, and it is clear where each step is worth taking:

The big gains are in the first two steps. Going below four bits is a bad deal — unless the question is whether it fits at all.

And speed comes with it: since generation is limited by memory bandwidth, a smaller file also generates faster. Prompt processing does not work that way, though — there, speed varies irregularly from format to format, because the kernels differ. That is covered in the hardware for local AI post in more detail.

The other half, which everyone forgets

Most people quantize the weights and leave the context alone. Yet above 32k tokens the KV cache is what eats the memory — on the same model, the context can take up four times the model itself.

The KV cache can be quantized too, and by measurement it is almost free:

An eight-bit KV cache halves the memory the context needs, for a quality loss that is effectively unmeasurable. Two exceptions the author of the measurement warns about: do not use it with embedding and vision models, and be careful with models that have many attention heads.

Which tool? — and what has died

This section ages fast, so it is worth checking. Status as of September 2026:

The lesson worth stating: anyone installing the quantizer under its classic name in 2026 is installing from an archived repository. The functionality is alive, just under a different name.

One technical detail that matters a lot: the importance matrix. This is a weighting measured on a calibration dataset, with which the quantizer spares the channels that matter. According to the official documentation it is explicitly recommended below Q5_K_M, and it typically improves the result by 10-30%.

Which one should you pick?

SHORT, ENGLISH TASK, LITTLE MEMORY
-> Q4_K_S or Q4_K_M
   This is the general default.

QUALITY FIRST, ROOM TO SPARE
-> Q5_0 or Q5_K_M
   In one measurement Q5_0 averaged BETTER
   than the 16-bit reference.

LONG CONTEXT OR NON-ENGLISH TEXT
-> Q8_0, or at least Q5
   Here 4-bit is not the same bargain.

THE ONLY WAY IT FITS
-> Q3_K_L, NOT Q3_K_S
   At the same nominal bit width, K_L recovers
   almost all of the loss.

AND IN EVERY CASE
-> turn on the 8-bit KV cache
   (except with embedding and vision models)

Common misconceptions

• "Quantization dumbs the model down." At four bits, across the full benchmark set, it takes an average of 0.43 percentage points for a 71% cut in size. On short, English tasks that is essentially free.
• "Perplexity tells you how much it degraded." It does not. Variants with identical perplexity had significantly different task accuracy in the same measurement.
• "The bit count tells you everything." The details of the format matter just as much: at the same nominal bit width, Q3_K_L recovers the loss Q3_K_S takes.
• "What holds in English holds everywhere." In the 26-language measurement, degradation outside English was up to five times the English figure on long input.
• "Quantizing the weights is enough." Above 32k tokens the KV cache can be the bigger item. An eight-bit KV cache halves it, at an unmeasurable price.
• "The perplexity table doing the rounds on blogs is current." It is from 2023, and newer models are more sensitive: for the same level, today's figure is +0.24 rather than +0.05.

What to take away

Quantization is the best deal in local inference — the four-bit level saves seventy percent of the memory for half a percentage point of accuracy, and for most tasks that is the end of it.

But in two places it does not work that way, and both matter: on long input and in languages other than English the degradation is an order of magnitude bigger, up to fivefold according to the study. If you work with Hungarian documents on a local model, step up to five or eight bits — four bits is not the same bargain there.

And one setting almost nobody switches on: the eight-bit KV cache. It halves the memory the context needs, and by measurement the quality loss is effectively immeasurable. The runners are lined up side by side in the local AI runners comparison, and the payback side is what the why run AI locally post works out.