Hardware for Local AI: VRAM, RAM and Apple Silicon

First half: the weights

[Own calculation] The formula is simple:

weight size [GiB] = bits_per_weight × parameter count / 8 / 2^30

bits_per_weight depends on the quantization level. Here is the finished table, weights only:

An empirical correction that is written down almost nowhere. [Own calculation from measured file sizes] At the four-, five- and six-bit levels the real file is 6-7% larger than the nominal figure, because the quantizer leaves the embedding and output layers at higher precision. The eight-bit level, by contrast, comes out at exactly what the formula gives.

In other words: at the four-bit levels, add seven percent.

Second half: the KV cache everyone skips

This is the most important part of the article, because most sizing mistakes start here.

While it generates, the model stores a key and a value vector for every token it has processed, so it does not have to recompute them. That is the KV cache, and it grows linearly with context length.

[Own calculation] The formula:

KV cache [bytes/token] = 2 × layers × KV heads × head dim × bytes

  2       : key and value stored separately
  bytes   : 2 at sixteen bits, 1 at eight bits, 0.5 at four bits

On one concrete eight-billion-parameter model — 36 layers, 8 KV heads, head dimension 128:

Read that together with the first table: the four-bit weights of the same model eat 4.6 GiB. Its native context eats 4.5 GiB as well. Its full, extended context eats 18 GiB — four times the model.

Two practical consequences:

1. The KV cache can be quantized, and it is close to free. According to one developer measurement, an eight-bit KV cache costs +0.0043 perplexity — effectively unmeasurable. Four-bit already costs +0.21, which is noticeable but acceptable for plenty of tasks. The saving, on the other hand, is half and a quarter respectively.

(A warning from the author of that measurement: do not use it with embedding and vision models, and be careful with models that have many attention heads.)

2. Grouped attention is the only reason any of this fits at all. [Own calculation] If the same model used classic attention with 32 KV heads instead of 8, its 128k-token context would be 72 GiB. As it is, 18.

The full formula

Memory needed ≈

    weights
  + KV cache × number of parallel requests
  + activations and compute buffer (0.5-2 GiB)
  + runtime (0.5-1 GiB)
  + [on Apple machines] the OS and everything else

The "parameter count divided by two" rule holds at short context. Above 32k tokens it is off by an order of magnitude — in the example above, the KV cache alone is a 400% surcharge.

The surprise: prompt and generation are not the same thing

Here is the least known fact in this article, and it comes from a primary source.

From the manufacturer's own research blog: "generating the first token is compute-bound" and fully saturates the accelerators. "Generating the following tokens is limited by memory bandwidth," not by compute.

On a first reading that sounds like a technical footnote. Then you look at what it means in practice:

The cheaper chip, with half the GPU cores and 56% of the bandwidth, processes a prompt almost twice as fast — and generates a fifth slower.

The practical decision follows from that:

FEEDING IN LONG PROMPTS?
(codebase, documents, RAG chunks)
-> COMPUTE is what counts
-> newer-generation chip, with matrix accelerators

GENERATING LONG ANSWERS?
(writing, agent steps, summarising)
-> BANDWIDTH is what counts
-> bigger, more expensive chip

AND THE MOST IMPORTANT PART:
a "tokens per second" figure in any marketing
deck ALWAYS measures one of the two,
never both.

Apple machine or graphics card?

The point is not "shared versus dedicated memory" but three numbers: capacity, bandwidth, and the ratio between them.

The real difference comes down to four points:

1. Capacity. On the top consumer graphics card 32 gigabytes is the ceiling. The four-bit weights of a 70-billion-parameter model come to 37.3 GiB — they do not fit on any consumer graphics card, but they do fit in a 64-gigabyte Apple machine.
2. Bandwidth. The graphics card has roughly three times the bandwidth of the high-end Apple chip. Whatever does fit, the graphics card generates much faster.
3. No copying. With unified memory the CPU and the GPU see the same thing. With a discrete card the bus is the bottleneck whenever layers have to be moved.
4. Power draw. The high-end Apple desktop peaks at 145 W; the top graphics card at 575. That feeds into the payback calculation too, which the why run AI locally post works through step by step.

One usable rule of thumb [Own calculation]:

Theoretical max generation speed ≈ bandwidth / model size

(because every generated token requires reading
 the whole model out of memory)

Measured values typically land at 45-82% of that, depending on the chip. One example: a 3.8-gigabyte model, 614 GB/s of bandwidth → 161 tokens/s in theory, 120 measured — 74% utilisation.

Measured speeds

Figures from an official, continuously maintained performance table, all with the same seven-billion-parameter model. PP = prompt processing, TG = text generation:

And one lesson that matters as much as the hardware choice: on the same machine with the same model, the choice of runner produced a 1.2-1.9× difference in an academic measurement. In another measurement, on hardware from a different manufacturer, the choice of backend library produced a sevenfold difference in raw compute throughput. Runners are compared in the local AI runners piece, one by one.

How much do you need, and what does it cost?

Derived from the numbers above, at four-bit quantization and with realistic context lengths:

* The weights column here already includes the +7% correction above; the bare formula gives 37.3 GiB.

The right-hand column also covers the OS and everything else you have running — which is why the two middle columns do not add up to it.

And one market warning that cannot be left out in 2026: memory prices are not on the buyer's side. In September 2026 the street price of high-end graphics cards was two to two and a half times the manufacturer's list price, driven by data-centre demand. That shifts the payback maths materially.

The how-to

1. COUNT THE KV CACHE TOO, NOT JUST THE WEIGHTS
   At long context the KV cache can be several
   times the model. This is the most common error.

2. DECIDE WHICH YOU DO MORE OF
   Feeding in long prompts, or generating long
   answers? A different chip wins each one.

3. FOUR-BIT QUANTIZATION IS THE DEFAULT
   Start there. The numbers are in the quantization
   article - four-bit is essentially free.

4. TURN ON THE EIGHT-BIT KV CACHE
   Halves the memory the context needs, for an
   unmeasurable quality cost. Do NOT use it with
   embedding and vision models.

5. IF IT DOES NOT FIT, TRY A SMALLER MODEL FIRST
   A fast smaller model beats a big one that runs
   half off the disk.

6. COUNT THE IDLE POWER DRAW AS WELL
   An always-on desktop can burn 600 kWh a year
   doing nothing at all.

7. TEST BEFORE YOU BUY
   Same model, same machine, a different runner:
   up to a twofold speed difference.

Common misconceptions

• "Parameter count divided by two, done." True at short context. Above 32k tokens the KV cache adds an order of magnitude.
• "More GPU cores, faster generation." Generation is limited by bandwidth. In the measurement the chip with half the cores processed the prompt faster and generated slower.
• "Shared memory is always worse." On capacity it is better: a 70-billion-parameter model does not fit on a consumer graphics card, but it does fit in 64 gigabytes of unified memory.
• "Four-bit quantization degrades quality significantly." In a unified 2026 measurement the perplexity increase is +0.24 at the most common level — smaller than the spread between models.
• "The hardware decides everything." On the same machine, the choice of runner and of backend library produced a twofold and a sevenfold difference respectively in measurements.
• "The 'tokens per second' number is comparable." Only if you know whether it refers to prompt processing or to generation. The two can differ by as much as twenty-five-fold on the same machine.

What to take away

Sizing is made of two numbers, not one: the weights and the KV cache. The second grows with context length, and at long context it can be several times the model — the most common and most expensive planning mistake there is.

For the hardware choice, a single question decides it: do you feed in long prompts, or generate long answers? The first needs compute, the second needs bandwidth — and the same money buys a different machine in each case.

And before you buy anything: the same model on the same machine, with a different runner, showed a twofold speed difference in measurement. It is worth optimising the software before swapping the hardware. Quantization is the subject of the quantization post, in detail.