I did everything the last three posts told me to. Quantized the KV cache down to q8. Dropped the context from maxed-out to 8k. TTFT dropped, tokens per second climbed, the SSD stopped thrashing.
And it was still slow. 26 tokens a second — on a model that now fit with room on every side.
That's the part that finally made me read instead of tweak. I'd removed every reason for it to be slow. Nothing was spilling, nothing overflowing. So what was left to blame?
The answer isn't a bug or a setting. It's the speed of memory itself.
Here's the rule that reorganized how I think about all of this. To generate a single token, the model reads essentially all of its weights out of memory: the whole file, start to finish, once per token. Generation isn't the chip thinking hard. It's the chip waiting for memory to hand it the next pile of numbers, over and over.
So your speed has a ceiling, and the ceiling is roughly memory bandwidth divided by model size. How many gigabytes the memory delivers each second, split across how many gigabytes the model makes it read per token. Not how fast the processor computes. How fast it gets fed.
That's what set my 26. The base Mac Mini moves memory at 120 GB/s. That's the number — not the core count, not the neural engine. For a model this size, 120 GB/s lands you in the tens of tokens a second, and 26 is right in that band.
Now the part that reframes the whole machine. The top Apple Silicon, the Ultra chips, read memory at 800 GB/s. Same model, same quantization, nearly seven times the feed rate. That multiplier is most of the tokens-per-second gap between my mini and a maxed Mac Studio. You're not buying a faster brain. You're buying a wider pipe to the same one.
Which is why "Apple's unified memory is a gimmick" gets it backwards. The unified pool is what lets the model fit (check here). The bandwidth is what makes it run. And Apple prices the whole lineup on that second number, climbing 120, 273, 546, 800 in lockstep with the spec that governs how fast a model talks. That's not marketing. For this workload it's the most honest line on the page.
It generalizes past Apple. In a previous post, I waved at this: a datacenter card moves data far faster than a Mac, and that gap is most of your tokens per second. Here's the wave made concrete. Those cards feed memory at multiple terabytes a second, an order of magnitude past anything in a Mac. That's the real reason the API felt instant and your laptop doesn't. Same math, wildly different pipe.
Sizing a machine to run models is two numbers, not one. Capacity (does it fit) is the VRAM question from post 2. Speed (how fast it runs) is bandwidth, and almost nobody looks it up. Find your machine's memory bandwidth, divide by your model's size in gigabytes, and you've got a rough tokens-per-second ceiling before you download a thing. If that ceiling's too low, no setting saves you. Only faster memory does.
So: you can probably recite your machine's RAM and its core count. Can you name its memory bandwidth? That's the one that sets your speed, and it's the one nobody quotes.
Subscribe and I'll send each one as it lands.
The model was never thinking slowly. It was waiting on memory the whole time.