The 9B model was 6.6GB. It sat inside the Mac Mini's 16GB of memory with room to spare. So what dragged "hello" down to one word a second?
Not the model. Something that wasn't in the downloaded file at all.
It's called the KV cache, and it's the part of local inference nobody warns you about — because on the API you never paid for it in memory, only in tokens.
Here's what it is. When the model reads your conversation, every token leaves behind a key and a value at each layer. (Those floors from the layers post? Each one keeps its own set.) The model stashes them all. For the next token, it reuses the whole stash instead of computing it again from scratch.
That stash is the KV cache, and it isn't an optimization you can switch off. Without it, generating token 500 means re-reading all 499 before it, at every step. The cost balloons quadratically. The cache is what turns that into something linear. It's why local generation is fast enough to bother with.
But it has to live somewhere. In the same fast memory as the weights — right next to them. And unlike the weights, it doesn't sit still. It grows with every token, yours and the model's alike. Longer conversation, bigger cache.
This is the part that catches people. The weights are a fixed cost you can check before you start. The cache is a running cost that climbs while you talk.
How fast? A 7B model, smaller than mine, generates enough cache at 32K tokens of context to blow past a 24GB graphics card. The weights are a few gigabytes. The cache, stretched long enough, dwarfs them.
So picture what I did in post 1. I cranked the context to the model's maximum out of reflex, the way you would on an API where context feels free. The weights loaded fine. Then the cache I'd authorized grew until it crossed what the Mac had, and macOS did the only thing it could: it spilled the overflow onto the SSD. Reading the conversation's memory off a disk instead of RAM. That's the one token per second. The model never overran the machine. The cache I reserved did.
The thing that makes generation fast and the thing that ran me out of memory are the same thing.
Be cautious of num_ctx by reflex. Maxing it isn't grabbing free headroom, you're reserving memory. Set the context to what the task actually needs — a summary of three emails doesn't want 128K tokens of room, and every token you reserve is cache you pre-pay for. And when a long chat starts to crawl, don't blame the model. Watch memory climb instead. You're watching the cache fill the room.
So: I trimmed the cache and dropped the context, and the machine sped up — then flattened out, still slow. Because fitting was never the speed problem. That's the next post.
Subscribe and I'll send each one as it lands.
The model fit on day one. What I never counted was the memory of the conversation itself.