The bandwidth post ended like a sentence handed down by a judge. Your generation speed is memory bandwidth divided by model size, and bandwidth is soldered into the machine. Want more tokens per second? Buy a faster Mac. Nothing else moves the number.
That conclusion is wrong, and the loophole is hiding inside the formula I gave you.
Look at it again. Bandwidth divided by model size. Bandwidth is fixed, that part's hardware. But the formula smuggled in an assumption neither of us questioned: one full read of the model per token. Every word you generate, you haul the entire model out of memory once. That "once per token" isn't a law. It's just how decoding has always worked. And it's the term you can actually attack.
Here's why attacking it is even possible. When the chip reads the model to make a token, it does almost no math with those weights — the reading is the whole cost, and the compute sits idle. You've got arithmetic to spare and no way to spend it. Hold that thought.
Speculative decoding spends it. A small, fast draft model, a fraction of the size, guesses the next handful of tokens. Cheap, because it's tiny. Then your real model checks all those guesses in a single pass. And checking five tokens costs it about the same as making one: the expensive read happens once either way, and verifying a few extra positions is just a little more math, the thing you had lying around unused.
When the guesses are right, you walk away with several tokens from a single read. The formula's hidden constant, one read per token, just became one read per four, or five. Same hardware. More speed.
Guess wrong, and the big model catches it in that same pass and swaps in the right token. Nothing wrong escapes. The output is identical, character for character, to what the big model would have produced alone. Not close. Identical.
That's the part worth underlining. You didn't trade quality for speed the way quantizing trades quality for size. You changed a number in the formula that wasn't the hardware — and got speed for free.
There's a cost, of course. The draft model has to sit in memory too, which on a tight machine is room you may not have. And the win scales with how often the little model guesses right: strong on predictable text like code, weaker when the model lunges somewhere the draft couldn't follow.
If your runtime supports a draft model (llama.cpp does), pair your main model with a tiny one from the same family and turn it on. You won't have changed a single piece of hardware. You'll have stopped paying a full memory read for every single token, which is the only reason the ceiling sat where it did.
And keep the bigger move in your pocket. When you hit a hardware wall, read the formula before you read the price tags. Sometimes the term pinning you isn't the hardware at all. It's an assumption sitting right next to it.
Subscribe and I'll send each one as it lands.
Bandwidth was never the ceiling. "One read per token" was. And that one, you get to change.