Every post in this series has sold you the same grim deal. Smarter means bigger. Bigger means slower: more weights to drag out of memory for every token. Want more capability? Pay for it in speed. By now it feels like physics.
The biggest, smartest models break it. A model can be enormous and still generate fast. Here's the trick the frontier runs on, and the catch nobody mentions.
Start with what you know. A normal "dense" model runs every one of its weights for every token. All those floors from the layers post, top to bottom. That's why size and speed are joined at the hip: more weights, more reading, slower output.
A Mixture-of-Experts model refuses to do that. It splits its guts into many sub-networks called experts, and adds a tiny router that, for each token, picks just a couple of them to run. The rest sit out that turn.
Mixtral is the clean example. 47 billion parameters total, but any given token only fires about 13 billion of them. It matched Llama 2:70B on quality while running roughly six times faster. Knowledge of a 70B. Speed closer to a 13B. The router makes the difference: each word only pays for the experts it actually needs.
That's how DeepSeek-V3 holds 671 billion parameters and only wakes 37 billion per token. Most frontier models released in the last two years work this way. It's why they keep getting bigger without getting proportionally slower, and, together with batching, much of why the hosted ones feel quick.
Now the catch, because there is a real one.
All 47 billion of Mixtral's parameters still have to be in memory. The router is allowed to reach for any expert on the next token, so every expert has to be sitting there, loaded, ready. MoE shrinks how much the model runs per token. It does nothing for how much the model weighs.
This wrecks the obvious assumption. Mixtral "runs like a 13B," so you'd guess it fits like one. It doesn't. At Q4 it wants around 26GB, more than a 24GB graphics card and nowhere near my 16GB Mac. A plain dense 32B would load on hardware this "13B-speed" model won't touch. The thing that computes small still stores huge.
Here's what changed. Until now, "model size" was one number that set both whether it fit and how fast it ran. MoE pries that number in two. There's a total size, what you must fit in memory. And an active size, what gets read per token, which sets your speed. On a dense model they're identical. On an MoE they're worlds apart.
When a model is labeled 8x7B or 671B/37B, read both numbers. The first is your fit budget — judge it against your RAM, and don't let "runs like a small one" fool you. The active number is your speed budget. On a tight machine like Mac Mini Base model, MoE won't help you fit anything; total size is still the wall. But the day you have the memory, an MoE buys more brain per token of speed than any dense model. That's the pitch.
So: that model you were eyeing because it "runs fast for its size" — do you know what it actually weighs? Those are two different questions now. They didn't used to be.
Subscribe and I'll send each one as it lands.
Smaller to run, same size to hold. The frontier didn't beat the tradeoff. It moved it somewhere you weren't looking.
PS : the MoE mechanism itself changes nothing about the KV cache cause this is the pre-calculated token in the context.