You already know long context costs you. The KV cache grows with every token you add, gigabyte by gigabyte. That's the cost you can see.
There's a second one, during processing, that's far worse, and by rights it should have made long prompts unusable. It didn't. The reason is an optimization running on your machine right now that you never switched on, and have never once thanked.
Here's the cost. To work through your prompt, attention compares every token with every other token. A thousand tokens means a thousand-by-thousand grid of comparisons. Double the prompt and you quadruple the grid. It grows with the square of your context, not in step with it.
The obvious way to compute that is to build the whole grid in memory, then boil it down. And the grid is enormous: at a 16,000-token context, a single one of these score matrices runs about 512MB — per attention head, and there are many. The model writes all of that to slow memory, reads it back, runs softmax, and throws nearly all of it away, because the only thing it actually needed was a much smaller final answer. That round trip, not the arithmetic, is what made long context crawl.
Flash attention just refuses to build the grid.
Instead it walks your prompt in small blocks sized to fit in the chip's tiny pool of fast on-chip memory, computing each piece there and keeping a running tally as it goes. The giant matrix never gets written down anywhere. And here's the part that matters: the answer is exact. Not an approximation, not a close-enough shortcut, the identical numbers standard attention would produce, arrived at without ever materializing the thing that was clogging memory.
The math stays quadratic; you can't cheat the comparisons. But the memory traffic, the part that was actually killing you, drops from quadratic to linear. In practice that's two to four times faster on attention, and a memory footprint that no longer explodes with length. It's the difference between a 100K-token window being a research demo and being something you just use.
And you did nothing to get it. Flash attention is on by default in llama.cpp and every serious runtime. The single biggest reason your long prompts run at all is a thing you never enabled and would never have known to ask for. Your runtime turned it on and never mentioned it.
There's no switch to flip, it's already flipped. The useful move is to stop fearing long context for the wrong reason. The scary quadratic, the attention matrix, is handled and invisible. What still grows on you is the linear one: the KV cache, which you can see and run out of. So when a long prompt finally hurts, it's almost never the attention. It's the cache. Aim your worry at the cost that's actually still yours.
So: you've spent this series learning the costs the API hid. Here's one your own runtime hid, and you were better off for it. How many more of those are holding your setup together right now?
Subscribe and I'll send each one as it lands.
The best optimization you run is the one you never knew you turned on.