The biggest speedup in local inference is one you never turned on
To work through your prompt, attention compares every token with every other token. A thousand tokens means a thousand-by-thousand grid of comparisons. Flash attention just refuses to build the grid.
The runtime is the new API. You already picked one.
I went local to stop letting a company quietly decide things for me. My third command was ollama run. And just like that, something was quietly deciding everything again.
Bigger means slower: more weights to drag out of memory for every token. Want more capability? Pay for it in speed. By now it feels like physics. But there is a hack!
When I quantized the cache and dropped the context, two numbers got better at once. The wait before the first word, and the speed of the words after it. I filed it under "faster" and moved on.
I did everything the last three posts told me to. Quantized the KV cache down to q8. Dropped the context from maxed-out to 8k. TTFT dropped, tokens per second climbed, the SSD stopped thrashing
