Aug 7, 2026
•
4 min read
Optimization techniques
+1
Most multi-turn RAG chatbots work like this: append each turn to a message list, send the whole thing, let the model sort it out. Splitting them is a small change with a measurable payoff.
Jul 1, 2026
2 min read
+2
To work through your prompt, attention compares every token with every other token. A thousand tokens means a thousand-by-thousand grid of comparisons. Flash attention just refuses to build the grid.
Jun 28, 2026
Best case, several tokens for the price of one. Worst case, slower than doing nothing. And what you're wagering on is how predictable your output is.
Jun 27, 2026
LLM Concepts
I went local to stop letting a company quietly decide things for me. My third command was ollama run. And just like that, something was quietly deciding everything again.
Jun 24, 2026
Pop-up wardrobes, tape-to-stream labs, and story prompts powering this week’s experiments.
Jun 22, 2026
Bigger means slower: more weights to drag out of memory for every token. Want more capability? Pay for it in speed. By now it feels like physics. But there is a hack!
Jun 18, 2026
When I quantized the cache and dropped the context, two numbers got better at once. The wait before the first word, and the speed of the words after it. I filed it under "faster" and moved on.
I did everything the last three posts told me to. Quantized the KV cache down to q8. Dropped the context from maxed-out to 8k. TTFT dropped, tokens per second climbed, the SSD stopped thrashing
The 9B model was 6.6GB. It sat inside the Mac Mini's 16GB of memory with room to spare. So what dragged "hello" down to one word a second?
