When I quantized the cache and dropped the context, two numbers got better at once. The wait before the first word, and the speed of the words after it. I filed it under "faster" and moved on.
Wrong frame. Those aren't two readings of one speed. They're two different machines, and they fail for completely different reasons.
Every time a model runs, it goes through two phases. Almost nobody coming from an API knows there are two, because the dashboard only ever showed you the symptoms.
The first phase is prefill. The instant you hit enter, the model reads your entire prompt at once, all of it in parallel, and grinds through a mountain of matrix math to build up what you said (this is also where the KV cache from previous post gets created). Nothing comes out yet. This phase is compute-bound: it's the GPU crunching raw math, and the bigger your prompt, the more there is to crunch. Attention makes it worse than linear — double a long prompt and you roughly quadruple the work.
That wait, before the first token appears, is your TTFT. A one-line question: near-instant. A 30K-token chat history or a fat RAG dump pasted up front: seconds of silence while prefill churns.
Then the second phase takes over: decode. Now the model emits tokens one at a time, and each one reads the whole model out of memory. That's the bandwidth story (check here): memory-bound and steady, capped by how fast the chip gets fed. This phase is your tokens per second, the streaming speed after the first word lands.
Here's the part that rewired how I debug. Prefill wants compute. Decode wants bandwidth. They aren't a single speed you dial up or down; each one answers to different hardware. A huge prompt can leave you staring at nothing for five seconds, then stream like lightning. A tiny prompt can pop instantly, then crawl out at a word a second. The two numbers move independently because different limits govern each.
And once you see it, every API dashboard you ever squinted at reads differently. TTFT was prefill. Tokens per second was decode. You were watching both phases the whole time — you just didn't know the show had two acts.
When a local model feels slow, don't grab a fix yet. Ask which number is bad. A long pause before anything appears is prefill: your prompt is too big or your compute too weak, and trimming the conversation helps. Slow streaming once it starts is decode: bandwidth again, and shrinking the prompt does nothing for it. Most failed "speedups" are aimed at the wrong machine. Name the phase first.
So: next time your setup drags, which is it — the wait, or the stream? They're different problems with different cures, and now you can tell them apart at a glance.
Subscribe and I'll send each one as it lands.
I thought I had one speed dial. I had two engines, and they were never running the same race.