I thought I'd found a way to delete the draft model entirely.
The pitch of speculative decoding is that a small model guesses a few tokens ahead, and the big model verifies them in one pass, cheap because the chip's compute sits idle anyway. And whatever the draft proposes, the verified output is mathematically identical to running the big model alone. Garbage in, correct out.
So here was my shortcut. If the output is identical no matter what the draft guesses, why run a draft at all? Hand the big model five random tokens and let it fix them. Right?
Wrong, and the reason is the most important thing about how this works.
Verification isn't five independent checks. It's a chain. In that single pass, the big model scores token 2 assuming token 1 was right, token 3 assuming 1 and 2 were, and on down the line. So the instant one token is rejected, every token after it was built on a prefix that's now wrong, and all of it gets thrown away.
Walk my random idea through that. Token 1 is random, so it's rejected almost immediately. Reject at position 1, and positions 2 through 5 vanish with it. You emit exactly one token: the big model's correction at the mismatch. That's the same single token vanilla decoding gives you per pass — except you also paid to run the draft and to verify four dead positions on top.
Which flips the whole thing on its head. The big model fixing mistakes was never the win. Fixing only ever gets you back to one token per step, normal speed. The win is the draft being right. Every token it guesses correctly is a token the big model skips generating, and that's the only place the speedup comes from.
And the draft can be right shockingly often, because language is predictable. The closing bracket after an opening one. "the," "of," the back half of a word you've half-typed. The same variable name, again, three lines down in code. A tiny model nails those. Land 70% of them and you emit three or four tokens per big-model pass instead of one.
Now the part that corrects what I told you before. I called this speedup free. It isn't. When the guessing goes badly, say unusual text or a draft that doesn't match the big model, acceptance collapses, and you're back to one token a pass while still paying the draft tax. Below roughly fifty percent acceptance, speculative decoding runs slower than plain decoding. Worst case isn't break-even. It's a loss.
So it's a bet. Best case, several tokens for the price of one. Worst case, slower than doing nothing. And what you're wagering on is how predictable your output is.
Before you switch it on, look at what you generate. Code, JSON, structured output, summaries — predictable, high acceptance, near-certain win. Open-ended creative writing or unusual text: measure it, because you might be paying for a speedup you aren't getting. And the draft has to genuinely resemble the big model, same vocabulary, similar instincts; a mismatched guesser loses the bet for you. The speed was never free. It was earned, one correct guess at a time.
So: is your output the predictable kind, or the surprising kind? That one question decides whether this trick pays you or charges you.
Subscribe and I'll send each one as it lands.
The big model fixing your mistakes was never the point. The little model getting them right was.