Speculative Decoding: A Small Model Guesses, the Big One Checks

A small draft model guesses several tokens ahead, and the large model verifies the whole guess in one pass - producing the exact same output, just faster.

The core trick

Normally a large model generates text one token at a time, and each token needs its own full forward pass - slow, since most of that compute produces just one word. Speculative decoding adds a small, fast "draft" model that guesses several tokens ahead cheaply. The large model then checks the whole guessed sequence in a single pass, instead of generating token by token itself.

Why the output doesn't change

This is the part people don't expect: speculative decoding isn't an approximation. The large model still verifies every guessed token against what it would have generated on its own. Wherever the draft guessed right, that token is accepted for free. The moment a guess is wrong, everything from there is discarded and the large model generates the correct token itself. The final text is identical to plain, token-by-token generation alone.

Why checking a block is fast

Checking several tokens at once takes about the same compute as generating one, since verification runs all guessed positions through the network in parallel rather than sequentially. A good run of correct guesses gets several tokens for the price of one.

What actually gets faster

Latency, not quality. The output is the same as un-sped-up generation alone - a serving optimization, not a capability change. It doesn't make the model smarter; it only reduces the wall-clock time to get the same answer.

EXAMPLE

Simplified walk-through generating "The quick brown fox jumps": 1. Draft model (small, fast) guesses 4 tokens ahead: "quick brown fox runs" 2. Large model checks all 4 positions in one parallel pass against what it would generate: - "quick" - matches, accept - "brown" - matches, accept - "fox" - matches, accept - "runs" - does NOT match (large model would say "jumps") - reject 3. Everything up to the mismatch is kept ("quick brown fox"), the large model generates the correct next token itself ("jumps"), and the draft model gets a fresh chance to guess ahead from there. Result: 3 tokens accepted "for free" from one large-model verification pass, instead of costing 4 separate full passes - even though the last guess was wrong and had to be corrected. Rough throughput comparison from published benchmarks (Leviathan et al., 2023): standard decoding on T5-XXL vs. speculative decoding with a small draft model showed a 2-3x wall-clock speedup, with byte-for-byte identical output.

🛠️ EXERCISE — TRY IT YOURSELF

Reason through an acceptance-rate scenario to build intuition for when speculative decoding helps most.

Take a piece of highly predictable text (e.g. boilerplate JSON or a common code pattern) and a piece of unusual, creative text (e.g. a surprising plot twist).
For each, guess how often you think a small draft model would correctly predict the next few tokens.
Write down your predicted acceptance rate (rough %) for each case.
Look up or reason through what a low vs. high acceptance rate does to the effective speedup.
Conclude which of your two texts would benefit more from speculative decoding, and why.

✅ SELF-CHECK

☐ Did you correctly predict that the more predictable/formulaic text would have a higher acceptance rate?
☐ Can you explain in your own words why a rejected guess never makes output slower than standard decoding, just less faster?

QUICK QUIZ

What happens to the final generated text when speculative decoding is used instead of standard token-by-token generation?