← the lab Hub

Demo 02

Speculative decoding

A big model is slow because every token costs a full forward pass. So let a small model guess several tokens ahead, then have the big one check the whole guess in a single pass. Everything it got right is free. Everything after the first mistake is thrown away. The interesting part: guessing further ahead does not keep helping.

6

How far the small model runs ahead before the big one checks.

0.75

Chance the big model agrees with any single guessed token.

0.15×

Cost of one draft step relative to one big-model pass.

Tokens per cycle
—

Expected accepted, plus the one the big model contributes free.

Cost per cycle
— passes

k draft steps plus one verification pass.

Speedup
—×

Against decoding one token at a time.

Best draft length
—

At this acceptance rate and draft cost.

Speedup vs draft length this acceptance rate ±0.10 break-even
One run, token by token

Why the curve turns over

Each cycle costs k cheap draft steps plus one expensive verification pass. It returns however many leading guesses survive. Because a single wrong token invalidates everything after it, the expected yield is a geometric series — it saturates. Doubling the draft length does not double the tokens you keep.

expected tokens per cycle  =  (1 − pk+1) / (1 − p)
cost per cycle  =  k·c + 1
speedup  =  tokens / cost

The numerator flattens out; the denominator keeps climbing in a straight line. So there is always a peak, and past it every extra guessed token is pure overhead. Drag acceptance up and watch the peak slide right — the more often the draft is correct, the further ahead it is worth running.

Acceptance rate is not the target. When this was tuned on real prompts, the configuration with the best acceptance rate delivered the worst throughput. Acceptance is an input to the arithmetic above, not the thing being maximised. It is easy to raise it by being more conservative, and be slower for it.

What this model leaves out. It assumes each token is accepted independently with the same probability, that draft cost is linear in k, and that verification costs exactly one pass regardless of k. Real systems violate all three — acceptance is bursty, batching changes everything, and a longer draft makes the verify pass slightly more expensive. The shape holds; treat the absolute numbers as the shape of the trade-off rather than a prediction.