Speculative Decoding and MTP: Why Guessing Is Free

A forward pass over five tokens costs about the same as over one. That gap is the entire speedup, and MTP is how the model drafts guesses to fill it.

Read Original

Related