r/TheMachineLearning • • 20h ago

Nvidia's speculative decoding draft boosts token acceptance by 46%

Post image

Anyone tested the actual tok/s speedup yet?

45 Upvotes

4 comments sorted by

3

u/This_Maintenance_834 6h ago

there is no engine ready to use this. this is all academic research.

1

u/txgsync 4h ago

Incorrect; you can plug any drafter into oMLX that you like today for any model.

1

u/This_Maintenance_834 1h ago

sure, then run it on your machine, and let us know how much improvement you get.

any engine will take any model as draft, but it will produce garbage most of the time and drag down your token generation with it.

1

u/CryptographerOne7003 3h ago

I'm all ears, but is it the same or comparable size as the build in head?