speculative-decoding

Use Speculative Decoding when you need to: Key Techniques : Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration) Papers : Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024) Idea : Use small draft model to generate candidates, large target model to verify in parallel. Algorithm : Performance : Source : arXiv 2401.10774 (2024) Innovation : Add multiple prediction heads to existing model, predict future tokens without separate draft model. Architecture : Training : Tree-based Attention : Advantages : Source : ICML 2024 Core idea : Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration. Mathematical formulation : Two branches : Lookahead Branch : Generate n-grams in parallel Verification Branch : Verify promising n-grams Performance : Draft Model Speculative : Medusa : Lookahead : - Speed up inference by 1.5-3.6× without quality loss - Reduce latency for real-time applications (chatbots, code generation) - Optimize throughput for high-volume serving - Deploy efficiently on limited hardware - Generate faster without changing model architecture - Draft model generates K tokens speculatively - Target model evaluates all K tokens in parallel (single forward pass) - Accept tokens where draft and target agree - Reject first disagreement,...