Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior.
Large language models (LLMs) support a wide range of applications, but serving them at scale requires careful optimization. Standard autoregressive decoding is the baseline used by most LLM serving systems: the model generates one token, appends it to the sequence, and then uses the updated sequence to generate the next token. This process is simple and reliable, but the serving loop still advances one committed token at a time because output tokens must be produced in strict left-to-right order.
Speculative decoding builds on this baseline through a draft-and-verify mechanism. A lightweight draft component proposes candidate future tokens, and the target model verifies those candidates before they are committed. When several draft tokens are accepted, the system can commit multiple output tokens from a single target-model verification step while preserving the target model's output behavior.
This post explores how speculative decoding works in vLLM and shares measurements from the test environment. It first reviews the autoregressive decoding baseline and the draft-and-verify process. It then examines five speculative-drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods differ in how the draft component receives information from the target model and whether candidate tokens are generated sequentially, autoregressively, in parallel, or through a hybrid approach. Finally, it shows how to enable the methods tested in the environment, reports measurements from experiments on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ open software platform, and discusses practical tuning and observability considerations.
In standard autoregressive decoding, each decode step produces and commits one new token. For example, generating four output tokens requires four sequential decode steps. After each step, the generated token is appended to the sequence and becomes part of the input for the next step. This makes the decoding loop straightforward, but it also requires one model decode step for every output token. During long generations, this token-by-token loop can dominate latency and limit serving throughput.
The key question behind speculative decoding is whether it can preserve the output behavior of the original model while reducing how often generation advances by only one token at a time. Speculative decoding addresses this by separating proposal from verification. A draft component first proposes several candidate future tokens. The original model, acting as the target model, then verifies those candidates before they are committed.
Speculative decoding does not replace the original model. Instead, it keeps the original model as the target model, which remains responsible for the final output, and adds a faster proposal stage in front of it. During each speculative decoding round, a lightweight draft component proposes one or more future tokens. These tokens are only candidates and are not committed immediately. The target model then evaluates the candidate token sequence in one verification pass.
Verification proceeds from left to right. Each draft token is checked using the target model's result at the corresponding position. Accepted tokens are committed to the output sequence. When a draft token is rejected, later candidates from the same proposal are no longer accepted. If a draft token is rejected, the target model provides the next token. The remaining draft tokens are discarded, and generation continues from the updated sequence.
Although all speculative decoding methods follow the same overall draft-and-verify process, they differ in how the draft component is designed and how it works with the target model. Based on these differences, the drafting methods discussed can be grouped into three broad categories: native MTP modules, separate MTP drafters, and dedicated target-conditioned draft networks. These categories describe the draft component architecture, not the target-model family. A target model may support native MTP while also having separately trained EAGLE-3, DFlash, or DSpark draft models.
The draft component does not operate entirely on its own. Depending on the method, the draft component may receive various types of information from the target model. Multi-Token Prediction (MTP) refers to a family of model-native mechanisms for predicting tokens beyond the immediate next token. In vLLM, native MTP is available when the target model includes a compatible auxiliary prediction component. The exact MTP architecture varies across model families, but each implementation provides an auxiliary path for proposing future tokens.
At the first speculative step, the MTP component combines a hidden representation from the target model with information from the current token to predict the first draft token. At subsequent steps, the newly drafted token and the hidden state produced by the previous MTP step are used to predict the next candidate. After the configured number of candidates has been proposed, the target model evaluates them together in one verification pass.
Many native MTP implementations follow a similar pattern. A hidden representation from the target model or from the previous MTP prediction is combined with the embedding of a shifted input token or the latest drafted token. The two inputs serve different purposes: the hidden representation carries information about the preceding sequence, and the token embedding identifies the latest token from which drafting continues. In common implementations, they are combined along the hidden dimension and transformed before entering the auxiliary prediction layer.
The number of physical MTP layers and the configured speculative length are separate concepts. When the number of speculative tokens exceeds the prediction depth directly provided by the checkpoint, vLLM can reuse the MTP path through additional forward passes. A larger value therefore proposes more candidates before verification, but also introduces more sequential drafting work.
Native MTP is closely tied to the target-model architecture. In many implementations, parts of the MTP path share components with the target model, which can keep the additional memory overhead relatively modest. However, generating multiple speculative tokens still requires sequential drafting before verification.
Gemma 4 uses a separately packaged MTP draft component paired with a specific target model. Although the draft component has its own checkpoint, it remains closely connected to the target model during inference. The draft component uses activations produced by the target model and shares the target model's KV cache. This allows it to reuse contextual information that the target has already computed instead of processing the accepted prefix independently.
As with native MTP, the number of layers in the draft component is separate from the configured speculative length. When several candidate tokens are requested, the draft component generates them sequentially. EAGLE-3 uses a dedicated draft network trained for a specific target model. The draft component has its own execution path, but it remains closely conditioned on information produced by the target model.
During the target-model forward pass, EAGLE-3 records hidden states from three stages of the target Transformer: near the beginning, around the middle, and near the end. These are contextual representations of the same accepted sequence at different stages of target-model processing. The three hidden states are concatenated and projected into a single fused target feature. This fused representation is then combined with the embedding of the sampled token before entering the EAGLE-3 draft decoder.
EAGLE-3 generates draft tokens autoregressively. For the first draft token, it uses the fused target feature computed from the accepted sequence together with the sampled-token embedding. After a draft token is produced, its embedding is fed into the next drafting stage. Because the target model has not yet processed the later speculative positions, target-model hidden states for those positions are not available. EAGLE-3 therefore uses the previous draft-component output when continuing the draft sequence. This sequential feedback gives later draft tokens direct dependence on earlier drafted tokens along the proposed sequence. However, generating more speculative tokens also requires more sequential drafting work before verification.
DFlash uses a dedicated draft network trained for a specific target model. Unlike MTP and EAGLE-3, which generate candidate tokens sequentially, DFlash predicts a whole block of future positions in parallel. DFlash begins each draft block with an anchor token. The anchor is a known token produced or confirmed by the target model, so DFlash does not need to predict it. Instead, it provides a known starting point for the masked positions that follow. In later decoding rounds, this is typically the additional target token returned by the previous verification pass. The anchor occupies the first position of the block, while the remaining positions are masked and predicted in parallel. A single DFlash forward pass predicts all masked positions together. Like EAGLE-3, DFlash first combines hidden states from several target-model layers into a fused representation. The main difference is how this fused representation is used. EAGLE-3 combines it with the sampled-token embedding at the input of its autoregressive draft network. DFlash instead converts the fused target context into additional Key and Value representations that are available in every layer of the draft network. Queries from the masked draft positions can therefore attend to both the target-model context, which remains available throughout the draft network.