3.7x faster inference with a DSpark (speculative decoding) draft model for gemma-4-E2B-it
Gemma-4-E2B-it-DSpark
A DSpark draft model for speculative decoding with google/gemma-4-E2B-it as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone. This makes it a lossless speedup. Mean acceptance length is 2.89 tokens committed per verification step, up to 5.53 on math_reasoning. On an A100 at concurrency 1, that gives a throughput speedup over the verifier alone of 3.74× on math_reasoning and 2.42× on HumanEval.
Training code: rasyosef/train-dspark-draft-models.
Trained on 100,000 samples.
Usage
vLLM loads the verifier automatically from the config, so don't pass it separately.
vllm serve rasyosef/gemma-4-E2B-it-dspark --port 8000 --gpu-memory-utilization 0.8
Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
Details
Architecture. A Qwen3-style decoder with 5 draft layers, ~0.25B params, bfloat16. It has hidden size 1536, MLP size 6144, and 12 query heads over 2 KV heads (GQA, head dim 128). The first four layers use sliding-window attention (window 2048) and the last uses full attention. Block size is 8. The draft vocabulary is reduced to 50,000 tokens from the verifier's full 262,144, selected by token frequency over the training data. Aux hidden states come from verifier layers 2/11/22/33.
Training. 3 epochs at lr 3e-4 (AdamW, cosine schedule with 4% warmup) with a {"ce": 0.1, "tv": 0.9} loss. The data is 100,000 filtered Open PerfectBlend prompts regenerated by the verifier itself, split 96/4 into train and validation. Prompts were prepared at 768 tokens. Training sequence length was 8192, with up to 1024 anchors per sample. Verifier hidden states were pulled on demand from a running vLLM server during training and deleted after use, rather than staged to disk up front.
Evaluation
evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets. acceptance_length is mean tokens committed per verification step, including the bonus token. Its floor is 1.0 and its ceiling is 9.0 at block size 8.
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 5.534 | 87.1% | 76.4% | 66.8% | 59.1% | 51.0% | 43.9% | 37.6% | 31.4% |
| HumanEval | 3.842 | 77.1% | 59.5% | 45.0% | 33.7% | 25.6% | 19.2% | 14.1% | 9.9% |
| tool_call | 2.625 | 61.3% | 38.1% | 24.8% | 16.4% | 10.4% | 6.1% | 3.5% | 1.9% |
| writing | 2.572 | 59.0% | 35.2% | 21.9% | 14.8% | 10.5% | 7.2% | 5.1% | 3.5% |
| question | 2.496 | 57.3% | 33.7% | 20.9% | 14.0% | 9.7% | 6.5% | 4.5% | 3.1% |
| translation | 2.427 | 53.4% | 33.0% | 22.5% | 14.6% | 8.8% | 5.7% | 3.2% | 1.5% |
| rag | 2.311 | 58.1% | 32.6% | 17.9% | 11.2% | 6.3% | 3.0% | 1.3% | 0.7% |
| qa | 2.194 | 53.5% | 29.0% | 16.0% | 9.4% | 5.8% | 3.1% | 1.5% | 0.9% |
| summarization | 1.828 | 47.2% | 20.0% | 8.5% | 4.1% | 1.9% | 0.7% | 0.4% | 0.1% |
Weighted across all subsets: 2.885 over 114,104 verification steps.
Acceptance is highest where the verifier's next token is most predictable, and math_reasoning is in a class of its own. It leads HumanEval by nearly 1.7 tokens and still accepts more than half of pos_4 drafts (51.0%). Its last position (pos_7, 31.4%) sits above HumanEval's pos_4 (25.6%). math_reasoning is the one subset that makes real use of the full 8-token block. HumanEval is clearly second at 3.84, well ahead of everything else.
Throughput (tokens/s)
Mean output throughput in tokens/s on a single A100 at max concurrency 1. The average row gives the speedup over the verifier alone in parentheses. gemma-4-E2B-it-dflash2 is included for comparison.
| subset | baseline (no drafter) | MTP | DSpark (this model) | DFlash2 |
|---|---|---|---|---|
| math_reasoning | 167.6 | 520.2 | 626.5 | 626.8 |
| HumanEval | 164.9 | 478.5 | 399.3 | 515.5 |
| tool_call | 155.3 | 346.7 | 275.8 | 321.8 |
| writing | 166.7 | 265.2 | 268.3 | 270.2 |
| question | 167.3 | 266.8 | 272.2 | 276.8 |
| translation | 168.2 | 417.4 | 298.5 | 350.1 |
| rag | 153.3 | 270.8 | 216.4 | 243.6 |
| qa | 166.2 | 257.9 | 261.6 | 265.8 |
| summarization | 159.9 | 237.0 | 194.0 | 220.4 |
| average | 163.3 | 340.1 (2.08×) | 312.5 (1.91×) | 343.4 (2.10×) |
The throughput ranking follows acceptance length: math_reasoning and HumanEval see the largest gains, summarization the smallest. DSpark's best result is math_reasoning, where it is effectively tied with DFlash2 (626.5 vs 626.8 tokens/s, both 3.74× the baseline) and well ahead of MTP.
Comparison with MTP and DFlash2
Acceptance length on the same nine subsets for Gemma's MTP drafter (google/gemma-4-E2B-it-assistant) and the DFlash2 drafter. The three propose different numbers of tokens per step, so their ceilings differ (9.0 for MTP, 9.0 for DSpark, 8.0 for DFlash2). A longer draft also costs more per step, so acceptance length alone doesn't determine speed. See the throughput section below for that.
| subset | MTP (8 draft tokens) | DSpark (8 draft tokens) | DFlash2 (7 draft tokens) | best |
|---|---|---|---|---|
| math_reasoning | 4.867 | 5.534 | 5.602 | DFlash2 |
| HumanEval | 4.680 | 3.842 | 4.270 | MTP |
| tool_call | 3.208 | 2.625 | 2.943 | MTP |
| translation | 3.793 | 2.427 | 2.926 | MTP |
| rag | 3.164 | 2.311 | 2.821 | MTP |
| question | 2.780 | 2.496 | 2.696 | MTP |
| writing | 2.785 | 2.572 | 2.726 | MTP |
| qa | 2.360 | 2.194 | 2.305 | MTP |
| summarization | 2.418 | 1.828 | 2.126 | MTP |
| mean (unweighted) | 3.339 | 2.870 | 3.157 | MTP |
MTP has the highest acceptance length on 8 of 9 subsets and on average. DFlash2 leads on math_reasoning, narrowly ahead of DSpark (5.602 vs 5.534), and is ahead of DSpark on every other subset as well. math_reasoning is still where DSpark comes closest to the leader, and where it beats MTP by the widest margin. The mean here is unweighted across subsets, so DSpark's 2.870 differs slightly from its weighted 2.885 above.
Limitations
Works only with gemma-4-E2B-it and is not usable as a standalone model. Acceptance falls off steeply past the first two positions on prose-like traffic, most of all on summarization, so the gains concentrate in math and code. On prose-heavy workloads the tail of the 8-token block rarely gets accepted. Real-world speedup depends on your traffic mix. Because verification is lossless, the verifier's own behavior and biases carry through unchanged.
License
Apache 2.0, matching the verifier's license. The speculators training code is Apache-2.0.