⚡ Up to 5.1x faster inference with DSpark speculative decoding (71 --> 364 tokens/sec)
K2-Horizon-7B-DSpark
Model Link: https://huggingface.co/rasyosef/K2-Horizon-7B-DSpark
A lossless DSpark draft model for IFM/K2-Horizon-7B, trained with speculators (training code).
- Up to 5.11× faster on math and 3.01× on code, 3.00× on average (temperature 0)
- Mean acceptance length 3.37 (block size 8)
Usage
vllm serve IFM/K2-Horizon-7B --port 8000 \
--trust-remote-code \
--reasoning-parser k2_horizon \
--max-model-len 32768 \
--speculative-config '{"model": "rasyosef/K2-Horizon-7B-DSpark", "num_speculative_tokens": 8}' \
--override-generation-config '{"temperature": 0}'
For tool use, add --enable-auto-tool-choice --tool-call-parser k2_horizon.
Details
- Qwen3-style, 4 layers, ~0.9B params, bfloat16, block size 8
- Hidden size 4096, MLP 8192, 32 query / 8 KV heads (head dim 128)
- First 3 layers sliding-window (512), last full attention
- Draft vocab 32,000 (from 250,624); aux hidden states from verifier layers 2/10/18/26/34
- 100,000 Open PerfectBlend prompts with verifier-generated responses, seq length 8192
- 3 epochs, lr 3e-4, cosine (4% warmup), loss
{"ce": 0.1, "tv": 0.9}
Evaluation
RedHatAI/speculator_benchmarks, temperature 0, single A100, concurrency 1.
Throughput (tokens/s)
Mean output throughput. Baseline is K2-Horizon-7B without speculative decoding.
| subset | baseline (no drafter) | DSpark (this model) | speedup |
|---|---|---|---|
| math_reasoning | 71.4 | 364.5 | 5.11× |
| HumanEval | 71.0 | 213.5 | 3.01× |
| rag | 70.7 | 214.1 | 3.03× |
| qa | 70.8 | 217.6 | 3.07× |
| question | 71.2 | 176.9 | 2.48× |
| writing | 71.0 | 176.3 | 2.48× |
| translation | 71.2 | 190.9 | 2.68× |
| tool_call | 70.4 | 190.5 | 2.71× |
| summarization | 70.7 | 172.8 | 2.44× |
| average | 70.9 | 213.0 | 3.00× |
Acceptance at temperature 0
acceptance_length is mean tokens committed per verification step, including the bonus token (max 9.0 at block size 8).
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 5.350 | 87.8% | 75.6% | 65.4% | 55.8% | 47.7% | 40.5% | 34.0% | 28.1% |
| HumanEval | 3.625 | 78.1% | 57.3% | 41.4% | 29.7% | 21.4% | 15.5% | 11.1% | 7.9% |
| rag | 3.411 | 75.4% | 54.5% | 38.6% | 26.8% | 18.7% | 12.4% | 8.6% | 6.0% |
| qa | 3.326 | 70.9% | 49.9% | 35.8% | 25.6% | 18.8% | 14.0% | 10.3% | 7.2% |
| question | 3.316 | 69.8% | 48.4% | 34.0% | 25.1% | 19.2% | 14.6% | 11.5% | 8.8% |
| writing | 3.284 | 69.3% | 47.9% | 34.0% | 25.0% | 18.9% | 14.3% | 11.0% | 7.9% |
| translation | 3.050 | 68.6% | 47.3% | 32.1% | 21.8% | 14.8% | 9.9% | 6.4% | 4.1% |
| tool_call | 3.018 | 72.0% | 48.5% | 32.0% | 20.4% | 12.9% | 8.0% | 5.0% | 3.0% |
| summarization | 2.863 | 68.6% | 44.5% | 28.9% | 18.9% | 11.8% | 7.1% | 4.1% | 2.4% |
Weighted across all subsets: 3.370 over 298,078 verification steps.
Limitations
Only works with K2-Horizon-7B. Sampling, concurrent load, and contexts beyond 8K tokens were not evaluated.
License
Apache 2.0
Can you please benchmark this against our provided inference acceleration method via diffusion adapters called Uno?
Independent throughput benchmark on RTX PRO 6000 (Blackwell): Uno vs DSpark at batch 1 and 64
I benchmarked K2-Horizon-7B-Uno against rasyosef/K2-Horizon-7B-DSpark, a DSpark draft model I trained for the same base model. Sharing the results in case they're useful, and corrections to the setup are welcome.
Setup
- 1× NVIDIA RTX PRO 6000 Blackwell (96 GB, Colab G4), temperature 0, max 2,048 output tokens
- Prompts:
math_reasoning(80),HumanEval(164) andwriting(80) fromRedHatAI/speculator_benchmarks - Batch 1: first 10 prompts per subset, one request at a time, mean per-request tok/s
- Batch 64: all prompts, 64 requests in flight, total output tokens / wall time
- Uno: official
ifm-ai/unoengine, linear Ψ-Spec sampler, FA2. Baseline = same engine at block size 1 - DSpark: vLLM 0.33.0. Baseline = same vLLM server without the drafter
- Both engines load the same base revision (
1ad5223); Uno adapter revision669f041. Prompt token IDs are identical across engines, there were no preemptions, and output token totals are within ~4% between engines - Acceptance length = tokens committed per draft-verify step, bonus token included
Draft size 8 (Uno block size 8, DSpark num_speculative_tokens=8)
| subset | batch | Uno base | Uno | Uno speedup | Uno acc. len | vLLM base | DSpark | DSpark speedup | DSpark acc. len | DSpark ÷ Uno |
|---|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 1 | 74.2 | 148.8 | 2.01× | 4.34 | 80.0 | 397.3 | 4.97× | 5.79 | 2.67× |
| HumanEval | 1 | 73.9 | 121.8 | 1.65× | 3.73 | 80.5 | 238.8 | 2.97× | 3.37 | 1.96× |
| writing | 1 | 74.2 | 118.5 | 1.60× | 3.64 | 80.5 | 170.2 | 2.11× | 2.38 | 1.44× |
| math_reasoning | 64 | 1239.9 | 1850.0 | 1.49× | 4.65 | 1343.8 | 3940.3 | 2.93× | 5.35 | 2.13× |
| HumanEval | 64 | 2256.0 | 2487.5 | 1.10× | 3.90 | 2730.7 | 4436.1 | 1.62× | 3.62 | 1.78× |
| writing | 64 | 1963.8 | 2001.0 | 1.02× | 3.73 | 2325.8 | 3293.0 | 1.42× | 3.08 | 1.65× |
Draft size 4 (Uno block size 4, DSpark num_speculative_tokens=4)
| subset | batch | Uno base | Uno | Uno speedup | Uno acc. len | vLLM base | DSpark | DSpark speedup | DSpark acc. len | DSpark ÷ Uno |
|---|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 1 | 74.3 | 129.3 | 1.74× | 3.72 | 80.1 | 283.0 | 3.53× | 4.11 | 2.19× |
| HumanEval | 1 | 74.0 | 113.0 | 1.53× | 3.36 | 80.6 | 213.6 | 2.65× | 3.01 | 1.89× |
| writing | 1 | 74.3 | 108.2 | 1.46× | 3.29 | 80.6 | 162.7 | 2.02× | 2.28 | 1.50× |
| math_reasoning | 64 | 1242.0 | 1787.4 | 1.44× | 3.82 | 1432.7 | 3682.8 | 2.57× | 3.96 | 2.06× |
| HumanEval | 64 | 2256.3 | 2820.2 | 1.25× | 3.45 | 2699.8 | 5128.7 | 1.90× | 3.12 | 1.82× |
| writing | 64 | 1964.4 | 2448.6 | 1.25× | 3.40 | 2375.1 | 3741.0 | 1.58× | 2.77 | 1.53× |
Throughput in tok/s. Speedups are against each method's own engine baseline.
Observations
- As the paper suggests, the smaller block helps Uno under load (HumanEval 1.10× → 1.25×, writing 1.02× → 1.25×) and hurts it at batch 1. DSpark shows the same pattern.
- On HumanEval and writing, Uno has a higher acceptance length than DSpark. But each Uno step runs the full 7B model twice (draft + verify), so it commits only ~1.6–2.3 tokens per 7B forward pass, while DSpark runs the 7B model once per step plus a small 4-layer drafter. That seems to be the main reason for the throughput gap.
- Taking each method's best draft size per row, DSpark is 1.4–2.7× faster than Uno.
Limitations
- No tree sampler: it needs FA3 (Hopper-only) or the FA4 path, which I didn't try on this GPU, so batch-1 results likely understate Uno.
- Not an H200 / Hopper GPU, temperature 0 only, and batch 1 uses only 10 prompts per subset.
- Batch 64 is close to the subset sizes (80 prompts), so the tail of each run has a shrinking batch.
Notebook to reproduce: https://colab.research.google.com/drive/16HTgCUxsqnENiwnvRME9CzRPW4nPet8u
Thanks for releasing Uno and the code!