Qwen3.5-4B-graft8

The fast end of a graft of Qwen/Qwen3.5-4B: the model is cut at layer 8 of 32, and only that first quarter reads your prompt. The other 24 layers never see it — they get a memory instead, the residual stream the encoder produced, mapped into each upper attention layer's own keys and values by a small identity-initialised adapter. Generated tokens still pass through all 32 layers, so decoding costs what the original costs.

Time-to-first-token is 3.1-3.7x faster from 16K tokens up. Do not give this model tools. Both halves of that sentence are measured below.

Prompt	Qwen3.5-4B TTFT	graft8 TTFT	Speedup	Prompt tok/s	Decode tok/s	Peak memory
4K	0.45 s	0.22 s	2.08x	9,048 -> 18,842	26.3 -> 25.7	8.59 -> 8.63 GB
16K	1.90 s	0.61 s	3.13x	8,608 -> 26,923	25.4 -> 25.5	10.39 -> 9.94 GB
32K	4.19 s	1.24 s	3.39x	7,818 -> 26,527	24.2 -> 23.7	12.89 -> 11.99 GB
64K	9.96 s	2.77 s	3.60x	6,581 -> 23,682	21.7 -> 21.6	17.89 -> 16.08 GB
128K	26.24 s	7.04 s	3.73x	4,996 -> 18,608	17.9 -> 17.5	27.88 -> 24.26 GB

A40 48 GB, bf16, flash-attention-2, batch 1, greedy, 7 timed reps after 2 warmups per length; median reported (min and p95 within 1% of it). TTFT is prefill plus the first sampled token; decode is the steady-state rate over the next 31 tokens. bench_ttft.py, included here, reproduces the table.

This is one instance of a general operation we call model grafting — cutting a trained model, changing how information moves through it, and healing the result with a small amount of continued training. The method, the cost law that predicts how much healing a change needs, and the speed-versus-capability frontier this checkpoint sits at the fast end of are written up here:

→ Model grafting

What it is for, and what it is not for

For: reading and summarising long documents, question answering over a long context, and anything where the wait is dominated by the model reading what you wrote. Knowledge, instruction following and arithmetic survive the cut almost intact.

Not for: tool calling or agent loops. Function-call accuracy falls apart, and chained retrieval through a long context fails completely. If your workload calls tools, use the 16-of-32 cut instead, which keeps tool calling within 3 points of the parent for a 1.95x speedup.

Results

Same harness and prompts for both models: MMLU 5-shot (50 per subject), GSM8K 5-shot (400), IFEval (541 prompts), HumanEval-instruct, BFCL v4 non-live AST (100 per category, official bfcl_eval checker), HashHop (Magic's generator, 25 instances per hop count over 1-4 hops).

Benchmark	Qwen3.5-4B	graft8	graft16 (for reference)
MMLU	0.713	0.703	0.709
GSM8K	0.915	0.880	0.907
IFEval (prompt-level strict)	0.815	0.802	0.800
HumanEval	0.829	0.768	0.799
BFCL (non-live AST mean)	0.848	0.658	0.820
HashHop (~8K prompt)	0.290	0.000	0.190
Mean of six	0.735	0.635	0.704
HashHop (~33K prompt)	0.100	0.000	0.010

The two failures in detail:

• Function calls. BFCL by category (parent / graft8): simple 0.89/0.81, multiple 0.94/0.84, parallel 0.73/0.50, parallel-multiple 0.83/0.48, irrelevance 0.91/0.90. A synthetic probe that puts eight tool schemas in a long prompt and asks for one by index tells the same story more sharply: the graft picks the right tool 82% of the time but gets the whole call right only 44% of the time (parent: 98% and 96%). It is an argument-copying failure, not a tool-choice failure. Note that irrelevance is untouched — it still knows when not to call.
• Multi-hop retrieval: zero. Not "degraded" — the model does not follow a single hash to its value at any prompt length tested. This is the cost of letting only 2 of the model's 8 full-attention layers read the prompt.

Trained on 150M tokens, not 500M: on the 16-of-32 cut, going from 100M to 500M tokens moved the six-benchmark mean by 0.7 points, and this checkpoint's own 100M and 150M gates agree within noise (0.439 vs 0.444 on the probe). The plateau arrives early; more tokens is not the lever that fixes the two failures above.

How it was trained

• Objective: self-distillation only against the unmodified Qwen3.5-4B's next-token distribution (top-64 KL), not the data's labels — that is what preserves instruction following, chat format and thinking mode.
• Data per sequence: 45% agent trajectories (UltraData-SFT-Agent-2609, rendered with Qwen3.5's own chat template), 15% code (UltraData-Code L2, quality >= 5, HumanEval-decontaminated), 40% FineWeb-Edu; sequence lengths 4K/8K/16K mixed 20/40/40.
• What trains: the 8 encoder layers, the six memory-site attention modules and the adapters — 1.15B of 4.23B parameters. Everything above the cut is frozen, which both prevents capability erosion and cuts optimizer memory.
• Adapters: one identity-initialised 2560x2560 linear map per memory site (layers 11, 15, 19, 23, 27, 31), 39M parameters. The adapter maps memory into the host layer's input space and lets that layer's own input_layernorm, k_proj, k_norm, v_proj and RoPE build the prefix keys and values.
• Tail window 256: the last 256 prompt tokens go through the full decoder path rather than through memory, without which the upper recurrent layers have no state over the question.
• Hyperparameters: AdamW8bit, lr 5e-6 (pretrained) / 2e-5 (adapters) cosine to 10%, batch 262144 tokens, grad-checkpointing, bf16 frozen weights with fp32 masters, ~2,090 tok/s, 16.5 h on one A40.

Loading it

Not a drop-in transformers model. The loader, modeling_ced.py, ships in this repo, and the weights file holds only the parameters that were trained — the trainable_only marker means the frozen ones come from Qwen/Qwen3.5-4B (recorded in ced_config.json), so the base model must be available too.

from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
import sys

d = snapshot_download("LocalLLaMA/Qwen3.5-4B-graft8")
sys.path.insert(0, d)                            # modeling_ced.py ships with the weights
from modeling_ced import CEDForCausalLM

model = CEDForCausalLM.load(d)                   # frozen weights come from Qwen/Qwen3.5-4B
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")

prompt = tok.apply_chat_template([{"role": "user", "content": "..."}], tokenize=False,
                                 add_generation_prompt=True, enable_thinking=False)
ids = tok(prompt, return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=256, eos_id=tok.convert_tokens_to_ids("<|im_end|>"))
print(tok.decode(out[0], skip_special_tokens=True))

generate runs the encoder over the prompt once, builds the memory, then decodes through the full stack. model.cfg.tail (256) is the suffix that bypasses memory.

Files

File	What it is
ced.pt	trained parameters only, bf16
ced_config.json	base model, cut depth, tail, adapter type, memory sites
trainable_only	marker: frozen parameters come from the base model
log.jsonl	the full training log, including the fixed-batch KL probes
meta.json	the recipe the run was launched with
modeling_ced.py	the loader and the graft itself (encoder, adapters, prefix-KV attention, prefill/decode)
bench_ttft.py	the speed benchmark that produced the table above
results/*.json	raw benchmark and timing output for this checkpoint and its parent

Limits

• One cut depth at one scale. The 16-of-32 cut is the other measured point: 1.95x prefill at 128K for 3.1 points of six-benchmark mean, against 3.73x for 10 points here.
• Sample sizes are capped (50 MMLU per subject, 400 GSM8K, 100 BFCL per category, 25 HashHop per hop), so treat differences under about a point as noise. The two large failures above are far outside that.
• The KV cache is not smaller: the encoder's own attention keys and values are not compressed, so peak memory improves only 5-13%.
• Inherits Qwen3.5-4B's licence and limitations. The parent's vision tower is untouched and untested; everything here was measured text-only.