Qwen3.8-27B on AX8860 (llama.cpp + ggml-axera)

ๅœจ็ˆฑ่Šฏ AX8860 ็š„ NPU ไธŠ็”จ llama.cpp ่ฟ่กŒ Qwen3.8-27B๏ผšๆ–‡ๆœฌ + ๅ›พๅƒ่พ“ๅ…ฅ๏ผŒๆœ€้•ฟ 256k ไธŠไธ‹ๆ–‡๏ผŒๅฏ้€‰ MTP ๆŠ•ๆœบ่งฃ็ ๏ผŒOpenAI ๅ…ผๅฎนๆœๅŠกใ€‚ Qwen3.8-27B on the Axera AX8860 NPU with llama.cpp: text + image input, up to 256k context, optional MTP speculative decoding, OpenAI-compatible server.

ๆ€ง่ƒฝ / Performance

decode 13.8 tokens/s
decode๏ผŒMTP ๆŠ•ๆœบ่งฃ็  / with MTP ๆœ€้ซ˜ / up to 35.1 tokens/s๏ผˆ2.52x๏ผ‰
4k prompt๏ผŒMTP / MTP on 4k 13.5 โ†’ 20.8 tokens/s๏ผˆ1.54x๏ผŒaccepted 32/90๏ผ‰
4k prompt prefill 1262 tokens/s๏ผˆ3908 token 3.10 s๏ผ‰
ไธŠไธ‹ๆ–‡ / context ๆœ€้•ฟ / up to 262144

MTP๏ผˆMTP=1๏ผ‰ๆŒ‰ๅ†…ๅฎน / by content:

ๅ†…ๅฎน / content ๆ™ฎ้€š / plain MTP ๅŠ ้€Ÿ / speed-up ๆŽฅๅ—็އ / accept
ไปฃ็  / code 13.9 35.1 2.52x 0.785
ไธญ่ฏ‘่‹ฑ / translation 13.7 28.6 2.09x 0.596
Markdown ่กจๆ ผ / table 13.8 31.0 2.25x 0.687
ๆ•ฐๅญฆ่งฃ้ข˜ / maths 13.8 32.0 2.32x 0.709
ไธญๆ–‡็Ÿญๆ–‡ / Chinese essay 13.7 20.6 1.51x 0.342
่‹ฑๆ–‡ๆ•…ไบ‹ / English story 13.7 20.4 1.49x 0.338
ไปฃ็ ๏ผŒๆŽจ่้‡‡ๆ ท / code, sampled 13.7 29.0 2.12x 0.618
ไธญๆ–‡็Ÿญๆ–‡๏ผŒๆŽจ่้‡‡ๆ ท / essay, sampled 13.7 20.0 1.46x 0.322

ๅ•ไฝ tokens/s๏ผ›ๆฏ้กนๆœ€ๅคš็”Ÿๆˆ 256 token๏ผŒๆœชๆณจๆ˜Ž็š„ไธบ่ดชๅฟƒ่งฃ็  / tokens/s; up to 256 tokens each, greedy unless noted.

sh bench.sh ๅค็Žฐ / reproduce๏ผˆ็”Ÿๆˆ 128 token / 128 generated tokens๏ผ›็Ÿญ prompt ไป…ๅˆ—่€—ๆ—ถ / short prompts report time only๏ผ‰๏ผš

prompt -c prompt ่€—ๆ—ถ / time decode tokens/s
5 token 8192 0.22 s โ€”
47 token 8192 0.22 s โ€”
3908 token 8192 3.10 s 13.6
5 token 16384 0.21 s โ€”
47 token 16384 0.22 s โ€”
3908 token 16384 3.09 s 13.6
3908 token๏ผŒๆŽจ่้‡‡ๆ ท / sampled 8192 3.09 s 13.3

ๅฟซ้€Ÿๅผ€ๅง‹ / Quick start

sh run.sh "The capital of France is" 64           # ๆ–‡ๆœฌ / text
sh run.sh -i bus.jpg "Describe this image." 256   # ๅ›พๅƒ / image
sh run.sh -c                                      # ๅ‘ฝไปค่กŒๅฏน่ฏ / chat
sh run.sh -s 8080                                 # OpenAI ๅ…ผๅฎนๆœๅŠก / server
MTP=1 sh run.sh -s 8080                           # MTP ๆŠ•ๆœบ่งฃ็  / speculative decoding
CTX=131072 sh run.sh -s 8080                      # ้•ฟไธŠไธ‹ๆ–‡ / long context (default 16384)
sh bench.sh                                       # ๆ€ง่ƒฝๅค็Žฐ / benchmark

้œ€่ฆ AX8860๏ผŒSDK V0.5.1 ๆˆ–ๆ›ดๆ–ฐ๏ผ›HF ไธ‹่ฝฝ็š„ๆ–‡ไปถๆฒกๆœ‰ๆ‰ง่กŒๆƒ้™๏ผŒ็”จ sh ่ฐƒ็”จๅณๅฏ / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.

ๆ–‡ไปถ / Files

ๆ–‡ไปถ / File ่ฏดๆ˜Ž / Description
qwen3.8-27b-mxfp4.gguf ๆจกๅž‹ๆƒ้‡ / weights๏ผŒ14.8 GB๏ผŒmd5 32f5a603d4f75edb16ccf7ff3a35bdd6
mmproj-qwen3.8-27b-bf16.gguf ่ง†่ง‰็ผ–็ ๅ™จ / vision encoder๏ผŒ931 MB๏ผŒmd5 d2b989d06a0eb9109bab182ec2ff03f2
bin/ llama.cpp ๆฟ็ซฏ็จ‹ๅบ๏ผˆ้™ๆ€้“พๆŽฅ๏ผ‰/ board programs (static)
kernels/ NPU kernel + manifest.json
run.sh๏ผŒchat_template.jinja ่ฟ่กŒ่„šๆœฌไธŽๆœๅŠก็ซฏๅฏน่ฏๆจกๆฟ / run script and the server's chat template
bench.sh ๅŠ prompt ๆ€ง่ƒฝๅค็Žฐ / benchmark

่ฏดๆ˜Ž / Notes

  • MTP ไธบๅฏ้€‰๏ผšMTP=1 ./run.sh -s ๆ˜พๅผๅผ€ๅฏ๏ผŒๆœๅŠก็ซฏไธๅ†็”จ่‡ช้€‚ๅบ”้—จๆŽงๆŠŠๅฎƒๅ…ณๆމ๏ผ›ไธ่ฎพ็ฝฎ MTP ๆ—ถไฟๆŒๆ™ฎ้€š่งฃ็ ใ€‚่ดชๅฟƒ่ฏทๆฑ‚่ฏทๆ˜พๅผๅธฆ presence_penalty: 0 / MTP is opt-in: MTP=1 ./run.sh -s enables it explicitly and bypasses the adaptive gate; without MTP the server stays non-speculative. Greedy requests should set presence_penalty: 0.
  • ๆœๅŠก็ซฏ้ป˜่ฎคไฝฟ็”จๆจกๅž‹ๅกๆŽจ่็š„้‡‡ๆ ทๅ‚ๆ•ฐ๏ผŒไฝ† presence penalty ้ป˜่ฎคไธบ 0๏ผˆๆจกๅž‹ๅก็š„ 1.5 ไผš่ฎฉๆจกๅž‹ๅ›ž้ฟๅค่ฟฐๆ็คบ้‡Œๅ‡บ็Žฐ่ฟ‡็š„ๅ†…ๅฎน๏ผŒๅฆ‚ๆ–‡ไปถๅใ€ๆ•ฐๅญ—ใ€ไปฃ็ ๏ผŒๅฝฑๅ“ agent ไธŽๅทฅๅ…ท่ฐƒ็”จ๏ผ‰๏ผ›PRESENCE_PENALTY=1.5 ./run.sh -s ๅฏๆขๅค๏ผŒ่ฏทๆฑ‚้‡Œ่‡ชๅธฆ็š„ presence_penalty ไผ˜ๅ…ˆใ€‚ๆ–‡ๆœฌ่กฅๅ…จไธŽๅ›พๅƒๆ่ฟฐไธบ่ดชๅฟƒ่งฃ็  / the server uses the model card's recommended sampling, except that the presence penalty defaults to 0 (the card's 1.5 makes the model avoid repeating text from the prompt such as file names, numbers and code, which hurts agents and tool calls); PRESENCE_PENALTY=1.5 ./run.sh -s restores it, and a request's own presence_penalty wins. Completion and image description are greedy.
  • ๆœๅŠก็ซฏ้ป˜่ฎคๅ…ณ้—ญๆ€่€ƒ๏ผˆ--reasoning off๏ผ‰๏ผŒๅนถๅ…่ฎธๅฏน่ฏไธญๆฎต็š„ system ๆถˆๆฏ๏ผˆCodex ็ญ‰ agent ๅฎขๆˆท็ซฏๅฏ็”จ๏ผ‰/ the server does not think by default and accepts a mid-conversation system message (works with agent clients such as Codex).
  • Prompt cache ้ป˜่ฎคๆ”พๅœจ 4 GB ๅ†…ๅญ˜๏ผ›่ฎพ็ฝฎ CACHE_DIR=<SSD ็›ฎๅฝ•> ๅฏ่ฝ็›˜๏ผŒ้‡ๅฏๅŽไป่ƒฝๅ‘ฝไธญ / the prompt cache lives in 4 GB of host memory; set CACHE_DIR=<an SSD directory> to keep it on disk across restarts.
  • ่ฟ่กŒๆ—ถ็‹ฌๅ  NPU 4 ๆ ธ๏ผ›ๆœๅŠก็ซฏ -np 1 / takes all 4 NPU cores; the server runs -np 1.
  • llama-bench ๅฏ็”จ๏ผŒ่ฏทๅŠ  -b 4096 -ub 4096 / llama-bench works; pass -b 4096 -ub 4096.
  • ไธชๅˆซ็Žฏๅขƒไธ‹่ฟž็ปญ่ฏทๆฑ‚ๅถๅ‘ ret = -3๏ผŒ้‡่ฏ•ๅณๅฏ / in rare cases a back-to-back request fails with ret = -3; retry it.
  • MTP ๆฏ่ฝฎ็Œœ 3 ไธช token๏ผŒ็”ฑ็›ฎๆ ‡ๆจกๅž‹้€ไธช้ชŒ่ฏ๏ผ›่ดชๅฟƒๆ—ถ่พ“ๅ‡บไธŽๆ™ฎ้€š่งฃ็ ไธ€่‡ด / MTP drafts 3 tokens per round, each verified by the target model; greedy output equals plain decoding.
  • ้•ฟไธŠไธ‹ๆ–‡ไฝฟ็”จๆปšๅŠจ checkpoint๏ผˆ้ป˜่ฎค -ctxcp 4๏ผ‰๏ผŒๅ†…ๅญ˜ไธ้šไธŠไธ‹ๆ–‡็บฟๆ€งๅขž้•ฟ / long context uses rolling checkpoints (default -ctxcp 4); memory does not grow linearly with context.

ๆญฃ็กฎๆ€ง / Correctness

  • "The capital of France is" ็š„่ดชๅฟƒ่พ“ๅ‡บไธŽไธŠไธ€็‰ˆ้€ๅญ—่Š‚็›ธๅŒ๏ผ›ๅ›พๅƒๆ่ฟฐๆญฃ็กฎ / the greedy output of "The capital of France is" is byte-identical to the previous release; image descriptions correct.
  • MTP ้ป˜่ฎคๅ…ณ้—ญ๏ผˆMTP=1 ๅผ€ๅฏ๏ผ‰๏ผ›ๅผ€ๅฏๅŽ่พ“ๅ‡บ็”ฑ็›ฎๆ ‡ๆจกๅž‹้€ token ้ชŒ่ฏ๏ผŒไธŽๆ™ฎ้€š่ดชๅฟƒ่งฃ็ ไธ€่‡ด / MTP is off by default (MTP=1 enables it); every draft is verified by the target model and greedy output equals plain decoding.
  • r3client.py grow ๅˆฐ โ‰ฅ60k prompt token ไธ OOM๏ผŒcheckpoint ๅ†…ๅญ˜ๅนณ็จณ / r3client.py grow reaches โ‰ฅ60k prompt tokens without OOM and checkpoint memory stays flat.

ๆบ็  / Sources

ๆ›ดๆ–ฐ / Changelog

  • 2026-11-11๏ผšๆœๅŠก็ซฏๆœชๆŒ‡ๅฎšๆจกๆฟๆ—ถ่‡ชๅŠจไฝฟ็”จๅŒ…ๅ†… chat_template.jinja๏ผˆๆ‰‹ๅŠจๅฏๅŠจ็š„ๆœๅŠกไธๅ†่ฎฉ agent ๅฎขๆˆท็ซฏๆ”ถๅˆฐ HTTP 500๏ผ‰๏ผ›ๆœๅŠกๆจกๅผ presence penalty ้ป˜่ฎคๆ”นไธบ 0 / the server uses the package's chat_template.jinja when no template is given (a hand-started server no longer answers agent clients with HTTP 500); the server's default presence penalty is now 0.
  • 2026-11-09๏ผš็ปŸไธ€ๆฟ็ซฏ bin๏ผ›MTP ้ชŒ่ฏ้€ป่พ‘ไฟฎๅค๏ผ›่ดชๅฟƒๅœบๆ™ฏๆ˜พๅผ presence_penalty: 0๏ผ›perplexity ๅ›บๅฎš่ตฐๅ…จ logits / unified board bins; MTP verification fixes; explicit presence_penalty: 0 in greedy scenes; perplexity keeps the full-logits head.
Downloads last month
2,298
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp

Base model

Qwen/Qwen3.8-27B
Quantized
(1,858)
this model

Collection including AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp