Instructions to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16 # Run inference directly in the terminal: llama cli -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16 # Run inference directly in the terminal: llama cli -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Use Docker
docker model run hf.co/AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
- Llama new
- LM Studio
- Jan
- Ollama
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with Ollama:
ollama run hf.co/AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
- Unsloth Desktop
- Pi
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with Docker Model Runner:
docker model run hf.co/AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
- Lemonade
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Run and chat with the model
lemonade run user.Qwen3.8-27B-AX8860-llama.cpp-BF16
List all available models
lemonade list
- Hermes Agent
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-27B on AX8860 (llama.cpp + ggml-axera)
ๅจ็ฑ่ฏ AX8860 ็ NPU ไธ็จ llama.cpp ่ฟ่ก Qwen3.8-27B๏ผๆๆฌ + ๅพๅ่พๅ ฅ๏ผๆ้ฟ 256k ไธไธๆ๏ผๅฏ้ MTP ๆๆบ่งฃ็ ๏ผOpenAI ๅ ผๅฎนๆๅกใ Qwen3.8-27B on the Axera AX8860 NPU with llama.cpp: text + image input, up to 256k context, optional MTP speculative decoding, OpenAI-compatible server.
ๆง่ฝ / Performance
| decode | 13.8 tokens/s |
| decode๏ผMTP ๆๆบ่งฃ็ / with MTP | ๆ้ซ / up to 35.1 tokens/s๏ผ2.52x๏ผ |
| 4k prompt๏ผMTP / MTP on 4k | 13.5 โ 20.8 tokens/s๏ผ1.54x๏ผaccepted 32/90๏ผ |
| 4k prompt prefill | 1262 tokens/s๏ผ3908 token 3.10 s๏ผ |
| ไธไธๆ / context | ๆ้ฟ / up to 262144 |
MTP๏ผMTP=1๏ผๆๅ
ๅฎน / by content:
| ๅ ๅฎน / content | ๆฎ้ / plain | MTP | ๅ ้ / speed-up | ๆฅๅ็ / accept |
|---|---|---|---|---|
| ไปฃ็ / code | 13.9 | 35.1 | 2.52x | 0.785 |
| ไธญ่ฏ่ฑ / translation | 13.7 | 28.6 | 2.09x | 0.596 |
| Markdown ่กจๆ ผ / table | 13.8 | 31.0 | 2.25x | 0.687 |
| ๆฐๅญฆ่งฃ้ข / maths | 13.8 | 32.0 | 2.32x | 0.709 |
| ไธญๆ็ญๆ / Chinese essay | 13.7 | 20.6 | 1.51x | 0.342 |
| ่ฑๆๆ ไบ / English story | 13.7 | 20.4 | 1.49x | 0.338 |
| ไปฃ็ ๏ผๆจ่้ๆ ท / code, sampled | 13.7 | 29.0 | 2.12x | 0.618 |
| ไธญๆ็ญๆ๏ผๆจ่้ๆ ท / essay, sampled | 13.7 | 20.0 | 1.46x | 0.322 |
ๅไฝ tokens/s๏ผๆฏ้กนๆๅค็ๆ 256 token๏ผๆชๆณจๆ็ไธบ่ดชๅฟ่งฃ็ / tokens/s; up to 256 tokens each, greedy unless noted.
sh bench.sh ๅค็ฐ / reproduce๏ผ็ๆ 128 token / 128 generated tokens๏ผ็ญ prompt ไป
ๅ่ๆถ / short prompts report time only๏ผ๏ผ
| prompt | -c |
prompt ่ๆถ / time | decode tokens/s |
|---|---|---|---|
| 5 token | 8192 | 0.22 s | โ |
| 47 token | 8192 | 0.22 s | โ |
| 3908 token | 8192 | 3.10 s | 13.6 |
| 5 token | 16384 | 0.21 s | โ |
| 47 token | 16384 | 0.22 s | โ |
| 3908 token | 16384 | 3.09 s | 13.6 |
| 3908 token๏ผๆจ่้ๆ ท / sampled | 8192 | 3.09 s | 13.3 |
ๅฟซ้ๅผๅง / Quick start
sh run.sh "The capital of France is" 64 # ๆๆฌ / text
sh run.sh -i bus.jpg "Describe this image." 256 # ๅพๅ / image
sh run.sh -c # ๅฝไปค่กๅฏน่ฏ / chat
sh run.sh -s 8080 # OpenAI ๅ
ผๅฎนๆๅก / server
MTP=1 sh run.sh -s 8080 # MTP ๆๆบ่งฃ็ / speculative decoding
CTX=131072 sh run.sh -s 8080 # ้ฟไธไธๆ / long context (default 16384)
sh bench.sh # ๆง่ฝๅค็ฐ / benchmark
้่ฆ AX8860๏ผSDK V0.5.1 ๆๆดๆฐ๏ผHF ไธ่ฝฝ็ๆไปถๆฒกๆๆง่กๆ้๏ผ็จ sh ่ฐ็จๅณๅฏ / needs an AX8860 with SDK V0.5.1+; call the scripts with sh.
ๆไปถ / Files
| ๆไปถ / File | ่ฏดๆ / Description |
|---|---|
qwen3.8-27b-mxfp4.gguf |
ๆจกๅๆ้ / weights๏ผ14.8 GB๏ผmd5 32f5a603d4f75edb16ccf7ff3a35bdd6 |
mmproj-qwen3.8-27b-bf16.gguf |
่ง่ง็ผ็ ๅจ / vision encoder๏ผ931 MB๏ผmd5 d2b989d06a0eb9109bab182ec2ff03f2 |
bin/ |
llama.cpp ๆฟ็ซฏ็จๅบ๏ผ้ๆ้พๆฅ๏ผ/ board programs (static) |
kernels/ |
NPU kernel + manifest.json |
run.sh๏ผchat_template.jinja |
่ฟ่ก่ๆฌไธๆๅก็ซฏๅฏน่ฏๆจกๆฟ / run script and the server's chat template |
bench.sh ๅ prompt |
ๆง่ฝๅค็ฐ / benchmark |
่ฏดๆ / Notes
- MTP ไธบๅฏ้๏ผ
MTP=1 ./run.sh -sๆพๅผๅผๅฏ๏ผๆๅก็ซฏไธๅ็จ่ช้ๅบ้จๆงๆๅฎๅ ณๆ๏ผไธ่ฎพ็ฝฎMTPๆถไฟๆๆฎ้่งฃ็ ใ่ดชๅฟ่ฏทๆฑ่ฏทๆพๅผๅธฆpresence_penalty: 0/ MTP is opt-in:MTP=1 ./run.sh -senables it explicitly and bypasses the adaptive gate; withoutMTPthe server stays non-speculative. Greedy requests should setpresence_penalty: 0. - ๆๅก็ซฏ้ป่ฎคไฝฟ็จๆจกๅๅกๆจ่็้ๆ ทๅๆฐ๏ผไฝ presence penalty ้ป่ฎคไธบ 0๏ผๆจกๅๅก็ 1.5 ไผ่ฎฉๆจกๅๅ้ฟๅค่ฟฐๆ็คบ้ๅบ็ฐ่ฟ็ๅ
ๅฎน๏ผๅฆๆไปถๅใๆฐๅญใไปฃ็ ๏ผๅฝฑๅ agent ไธๅทฅๅ
ท่ฐ็จ๏ผ๏ผ
PRESENCE_PENALTY=1.5 ./run.sh -sๅฏๆขๅค๏ผ่ฏทๆฑ้่ชๅธฆ็presence_penaltyไผๅ ใๆๆฌ่กฅๅ จไธๅพๅๆ่ฟฐไธบ่ดชๅฟ่งฃ็ / the server uses the model card's recommended sampling, except that the presence penalty defaults to 0 (the card's 1.5 makes the model avoid repeating text from the prompt such as file names, numbers and code, which hurts agents and tool calls);PRESENCE_PENALTY=1.5 ./run.sh -srestores it, and a request's ownpresence_penaltywins. Completion and image description are greedy. - ๆๅก็ซฏ้ป่ฎคๅ
ณ้ญๆ่๏ผ
--reasoning off๏ผ๏ผๅนถๅ ่ฎธๅฏน่ฏไธญๆฎต็ system ๆถๆฏ๏ผCodex ็ญ agent ๅฎขๆท็ซฏๅฏ็จ๏ผ/ the server does not think by default and accepts a mid-conversation system message (works with agent clients such as Codex). - Prompt cache ้ป่ฎคๆพๅจ 4 GB ๅ
ๅญ๏ผ่ฎพ็ฝฎ
CACHE_DIR=<SSD ็ฎๅฝ>ๅฏ่ฝ็๏ผ้ๅฏๅไป่ฝๅฝไธญ / the prompt cache lives in 4 GB of host memory; setCACHE_DIR=<an SSD directory>to keep it on disk across restarts. - ่ฟ่กๆถ็ฌๅ NPU 4 ๆ ธ๏ผๆๅก็ซฏ
-np 1/ takes all 4 NPU cores; the server runs-np 1. llama-benchๅฏ็จ๏ผ่ฏทๅ-b 4096 -ub 4096/llama-benchworks; pass-b 4096 -ub 4096.- ไธชๅซ็ฏๅขไธ่ฟ็ปญ่ฏทๆฑๅถๅ
ret = -3๏ผ้่ฏๅณๅฏ / in rare cases a back-to-back request fails withret = -3; retry it. - MTP ๆฏ่ฝฎ็ 3 ไธช token๏ผ็ฑ็ฎๆ ๆจกๅ้ไธช้ช่ฏ๏ผ่ดชๅฟๆถ่พๅบไธๆฎ้่งฃ็ ไธ่ด / MTP drafts 3 tokens per round, each verified by the target model; greedy output equals plain decoding.
- ้ฟไธไธๆไฝฟ็จๆปๅจ checkpoint๏ผ้ป่ฎค
-ctxcp 4๏ผ๏ผๅ ๅญไธ้ไธไธๆ็บฟๆงๅข้ฟ / long context uses rolling checkpoints (default-ctxcp 4); memory does not grow linearly with context.
ๆญฃ็กฎๆง / Correctness
- "The capital of France is" ็่ดชๅฟ่พๅบไธไธไธ็้ๅญ่็ธๅ๏ผๅพๅๆ่ฟฐๆญฃ็กฎ / the greedy output of "The capital of France is" is byte-identical to the previous release; image descriptions correct.
- MTP ้ป่ฎคๅ
ณ้ญ๏ผ
MTP=1ๅผๅฏ๏ผ๏ผๅผๅฏๅ่พๅบ็ฑ็ฎๆ ๆจกๅ้ token ้ช่ฏ๏ผไธๆฎ้่ดชๅฟ่งฃ็ ไธ่ด / MTP is off by default (MTP=1enables it); every draft is verified by the target model and greedy output equals plain decoding. r3client.py growๅฐ โฅ60k prompt token ไธ OOM๏ผcheckpoint ๅ ๅญๅนณ็จณ /r3client.py growreaches โฅ60k prompt tokens without OOM and checkpoint memory stays flat.
ๆบ็ / Sources
- llama.cpp๏ผZHEQIUSHUI/llama.cpp
axera-dev@0104880be
ๆดๆฐ / Changelog
- 2026-11-11๏ผๆๅก็ซฏๆชๆๅฎๆจกๆฟๆถ่ชๅจไฝฟ็จๅ
ๅ
chat_template.jinja๏ผๆๅจๅฏๅจ็ๆๅกไธๅ่ฎฉ agent ๅฎขๆท็ซฏๆถๅฐ HTTP 500๏ผ๏ผๆๅกๆจกๅผ presence penalty ้ป่ฎคๆนไธบ 0 / the server uses the package'schat_template.jinjawhen no template is given (a hand-started server no longer answers agent clients with HTTP 500); the server's default presence penalty is now 0. - 2026-11-09๏ผ็ปไธๆฟ็ซฏ bin๏ผMTP ้ช่ฏ้ป่พไฟฎๅค๏ผ่ดชๅฟๅบๆฏๆพๅผ
presence_penalty: 0๏ผperplexity ๅบๅฎ่ตฐๅ จ logits / unified board bins; MTP verification fixes; explicitpresence_penalty: 0in greedy scenes; perplexity keeps the full-logits head.
- Downloads last month
- 2,298
4-bit
Model tree for AXERA-TECH/Qwen3.8-27B-AX8860-llama.cpp
Base model
Qwen/Qwen3.8-27B