Your 4-bit model didn't forget how to call tools. Your scorer did.
Here is a result that would have gone into a model catalog as fact: after Q4_K_M quantization, Mistral-Nemo-Instruct-2407's pass rate on bare tool calls fell from 1.000 to 0.000. Read literally, a function-calling model could no longer call functions.
It wasn't true. The quantized model named the right tool every time. It wrapped its arguments as params instead of args and added one stray closing brace:
{"tool": "add", "params": {"a": 5, "b": 10}}}
The arguments were correct. The scorer couldn't read them.
Two kinds of failure
A behavioral evaluation produces two kinds of failure, and they need to be told apart before anyone publishes a number:
- Genuine degradation: the model really does worse. It picks the wrong answer, loses state, or computes the wrong value.
- Harness artifact: the model's behavior is fine, but its output drifted in format (a renamed key, a wrapper, a trailing character) and the scorer missed it.
Quantization changes output format as well as output quality, so evaluations of quantized models are where artifacts pile up. quant_eval's central rule is to classify every scored failure as one or the other, and to require evidence before calling a failure genuine.
How the harness measures behavior
quant_eval runs the full-weight model and its quantized version on the same seeded, hash-pinned fixtures, under the same decoding, and reports the difference per capability, not as one score. Version 7.21 covers eight capability families: single-step JSON planning, multi-step planning with a self-check, multiple choice, structured output mixed with prose, two-turn state tracking, tool calls with a computed answer, bare tool dispatch, and a randomized planning regression. The "correct" answer for the planning families is computed by an oracle, not by a reference model.
It runs the same pipeline across 16-bit Transformers, GGUF via llama.cpp, NormalFloat-4 via bitsandbytes, and W4A16 compressed-tensors via vLLM. The harness itself is private; the paper describes its design in full.
The worked example
Mistral-Nemo-Instruct-2407, FP16 against Q4_K_M, on an RTX 4090, seed 42. These runs used the quick screening profile, so families have between two and twenty cases. Read the rates as screening results, not production verdicts; the paper says so explicitly.
| Family | Metric | FP16 | Q4_K_M | Verdict |
|---|---|---|---|---|
json |
avg bucket | 10.000 | 10.000 | Preserved |
fuzz |
avg bucket | 10.000 | 10.000 | Preserved |
stateful_followup |
pass rate | 1.000 | 1.000 | Preserved |
mixed_brief_json |
pass rate | 1.000 | 1.000 | Preserved |
toolcall |
pass rate (stage 1) | 1.000 | 1.000 | Call format preserved |
toolcall |
avg bucket | 11.000 | 5.500 | Final value degraded |
json_multistep |
pass rate | 0.600 | 0.400 | Genuine degradation |
mcq |
avg bucket (of 1) | 0.600 | 0.400 | Genuine degradation |
toolcall_only |
pass rate | 1.000 | 0.000 | Harness artifact |
Three losses are genuine. Multi-step planning lost one hard case entirely, multiple choice flipped one answer, and on one tool-call case the call was well formed but the computed answer was wrong (123 where 10 + (−4) = 6). The paper confirms them by their persistence across re-runs.
How the artifact was confirmed, twice
- Normalization recovers it. The harness keeps format handling in one shared place rather than in each model adapter. Once it learned to take the first complete JSON object and to accept
paramsas an alias, re-scoring the same recorded outputs returned the correct arguments, {"a": 5, "b": 10} and {"a": 25, "b": 75}. The paper still reports the run as scored at the time (0.000), and says so. - A second backend doesn't reproduce it. A later run of the same model as W4A16 AWQ on vLLM scored
toolcall_onlyat 1.000. If 4-bit quantization had really broken tool dispatch, an independently quantized checkpoint should show it too. It didn't, and it agreed with Q4_K_M on the preserved families and on the fragile ones (json_multistepat 0.400,mcqat 3 of 5).
Without that classification, the catalog would have said this function-calling model can't call functions after quantization: a false result that would steer people away from a model that works.
The trade it sits against
From the run manifests: the F16 GGUF was about 24.5 GB and the Q4_K_M GGUF about 7.48 GB, roughly 3.3× smaller, which moves the model from the ceiling of a 24 GB GPU to an 8 GB card. Mean per-case latency in the harness was 30.24 s under FP16 Transformers, 1.42 s under Q4_K_M llama.cpp and 0.936 s under AWQ on vLLM. These are harness measurements under a fixed configuration, not production throughput.
The result is a capability profile, not a score: which model, on which hardware, for which behavior. For this model at Q4_K_M, tool-call formatting, state tracking and single-step structured output are available at a fraction of the footprint. Unassisted multi-step planning needs an external planner or validation loop.
Read the paper and the data
quant_eval: A Behavioral Evaluation Harness for Full-Weight and Quantized Large Language Models (CC BY-NC-ND 4.0).
- Read it in your browser: the two architecture diagrams carry full descriptions on hover or keyboard focus.
- Download the PDF from Zenodo: tagged for screen readers.
- The v7.21 runs: per-case results and run provenance (CC BY 4.0).
Evaluate your own model
- Talk it through: quantization and deployment questions are welcome on Discord and in GitHub Discussions.
- Before you ship a quantized model in an agent: we run this capability-resolved evaluation, with the artifact-versus-genuine review, on the models and precisions you plan to deploy, delivered as a written report: quantized model evaluation report.
— Patrick Hill, PBH Applied Systems
