--- license: apache-2.0 base_model: abacusai/Smaug-Mini base_model_relation: quantized pipeline_tag: image-text-to-text library_name: transformers tags: - smaug - qwen3.8 - nvfp4 - modelopt - quantization --- # Smaug-Mini NVFP4 Community NVFP4 quantization of [abacusai/Smaug-Mini](https://huggingface.co/abacusai/Smaug-Mini), with an FP8 KV cache. The language model is quantized with NVIDIA ModelOpt while the vision tower remains in BF16. ## Quantization - NVIDIA ModelOpt: `0.47.0` - Recipe: `general/ptq/nvfp4_default-kv_fp8_cast` - Calibration prompts: 1,024 - Calibration sequence length: 4,096 - KV cache: FP8 - Vision tower: BF16 The checkpoint is stored in ModelOpt's Hugging Face format and includes `hf_quant_config.json`. Hugging Face's automated safetensors metadata may display an `8-bit` tag and a lower parameter total for this checkpoint because packed FP4 tensors are stored in `uint8` containers. The model architecture remains the 27B Smaug-Mini architecture. ## Serving Example vLLM invocation: ```bash vllm serve WiktorMatuszek/smaug-mini-nvfp4 \ --quantization modelopt_fp4 \ --kv-cache-dtype fp8 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder ``` vLLM selects an NVFP4 linear backend according to the GPU and available kernels. ## Evaluation The full capability evaluation for the unquantized model is published on the [Smaug-Mini model card](https://huggingface.co/abacusai/Smaug-Mini). This repository does not claim an independent rerun of that benchmark suite. Quantization can change outputs, so evaluate the checkpoint on your own workload before deployment. ## License and attribution Apache-2.0, following the source checkpoint. Smaug-Mini is published by Abacus.AI; this quantization is an independent community conversion.