ToPo-ToPo/Inkling-Small-mlx-4bit

MLX 4bit conversion of thinkingmachines/Inkling-Small for Apple Silicon (mlx-vlm). 276B total / 12B active sparse MoE (42 layers, 256 routed experts top-6 + 2 shared), text + image + audio in, text out.

Requires mlx-vlm >= 0.6.9

0.6.9 is the first release whose models/inkling can load an official Inkling checkpoint through the public loader, and the first that implements the MoE global_scale / gate.bias tensors. On 0.6.7 / 0.6.8 this repo will not load.

from mlx_vlm import load, generate
model, processor = load("ToPo-ToPo/Inkling-Small-mlx-4bit")

The config is the official schema, unmodified — no key translation and no loader patches are needed.

Provenance (self-converted from official weights)

• Source: thinkingmachines/Inkling-Small (license: apache-2.0, bf16, 531.9 GB)
• Tool: mlx-vlm 0.6.9 — mlx_vlm.convert --hf-path thinkingmachines/Inkling-Small --mlx-path . -q --q-bits 4 --q-group-size 64
• Effective: 4.506 bits/weight (138 GiB on disk, ~148.7 GB peak RSS at inference)
• Only edit on top of the conversion: pad_token / eos_token added to tokenizer_config.json (the official TokenizersBackend config sets neither, so transformers raises on any padded call). Both point at existing ids — the vocabulary is unchanged.
• Verified end-to-end through an OpenAI-compatible gateway: text generation, and image input (a 640×480 test image expands to 204 vision tokens and is described correctly).

Reasoning effort

The chat template always injects a Thinking effort level: system message (default 0.9). Control it with the OpenAI-compatible reasoning_effort — "none" / "minimal" / "low" / "medium" / "high" / "max", or a float in [0.0, 0.99]. "none" disables thinking entirely.

When serving over mlx_vlm.server, note that Inkling wraps its answer in structural tokens (<|message_model|>, <|content_text|>, <|end_message|>) which the server's fixed _CONTENT_MARKERS list does not strip, and that its reasoning channel is <|content_thinking|> … <|end_message|><|message_model|> rather than one of the built-in marker pairs. Set MLX_VLM_THINKING_START_TOKEN / MLX_VLM_THINKING_END_TOKEN accordingly and strip the structural tokens, or the reasoning and those markers end up in content.

MTP (speculative decoding)

The conversion drops the built-in model.mtp.* weights (160 keys in the official bf16), as every quantized Inkling repo does. Split a drafter from the official bf16 instead:

python -m mlx_vlm.speculative.drafters.inkling_mtp.split \
    --model thinkingmachines/Inkling-Small --output Inkling-Small-MTP-bf16

As of mlx-vlm 0.6.9 the resulting drafter still cannot be used: the first draft block snapshots an empty cache and models/cache.py dereferences self.keys while it is None.

Revision history

• 2026-08-04 — reconverted with mlx-vlm 0.6.9. The previous upload had been converted with 0.6.7, whose models/inkling did not implement the MoE mlp.global_scale (50 keys) and mlp.gate.bias (40 keys) present in the official checkpoint, so those tensors were silently dropped. It also shipped a translated config (renamed intermediate_size / dense_intermediate_size, etc.) that 0.6.9 rejects. If you pulled this repo before this date, re-download it.