Optimize Models with NVIDIA Model Optimizer

Most of the fastest checkpoints on this site (every nvidia/*-NVFP4 model served with vLLM on Thor) were produced by NVIDIA Model Optimizer (ModelOpt).

ModelOpt is NVIDIA’s open-source library for compressing models before inference. It provides quantization, pruning, distillation, sparsity, and speculative decoding, and exports checkpoints that load directly into vLLM and TensorRT-LLM.

NVFP4 is the 4-bit format that Jetson Thor’s Blackwell FP4 Tensor Cores run natively: the weights of the linear (matrix-multiply) layers drop to 4-bit, so checkpoints shrink ~2–3× on disk and each token moves far fewer bytes through memory.

But many models never get an official NVFP4 upload. This tutorial shows you how to make your own: quantize Qwen3.6-27B to NVFP4 directly on Jetson Thor using NVIDIA’s own PTQ script, then serve it with vLLM.

These steps aren’t specific to Jetson: the same ModelOpt workflow runs on an x86 workstation too (swap the aarch64 vLLM image for the x86 build), and here we show it end to end on a Jetson Thor Dev Kit.

Everything below was run end to end on a Thor T5000, with a released ModelOpt from PyPI and no patches, no custom configs, and no edits to the exported checkpoint.

Configuration	Weights	c1 tok/s	c8 tok/s	GPQA Diamond
BF16 (original)	52 GB	3.86	26.25	80.6%
NVFP4	19 GB	13.29	70.14	76.2%

c1 is one request at a time, c8 is eight concurrent requests, both at 2048-token input and 128-token output. NVFP4 is 3.4× faster at c1 and 2.7× at c8 from a checkpoint 2.7× smaller, and costs 4.4 points of GPQA Diamond.

Prerequisites

Requirement	Details
Device	Jetson AGX Thor (T5000 128GB recommended; 32GB fits models up to ~8B)
Software	JetPack 7.x, Docker with NVIDIA runtime
Account	Not needed for `Qwen/Qwen3.6-27B` (ungated); a free Hugging Face account is only required if you substitute a gated model
Disk	~1.5× the BF16 model size free (base + export), plus ~30 GB per container image (~60 GB for the PyTorch and vLLM images together). Calibration data is streamed, so it needs no meaningful disk.

Unified memory sizing. Quantization loads the full BF16 model before compressing it, so the sizing constraint comes from the base model rather than the output. Qwen3.6-27B is 52 GB of BF16 weights, which fits comfortably on a 128 GB Thor. Size your model against free memory on that basis: on a 32 GB device, stay at or below about 8B parameters.

Which Technique Should I Use?

Technique	What it does	Where it runs	Effort
PTQ (post-training quantization)	Compress weights to NVFP4/FP8 after training	On Jetson	Minutes
QAT (quantization-aware training)	Recover accuracy lost to quantization with brief training	Jetson (≤4B) · workstation for larger	Hours
Pruning	Remove weights/layers to shrink the model itself	Workstation (output deploys on Jetson)	Hours to days
Distillation	Teach a small model to match a larger one	Workstation (output deploys on Jetson)	Days
Speculative decoding	Propose several tokens per step, verify them in one pass: one flag on models with Multi-Token Prediction (MTP) heads, or train a draft module (a small helper model that proposes tokens)	On Jetson (MTP) · workstation for draft training	Minutes (MTP)
Sparsity	Store only non-zero weights (2:4 pattern: two of every four consecutive weights zeroed)	Workstation · Jetson serving experimental	Hours

Pick by your goal: PTQ to NVFP4 to speed up a model you already have, QAT if quantization costs you accuracy, pruning or distillation to make the model itself smaller, and speculative decoding to cut latency without changing the weights. This tutorial covers PTQ; for the others, see Beyond PTQ.

Environment Setup

The NGC PyTorch container runs on Thor’s iGPU and has everything except the Python bindings.

Step 1: Launch the container

sudo docker run -it --rm --runtime=nvidia --network host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v $HOME/modelopt-work:/work \
  nvcr.io/nvidia/pytorch:25.11-py3

Qwen/Qwen3.6-27B itself is ungated, but the default calibration data is not, so you will need a Hugging Face token. Accept the licence for nvidia/Nemotron-Post-Training-Dataset-v2, then make your token available inside the container with hf auth login, or add -e HF_TOKEN=<your token> to the docker run line above.

Step 2: Install ModelOpt and the Hugging Face bindings

pip install transformers==5.14.1 accelerate datasets
pip install nvidia-modelopt==0.45.0

Install plain nvidia-modelopt, not the [hf] extra, which pins an older transformers and would downgrade the one you just installed.

Verify the stack:

python3 -c "import torch, modelopt; print(torch.cuda.get_device_name(0), '| modelopt', modelopt.__version__)"

Quantize a Model to NVFP4

ModelOpt ships a post-training quantization script, so you do not write one. Clone the repo at the release matching the library you installed, and run it:

git clone --depth 1 --branch 0.45.0 \
  https://github.com/NVIDIA/Model-Optimizer.git
cd Model-Optimizer/examples/llm_ptq

python3 hf_ptq.py \
  --pyt_ckpt_path Qwen/Qwen3.6-27B \
  --qformat nvfp4 \
  --export_path /work/qwen36-27b-nvfp4 \
  --trust_remote_code \
  --calib_size 512

The script loads the model, runs a short calibration pass (sample text through the model so ModelOpt can record the range of values each layer produces, which sets the 4-bit scales), quantizes, and writes a standard Hugging Face checkpoint with hf_quant_config.json, the same layout as NVIDIA’s official NVFP4 uploads.

On Thor T5000 the whole run took just over 14 minutes for this 27B model with the base weights already cached: 1699 quantizers inserted, then calibration, then a 92-second export. Calibration is inference, not training.

Accept the calibration dataset licence first. ModelOpt 0.45 and newer calibrate on cnn_nemotron_v2_mix by default, which includes nvidia/Nemotron-Post-Training-Dataset-v2. That dataset is gated: accept its licence on its Hugging Face page, then make your token available inside the container with hf auth login or -e HF_TOKEN. Without it the run stops immediately with DatasetNotFoundError. This is the same calibration data NVIDIA uses for its own published NVFP4 checkpoints.

What ModelOpt works out for itself

Qwen3.6-27B is not a plain text model. It is a vision-language model whose checkpoint holds 1,199 tensors: 850 for the language model, 333 for the vision tower, and 15 multi-token-prediction heads. Its attention is a hybrid, mixing standard attention with GatedDeltaNet linear-attention layers.

None of that needed a flag. The exported hf_quant_config.json shows ModelOpt generated a 147-entry exclusion list on its own, including:

model.visual*: the entire vision tower, as a single glob
model.language_model.layers.N.linear_attn.conv1d, in_proj_a, in_proj_b: the linear-attention layers, excluded per layer
lm_head and model.language_model.embed_tokens

with quant_algo: NVFP4, kv_cache_quant_algo: FP8, and group_size: 16.

Which quantization recipe?

--qformat nvfp4 quantizes every eligible linear layer. --qformat nvfp4_mlp_only leaves the attention projections in BF16 and quantizes only the MLP layers; attention holds few parameters but is sensitive to quantization error, so the trade is a bigger file for better accuracy. Several of NVIDIA’s own published checkpoints use the mlp_only style, excluding self_attn throughout.

Other formats swap in the same way: --qformat fp8 (safest quality), int8_sq, int4_awq, w4a8_awq. The full matrix is in the PTQ examples.

Deploy with vLLM on Thor

The export is a standard Hugging Face checkpoint. No edits, no repair step: point vLLM at it.

sudo docker run -it --rm --pull always \
  --runtime=nvidia --network host \
  -v $HOME/modelopt-work/qwen36-27b-nvfp4:/model \
  -v $HOME/.cache/vllm:/root/.cache/vllm \
  --entrypoint "" \
  vllm/vllm-openai:v0.26.0-aarch64-ubuntu2404 \
  vllm serve /model \
    --gpu-memory-utilization 0.6 \
    --max-model-len 40960 \
    --trust-remote-code \
    --reasoning-parser qwen3

This model thinks before it answers. Qwen3.6 reasons at length before producing an answer. A single graduate-level question can take over 12,000 tokens. --reasoning-parser qwen3 puts that reasoning in a separate reasoning_content field so content holds just the answer, and the larger --max-model-len gives it room to finish. Without enough headroom the model is cut off mid-thought and content comes back empty.

Startup takes a while the first time. vLLM reads the quantized weights quickly (18.77 GiB in 12.3 seconds in our run), but then spends several minutes on torch.compile, FP4 kernel autotuning, and CUDA graph capture before the server answers. Watch the log rather than assuming a hang. The very first run also pulls the ~30 GB image. The /root/.cache/vllm mount above is where vLLM keeps its compile and autotune caches; with --rm and no mount they are discarded when the container exits. On a board with other services resident, lower --gpu-memory-utilization.

That server runs in the foreground, so from a second terminal, query it:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "/model", "messages": [{"role": "user", "content": "Why is the sky blue?"}], "max_tokens": 16384}'

vLLM’s startup log should show quantization=modelopt_fp4, which confirms the FP4 Tensor Core path is active.

Beyond PTQ

ModelOpt also supports quantization-aware training, pruning, distillation, speculative decoding, and sparsity; runnable recipes for each are in the ModelOpt examples.

Troubleshooting

Symptom	Cause / fix
`DatasetNotFoundError: ... is a gated dataset`	Accept the licence for `nvidia/Nemotron-Post-Training-Dataset-v2` and make your HF token available in the container
`ImportError: cannot import name 'NVFP4StaticQuantizer'`	The `examples/` you cloned are newer than your installed ModelOpt; clone the tag matching `modelopt.__version__`
Host OOM while loading	The BF16 base doesn’t fit unified memory; use a bigger Thor or a smaller model (quantization can’t start from weights it can’t load)
vLLM won’t load the export	Use a vLLM new enough to know your model’s architecture; v0.26.0 serves this checkpoint
Quality noticeably worse after PTQ	Increase `--calib_size` (512 → 1024), try `--qformat nvfp4_mlp_only` or `--qformat fp8`, or step up to QAT

What’s Next

Serve your quantized model behind the same commands as any model on this site
Fine-tune on Jetson and quantize the result, all on-device
Browse NVIDIA’s pre-quantized checkpoint collection first in case someone has already quantized your model