Optimize Models with Unsloth

Fine-tune and deploy language models directly on NVIDIA Jetson with Unsloth and JetPack 7.2. This tutorial uses memory-efficient QLoRA, exports the results to GGUF, and runs them locally.

Two practical examples cover Qwen3.5-4B on Jetson Orin Nano and NVIDIA Nemotron 3.5 Lightning 30B-A3B on Jetson AGX Thor.

Bring Your Own Model to Jetson

Fine-tune your model with QLoRA, convert it to GGUF, or run an existing GGUF locally on Jetson.

Requirements

Install Unsloth

Create a Python virtual environment on NVMe:

sudo apt-get update
sudo apt-get install -y python3.12-venv python3.12-dev build-essential git

export JETSON_UNSLOTH_DIR=/mnt/ssd/unsloth-on-jetson
mkdir -p "$JETSON_UNSLOTH_DIR"
cd "$JETSON_UNSLOTH_DIR"

python3 -m venv .venv-unsloth
source .venv-unsloth/bin/activate
python -m pip install --upgrade pip uv

export UV_CACHE_DIR="$PWD/.cache/uv"
export PIP_CACHE_DIR="$PWD/.cache/pip"
export HF_HOME="$PWD/.cache/huggingface"
export TMPDIR="$PWD/tmp"
mkdir -p "$UV_CACHE_DIR" "$PIP_CACHE_DIR" "$HF_HOME" "$TMPDIR"

Change /mnt/ssd if your NVMe drive is mounted elsewhere.

Install Unsloth with the CUDA 13.0 PyTorch backend:

uv pip install --reinstall --no-deps \
  --default-index https://pypi.org/simple \
  torchao

uv pip install --upgrade \
  --torch-backend=cu130 \
  "unsloth>=2026"

Confirm that PyTorch sees the Jetson GPU:

python - <<'PY'
import torch

assert torch.cuda.is_available(), "CUDA is not available"
print(torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))
PY

Build llama.cpp

Build llama.cpp natively with CUDA support. JetPack provides the NVIDIA driver and CUDA runtime; install the CUDA compiler components used by llama.cpp inside the same virtual environment:

sudo apt-get update
sudo apt-get install -y pciutils build-essential curl libcurl4-openssl-dev git

uv pip install --upgrade cmake ninja \
  "nvidia-cuda-nvcc>=13.0,<13.1" \
  "nvidia-cuda-crt>=13.0,<13.1" \
  "nvidia-nvvm>=13.0,<13.1" \
  "nvidia-cuda-cccl>=13.0,<13.1" \
  "nvidia-cuda-runtime>=13.0,<13.1"

export CUDA_HOME="$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13"
export CUDACXX="$CUDA_HOME/bin/nvcc"
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib:${LD_LIBRARY_PATH:-}"

ln -sfn lib "$CUDA_HOME/lib64"
ln -sfn libcudart.so.13 "$CUDA_HOME/lib/libcudart.so"
ln -sfn libcublas.so.13 "$CUDA_HOME/lib/libcublas.so"
ln -sfn libcublasLt.so.13 "$CUDA_HOME/lib/libcublasLt.so"

CUDA_ARCH=$(python - <<'PY'
import torch
major, minor = torch.cuda.get_device_capability()
print(f"{major}{minor}")
PY
)

git clone https://github.com/ggml-org/llama.cpp

cmake llama.cpp -B llama.cpp/build -G Ninja \
  -DBUILD_SHARED_LIBS=OFF \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER="$CUDACXX" \
  -DCUDAToolkit_ROOT="$CUDA_HOME" \
  -DCMAKE_CUDA_ARCHITECTURES="$CUDA_ARCH"

cmake --build llama.cpp/build \
  --config Release \
  -j \
  --clean-first \
  --target llama-cli llama-mtmd-cli llama-server llama-quantize llama-gguf-split

cp llama.cpp/build/bin/llama-* llama.cpp/

The build detects 87 on Jetson Orin and 110 on Jetson AGX Thor. Keep the virtual environment active when running llama.cpp.

Fine-tune with QLoRA

Use this section when you want to adapt a model to your own dataset before deployment. If you already have a Hugging Face model or local checkpoint, skip to Convert a model to GGUF.

Unsloth provides FastVisionModel for vision-language models and FastLanguageModel for text-only language models. The following examples use QLoRA to reduce memory use on Jetson.

python - <<'PY'
from unsloth import FastVisionModel
from unsloth.trainer import UnslothVisionDataCollator
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

MODEL = "unsloth/Qwen3.5-4B"
OUTPUT = "outputs/qwen35-4b-orin-lora"
MAX_LENGTH = 512

instruction = "Write the LaTeX representation for this image."

def convert_to_conversation(sample):
    return {
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": instruction},
                    {"type": "image", "image": sample["image"]},
                ],
            },
            {
                "role": "assistant",
                "content": [{"type": "text", "text": sample["text"]}],
            },
        ]
    }

raw_dataset = load_dataset("unsloth/LaTeX_OCR", split="train[:100]")
dataset = [convert_to_conversation(sample) for sample in raw_dataset]

model, processor = FastVisionModel.from_pretrained(
    MODEL,
    max_seq_length=MAX_LENGTH,
    load_in_4bit=True,
    use_gradient_checkpointing="unsloth",
)

model = FastVisionModel.get_peft_model(
    model,
    r=8,
    lora_alpha=8,
)

FastVisionModel.for_training(model)

trainer = SFTTrainer(
    model=model,
    processing_class=processor,
    data_collator=UnslothVisionDataCollator(model, processor),
    train_dataset=dataset,
    args=SFTConfig(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        max_steps=30,
        learning_rate=2e-4,
        optim="adamw_8bit",
        output_dir=f"{OUTPUT}/trainer",
        report_to="none",
        bf16=True,
        remove_unused_columns=False,
        dataset_text_field="",
        dataset_kwargs={"skip_prepare_dataset": True},
        max_length=MAX_LENGTH,
    ),
)

trainer.train()
model.save_pretrained(OUTPUT)
processor.save_pretrained(OUTPUT)
raw_dataset[0]["image"].save(f"{OUTPUT}/sample.png")
print("Saved adapter to", OUTPUT)
PY

export TRITON_PTXAS_BLACKWELL_PATH="$CUDA_HOME/bin/ptxas"

export NEMOTRON_SWAP="$JETSON_UNSLOTH_DIR/.nemotron35-thor.swap"

sudo fallocate -l 32G "$NEMOTRON_SWAP"
sudo chmod 600 "$NEMOTRON_SWAP"
sudo mkswap "$NEMOTRON_SWAP"
sudo swapon "$NEMOTRON_SWAP"

mkdir -p models

hf download nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --local-dir models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python - <<'PY'
from pathlib import Path

from unsloth import FastLanguageModel

from datasets import Dataset
from trl import SFTConfig, SFTTrainer

MODEL = "models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
OUTPUT = Path("outputs/nemotron35-lightning-thor-lora")
MAX_LENGTH = 128
LORA_RANK = 8

pairs = [
    (
        "Which Jetson was this adapter trained on?",
        "This adapter was trained on Jetson AGX Thor.",
    ),
    (
        "What model does this adapter customize?",
        "It customizes NVIDIA Nemotron 3.5 Lightning 30B-A3B.",
    ),
    (
        "What is the deployment format?",
        "The deployment format is GGUF.",
    ),
    (
        "Describe the optimization workflow.",
        "Fine-tune with Unsloth, export to GGUF, and run the model on Jetson.",
    ),
]

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL,
    max_seq_length=MAX_LENGTH,
    load_in_4bit=True,
    load_in_8bit=False,
    full_finetuning=False,
    device_map={"": 0},
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    unsloth_force_compile=True,
    attn_implementation="eager",
    use_gradient_checkpointing=False,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=LORA_RANK,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "in_proj",
    ],
    lora_alpha=2 * LORA_RANK,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=False,
    random_state=3407,
)

texts = []
for prompt, answer in pairs:
    texts.append(tokenizer.apply_chat_template(
        [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": answer},
        ],
        tokenize=False,
        add_generation_prompt=False,
        enable_thinking=False,
    ))

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,
    train_dataset=Dataset.from_dict({"text": texts}),
    args=SFTConfig(
        dataset_text_field="text",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,
        max_steps=3,
        learning_rate=2e-4,
        logging_steps=1,
        optim="adamw_8bit",
        output_dir=str(OUTPUT / "trainer"),
        report_to="none",
        save_strategy="no",
        bf16=True,
        fp16=False,
        gradient_checkpointing=False,
        max_length=MAX_LENGTH,
        dataloader_pin_memory=False,
    ),
)

OUTPUT.mkdir(parents=True, exist_ok=True)
trainer.train()
model.save_pretrained(OUTPUT)
tokenizer.save_pretrained(OUTPUT)
print("Saved adapter to", OUTPUT)
PY

Convert a model to GGUF

Convert a Hugging Face-format model into a quantized GGUF file for local deployment.

If you completed the fine-tuning section, use the saved adapters shown in the examples below. If you are bringing your own model, replace model_name with its Hugging Face model ID or local checkpoint. Use FastVisionModel for a vision-language model and FastLanguageModel for a text-only model.

The examples continue with the Qwen and Nemotron adapters created earlier, but the same export stage can begin from a compatible model you already have.

python - <<'PY'
from unsloth import FastVisionModel

model, processor = FastVisionModel.from_pretrained(
    model_name="outputs/qwen35-4b-orin-lora",
    max_seq_length=512,
    load_in_4bit=True,
    full_finetuning=False,
    use_gradient_checkpointing=False,
)

model.save_pretrained_gguf(
    "outputs/qwen35-4b-orin-merged",
    processor,
    quantization_method="q4_k_m",
    maximum_memory_usage=0.45,
)
PY

outputs/qwen35-4b-orin-merged_gguf/

ls -lh outputs/qwen35-4b-orin-merged_gguf

python - <<'PY'
import json
from pathlib import Path

ADAPTER = Path("outputs/nemotron35-lightning-thor-lora")
MERGED = Path("outputs/nemotron35-lightning-thor-merged")

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=str(ADAPTER),
    max_seq_length=128,
    load_in_4bit=True,
    load_in_8bit=False,
    full_finetuning=False,
    device_map={"": 0},
    low_cpu_mem_usage=True,
    trust_remote_code=True,
    unsloth_force_compile=True,
    attn_implementation="eager",
    use_gradient_checkpointing=False,
)

MERGED.mkdir(parents=True, exist_ok=True)

model.save_pretrained_merged(
    str(MERGED),
    tokenizer,
    save_method="merged_16bit",
    maximum_memory_usage=0.65,
)

config_path = MERGED / "config.json"
config = json.loads(config_path.read_text())

if "num_hidden_layers" not in config:
    config["num_hidden_layers"] = int(model.config.num_hidden_layers)
    config_path.write_text(json.dumps(config, indent=2) + "\n")

print("Saved merged model to", MERGED)
PY

set -e

MERGED=outputs/nemotron35-lightning-thor-merged
GGUF_DIR=outputs/nemotron35-lightning-thor-merged_gguf
BF16="$GGUF_DIR/NVIDIA-Nemotron-3.5-Lightning-30B-A3B.NO_MTP.BF16.gguf"
Q4="$GGUF_DIR/NVIDIA-Nemotron-3.5-Lightning-30B-A3B.NO_MTP.Q4_K_M.gguf"

mkdir -p "$GGUF_DIR"

python llama.cpp/convert_hf_to_gguf.py "$MERGED" \
  --outfile "$BF16" \
  --outtype bf16 \
  --no-mtp

./llama.cpp/llama-quantize \
  "$BF16" \
  "$Q4" \
  Q4_K_M \
  14

rm "$BF16"

Run the GGUF with llama.cpp

Use the native llama.cpp build for CUDA-accelerated inference on Jetson. If you already have a compatible GGUF file, you can start with this section and point the commands to your model.

MODEL=$(find outputs/qwen35-4b-orin-merged_gguf -name '*Q4_K_M.gguf' ! -name '*mmproj*' -print -quit)
MMPROJ=$(find outputs/qwen35-4b-orin-merged_gguf -iname '*mmproj*.gguf' -print -quit)

./llama.cpp/llama-cli \
  --model "$MODEL" \
  --mmproj "$MMPROJ" \
  --image outputs/qwen35-4b-orin-lora/sample.png \
  --prompt "Write the LaTeX representation for this image." \
  --n-gpu-layers 99 \
  --ctx-size 512 \
  --n-predict 128 \
  --jinja

MODEL=outputs/nemotron35-lightning-thor-merged_gguf/NVIDIA-Nemotron-3.5-Lightning-30B-A3B.NO_MTP.Q4_K_M.gguf

./llama.cpp/llama-cli \
  --model "$MODEL" \
  --n-gpu-layers 999 \
  --ctx-size 2048 \
  --single-turn \
  --jinja \
  --reasoning off

sudo swapoff "$NEMOTRON_SWAP"
sudo rm "$NEMOTRON_SWAP"

curl -fsSL https://ollama.com/install.sh | sh

FROM /absolute/path/to/NVIDIA-Nemotron-3.5-Lightning-30B-A3B.NO_MTP.Q4_K_M.gguf
PARAMETER temperature 0
PARAMETER top_p 0.95
PARAMETER num_ctx 2048

ollama create nemotron35-lightning-thor -f Modelfile
ollama run nemotron35-lightning-thor

Optional: Unsloth Studio over SSH

Unsloth Studio is a browser UI, not a terminal UI. It is not required for this tutorial.

If Studio is installed on the Jetson, forward its local port from your workstation:

ssh -L 8888:127.0.0.1:8888 jetson@JETSON_IP

Start Studio in the SSH session:

unsloth studio -p 8888

Open http://127.0.0.1:8888 on your workstation. See the Unsloth Studio documentation for installation and updates.

References

• Unsloth Qwen3.5
• Unsloth Qwen3.5-4B vision notebook
• Unsloth LaTeX OCR dataset
• Unsloth Nemotron 3.5
• Saving models to GGUF with Unsloth