Fine-tune LLMs on Jetson

Fine-tuning lets you customize a pre-trained LLM on your own data so it becomes better at a specific task — like following domain-specific instructions, answering questions in a particular style, or understanding specialized vocabulary.

Prerequisites

Which Method Should I Use?

These times are rough guides for the defaults in each section below; larger —dataset_size or more —num_epochs will take proportionally longer.

Environment Setup

Step 1: Pull the PyTorch Container

docker pull nvcr.io/nvidia/pytorch:25.11-py3

Step 2: Launch the Container

Navigate to the desired directory first, then launch the container — $(pwd) mounts your current directory as /workspace:

cd ~/Desktop/train/finetune

docker run --runtime nvidia -it --rm --ipc=host \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -v $(pwd):/workspace \
  -w /workspace \
  nvcr.io/nvidia/pytorch:25.11-py3

Step 3: Install Dependencies

Inside the container:

pip install trl datasets accelerate peft bitsandbytes

Step 4: Authenticate with Hugging Face (Optional)

Qwen models are openly available and do not require authentication. This step is only needed if you use a gated model (e.g. Meta Llama) or experience rate-limiting during large downloads (the 27B QLoRA model is ~43 GB).

export HF_TOKEN="hf_your_token_here"

Replace hf_your_token_here with your actual token from huggingface.co/settings/tokens.

Step 5: Download the Scripts

wget https://www.jetson-ai-lab.com/code-samples/finetune/full_sft_finetuning.py
wget https://www.jetson-ai-lab.com/code-samples/finetune/lora_finetuning.py
wget https://www.jetson-ai-lab.com/code-samples/finetune/qlora_finetuning.py

Training Dataset

All three scripts use the tatsu-lab/alpaca dataset by default — a collection of ~52,000 instruction-following examples in this format:

Each example is formatted into a prompt template during training:

Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
### Instruction: {instruction}

### Input: {input}

### Response: {output}

The model learns to generate the ### Response: portion. By default, only a subset is used (500–512 samples) to keep training fast for demonstration.

Using Your Own Dataset

To fine-tune on your own data, prepare a JSON Lines (.jsonl) or JSON file with the same three fields — instruction, input, and output:

[
  {
    "instruction": "Classify the sentiment of this review.",
    "input": "The battery life is amazing and the screen is crystal clear.",
    "output": "Positive"
  },
  {
    "instruction": "Extract the part number from this text.",
    "input": "Please ship 50 units of PN-4820-X to warehouse B.",
    "output": "PN-4820-X"
  },
  {
    "instruction": "Translate the following to Spanish.",
    "input": "The system is operating normally.",
    "output": "El sistema está funcionando normalmente."
  }
]

Then modify the get_alpaca_dataset() function in whichever script you’re using to load your file instead:

def get_custom_dataset(eos_token, data_path):
    dataset = load_dataset("json", data_files=data_path, split="train").shuffle(seed=42)

    def preprocess(x):
        texts = [
            ALPACA_PROMPT_TEMPLATE.format(instruction, inp, output) + eos_token
            for instruction, inp, output in zip(x["instruction"], x["input"], x["output"])
        ]
        return {"text": texts}

    return dataset.map(preprocess, remove_columns=dataset.column_names, batched=True)

Option 1: Full SFT (Qwen3.5 4B)

Full Supervised Fine-Tuning updates every parameter in the model. This gives maximum flexibility but uses more memory since the optimizer must store states for all parameters.

python full_sft_finetuning.py --output_dir ./finetuned_model

You should see output like:

============================================================
TRAINING COMPLETED
============================================================
Training runtime: 298.18 seconds
Samples per second: 1.68
Steps per second: 0.21
Train loss: 1.1620
============================================================

Saving model to ./finetuned_model...
Writing model shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:09<00:00,  9.94s/it]
Model saved successfully!

python full_sft_finetuning.py —batch_size 2 —gradient_accumulation_steps 4 —output_dir ./finetuned_model

Option 2: LoRA (Qwen3.5 9B)

LoRA (Low-Rank Adaptation) freezes the base model and trains small adapter matrices injected into the attention and MLP layers. Only ~1–2% of parameters are trainable, which dramatically reduces memory usage and makes it possible to fine-tune larger models on Jetson.

python lora_finetuning.py --output_dir ./lora_adapter

The output will show how few parameters are actually trained:

============================================================
TRAINING COMPLETED
============================================================
Training runtime: 215.79 seconds
Samples per second: 2.37
Steps per second: 0.30
Train loss: 0.9587
============================================================

Saving LoRA adapter to ./lora_adapter...
LoRA adapter saved successfully!

Option 3: QLoRA (Qwen3.5 27B)

QLoRA (Quantized LoRA) loads the base model in 4-bit precision and trains LoRA adapters on top. This dramatically reduces memory — fine-tuning a 27B model uses less memory than Full SFT on a 4B model.

python qlora_finetuning.py --output_dir ./qlora_adapter

You should see output like:

============================================================
QLoRA FINE-TUNING CONFIGURATION
============================================================
Model: Qwen/Qwen3.5-27B
Training mode: QLoRA (4-bit, rank=16, alpha=32)
Batch size: 2
Gradient accumulation: 4
Effective batch size: 8
.....
============================================================
TRAINING COMPLETED
============================================================
Training runtime: 618.94 seconds
Samples per second: 0.83
Steps per second: 0.10
Train loss: 0.8245
============================================================

Saving QLoRA adapter to ./qlora_adapter...
QLoRA adapter saved successfully!

Deploying Your Fine-tuned Model

After fine-tuning, you have model weights saved in Hugging Face SafeTensors format.

If you used Full SFT, your ./finetuned_model is already a complete model — ready to deploy as-is.

If you used LoRA or QLoRA, the output directory contains only adapter weights. Merge the adapter into the base model to produce a standalone model:

# For LoRA (Qwen3.5 9B)
python -c "
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3.5-9B', torch_dtype='auto', trust_remote_code=True)
model = PeftModel.from_pretrained(base, './lora_adapter')
merged = model.merge_and_unload()
merged.save_pretrained('./merged_model')
AutoTokenizer.from_pretrained('Qwen/Qwen3.5-9B').save_pretrained('./merged_model')
print('Merged model saved to ./merged_model')
"

# For QLoRA (Qwen3.5 27B)
python -c "
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen3.5-27B', torch_dtype='auto', trust_remote_code=True)
model = PeftModel.from_pretrained(base, './qlora_adapter')
merged = model.merge_and_unload()
merged.save_pretrained('./merged_model_27b')
AutoTokenizer.from_pretrained('Qwen/Qwen3.5-27B').save_pretrained('./merged_model_27b')
print('Merged model saved to ./merged_model_27b')
"

Once you have a complete model, you can serve it with vLLM, Ollama, llama.cpp, or TensorRT-LLM. For a full walkthrough on deploying and serving models on Jetson, see the Introduction to GenAI on Jetson: How to Run LLMs and VLMs tutorial.

Troubleshooting

References

• Hugging Face TRL Documentation
• PEFT / LoRA Documentation
• bitsandbytes Documentation
• Alpaca Dataset
• Qwen Models
• DGX Spark PyTorch Fine-tuning Playbooks