Nemotron 3.5 Lightning
NVIDIA's fast open-weight model for responsive local agents, reasoning, coding, and tool use. It delivers performance comparable to the much larger Nemotron 3 Super.
Serve the model
Pick your Jetson, then copy the command and paste it into a terminal on the Jetson. New to Jetson? Setup guide
Model Details
View on HuggingFace
NVIDIA Nemotron 3.5 Lightning is a fast, open-weight 30B Mixture-of-Experts model that activates only 3B parameters per token. It brings performance close to the much larger Nemotron 3 Super while remaining practical for local coding assistants, research agents, tool-calling workflows, and other always-on applications.
The model supports context lengths of up to one million tokens. Reasoning can be enabled or disabled, and a configurable reasoning budget lets users control how much reasoning the model performs before answering.
Nemotron 3.5 Lightning supports several speculative decoding options, including built-in Multi-Token Prediction, DSpark, and DFlash. We tested the available approaches and found DSpark with vLLM delivered the best performance on both Jetson AGX Thor and Jetson AGX Orin.
Inputs and Outputs
Input: Text
Output: Text
Supported Platforms
- Jetson AGX Orin 64GB
- Jetson AGX Thor T4000
- Jetson AGX Thor T5000 Developer Kit
Why Nemotron 3.5 Lightning on Jetson
- Super-class capability: Performance close to Nemotron 3 Super in a model that is much faster and more practical to run locally.
- Responsive local agents: Strong performance for multi-step reasoning, tool calls, coding assistants, and research workflows.
- Controllable reasoning: Enable or disable reasoning and set a reasoning budget to balance quality, latency, and token usage.
- Long-context support: Work with context lengths of up to one million tokens when memory allows.
- Fast speculative decoding: Choose from MTP, DSpark, and DFlash, with DSpark providing the best results in our Jetson testing.
Speculative Decoding on Jetson
Nemotron 3.5 Lightning includes Multi-Token Prediction and is released with dedicated DSpark and DFlash checkpoints. The vLLM commands use DSpark with five speculative tokens, which was the fastest configuration in our testing on both supported Jetson platforms. The llama.cpp commands use the DFlash checkpoint.
The vLLM server exposes an OpenAI-compatible API on port 8000 with reasoning parsing, automatic tool selection, and the Qwen3 Coder tool-call parser enabled. The llama.cpp server exposes its API on port 8080.
TensorRT Edge-LLM on Jetson Thor
The TensorRT Edge-LLM command is an experimental, Thor-only NVFP4 path. Its default engine is intentionally conservative: batch size 1, maximum input length 2048 tokens, and KV-cache capacity 2200 tokens. The model’s 1M-token capability is not enabled by this engine configuration.
Standard OpenAI-compatible chat completions are validated on Jetson AGX Thor. For tool calling, structured reasoning parsing, or speculative decoding, use the vLLM command above.
The helper serves on port 8000 by default. If that port is already in use, append --port 8001 after --stage serve in the Docker command, then use port 8001 for the OpenAI-compatible API.
Additional Resources
Performance on Jetson
- 59 vLLM on AGX Thor: 59 tok/s 33.1 llama.cpp on AGX Thor: 33.1 tok/sAGX Thor128 GB
- 37.9 vLLM on AGX Orin: 37.9 tok/s 19.3 llama.cpp on AGX Orin: 19.3 tok/sAGX Orin64 GB
Output tokens/s
- 149 vLLM on AGX Thor: 149 tok/s 45.9 llama.cpp on AGX Thor: 45.9 tok/sAGX Thor128 GB
- 79 vLLM on AGX Orin: 79 tok/s 27.8 llama.cpp on AGX Orin: 27.8 tok/sAGX Orin64 GB
Output tokens/s
- 118 DSpark vLLM with DSpark on AGX Thor: 118 tok/s 90.8 MTP vLLM with MTP on AGX Thor: 90.8 tok/s 85.2 DFlash vLLM with DFlash on AGX Thor: 85.2 tok/s 63.2 DFlash llama.cpp with DFlash on AGX Thor: 63.2 tok/sAGX Thor128 GB
- 81 DSpark vLLM with DSpark on AGX Orin: 81 tok/s 70.9 MTP vLLM with MTP on AGX Orin: 70.9 tok/s 65.3 DFlash vLLM with DFlash on AGX Orin: 65.3 tok/s 38.7 DFlash llama.cpp with DFlash on AGX Orin: 38.7 tok/sAGX Orin64 GB
Output tokens/s
- 311 DSpark vLLM with DSpark on AGX Thor: 311 tok/s 242 MTP vLLM with MTP on AGX Thor: 242 tok/s 231 DFlash vLLM with DFlash on AGX Thor: 231 tok/s 48.1 DFlash llama.cpp with DFlash on AGX Thor: 48.1 tok/sAGX Thor128 GB
- 194 DSpark vLLM with DSpark on AGX Orin: 194 tok/s 167 MTP vLLM with MTP on AGX Orin: 167 tok/s 160 DFlash vLLM with DFlash on AGX Orin: 160 tok/s 31.3 DFlash llama.cpp with DFlash on AGX Orin: 31.3 tok/sAGX Orin64 GB
Output tokens/s