Gemma 4 12B
Google's mid-size dense Gemma 4 model — strong general reasoning and multimodal understanding for Jetson Thor and AGX Orin
Serve the model
Pick your Jetson, then copy the command and paste it into a terminal on the Jetson. New to Jetson? Setup guide
Model Details
If you want to learn more about the Gemma 4 family and the different ways to run it on Jetson, check out the Gemma 4 on Jetson tutorial .
Gemma 4 12B is Google’s mid-size dense Gemma 4 model — the step up from the edge-sized E2B/E4B variants for workloads that need stronger reasoning while fitting on a single Jetson. This page covers Google’s official quantization-aware-trained checkpoints: W4A16 for vLLM and Q4_0 GGUF for llama.cpp, on Thor and AGX Orin.
- Local assistants and RAG that outgrow the E-series models
- Document, chart, and image understanding workloads
- Coding help and repository Q&A on Thor- and Orin-class devices
- General-purpose reasoning where MoE routing overhead isn’t wanted
Inputs and Outputs
Input: Text and image
Output: Text
Supported Platforms
- Jetson Thor (vLLM W4A16, llama.cpp GGUF)
- Jetson AGX Orin 64GB (vLLM W4A16, llama.cpp GGUF)
Inference Engine
This model is configured to run on Jetson with vLLM and llama.cpp.
Official Highlights
- Google positions 12B as the dense mid-size option in the Gemma 4 family — a balance point between the edge-sized E2B/E4B and the frontier 26B-A4B/31B models.
- Supports 256K context, text/image input, and the Gemma 4 function-calling and long-context reasoning features.
- The official QAT (quantization-aware trained) Q4_0 release preserves near-BF16 quality at 4-bit, making it the recommended GGUF for llama.cpp deployment.
Gemma 4 Family
| Model | Parameters | Memory | Best For |
|---|---|---|---|
| Gemma 4 E2B | 2.3B effective (5.1B with embeddings) | 8GB RAM | Lightweight edge deployment |
| Gemma 4 E4B | 4.5B effective (8B with embeddings) | 8GB RAM | Edge multimodal assistants |
| Gemma 4 12B | 12B dense | 16GB RAM | Mid-size reasoning and multimodal |
| Gemma 4 26B-A4B | 25.8B total / 3.8B active | 24GB RAM | High-end MoE reasoning |
| Gemma 4 31B | 31B dense | 32GB RAM | Maximum quality in the family |
Performance on Jetson
- 20.4 vLLM on AGX Thor: 20.4 tok/s 18.1 llama.cpp on AGX Thor: 18.1 tok/sAGX Thor128 GB
- 13.2 vLLM on AGX Orin: 13.2 tok/s 11 llama.cpp on AGX Orin: 11 tok/sAGX Orin64 GB
Output tokens/s
- 58.1 vLLM on AGX Thor: 58.1 tok/s 33.5 llama.cpp on AGX Thor: 33.5 tok/sAGX Thor128 GB
- 36.8 vLLM on AGX Orin: 36.8 tok/s 21.6 llama.cpp on AGX Orin: 21.6 tok/sAGX Orin64 GB
Output tokens/s
- 49.5 MTP vLLM with MTP on AGX Thor: 49.5 tok/sAGX Thor128 GB
- 32.8 MTP vLLM with MTP on AGX Orin: 32.8 tok/sAGX Orin64 GB
Output tokens/s
- 216 MTP vLLM with MTP on AGX Thor: 216 tok/sAGX Thor128 GB
- 137 MTP vLLM with MTP on AGX Orin: 137 tok/sAGX Orin64 GB
Output tokens/s