Multimodal

Gemma 4 12B

Google's mid-size dense Gemma 4 model — strong general reasoning and multimodal understanding for Jetson Thor and AGX Orin

Parameters 12B
Inputs
Text Image Video Audio
Outputs
Text
Context Length 256K
License Apache 2.0
Precision
W4A16 QAT Q4_0 QAT GGUF

Serve the model

Pick your Jetson, then copy the command and paste it into a terminal on the Jetson. New to Jetson? Setup guide

Run on your Jetson
Engine
Speculative decoding
Weights
First, one-time setup

Model Details

If you want to learn more about the Gemma 4 family and the different ways to run it on Jetson, check out the Gemma 4 on Jetson tutorial .

Gemma 4 12B is Google’s mid-size dense Gemma 4 model — the step up from the edge-sized E2B/E4B variants for workloads that need stronger reasoning while fitting on a single Jetson. This page covers Google’s official quantization-aware-trained checkpoints: W4A16 for vLLM and Q4_0 GGUF for llama.cpp, on Thor and AGX Orin.

  • Local assistants and RAG that outgrow the E-series models
  • Document, chart, and image understanding workloads
  • Coding help and repository Q&A on Thor- and Orin-class devices
  • General-purpose reasoning where MoE routing overhead isn’t wanted

Inputs and Outputs

Input: Text and image

Output: Text

Supported Platforms

  • Jetson Thor (vLLM W4A16, llama.cpp GGUF)
  • Jetson AGX Orin 64GB (vLLM W4A16, llama.cpp GGUF)

Inference Engine

This model is configured to run on Jetson with vLLM and llama.cpp.

Official Highlights

  • Google positions 12B as the dense mid-size option in the Gemma 4 family — a balance point between the edge-sized E2B/E4B and the frontier 26B-A4B/31B models.
  • Supports 256K context, text/image input, and the Gemma 4 function-calling and long-context reasoning features.
  • The official QAT (quantization-aware trained) Q4_0 release preserves near-BF16 quality at 4-bit, making it the recommended GGUF for llama.cpp deployment.

Gemma 4 Family

ModelParametersMemoryBest For
Gemma 4 E2B2.3B effective (5.1B with embeddings)8GB RAMLightweight edge deployment
Gemma 4 E4B4.5B effective (8B with embeddings)8GB RAMEdge multimodal assistants
Gemma 4 12B12B dense16GB RAMMid-size reasoning and multimodal
Gemma 4 26B-A4B25.8B total / 3.8B active24GB RAMHigh-end MoE reasoning
Gemma 4 31B31B dense32GB RAMMaximum quality in the family

Performance on Jetson

2K input · 128 output tokens
Concurrent requests
vLLMllama.cpp
  • 20.4 vLLM on AGX Thor: 20.4 tok/s 18.1 llama.cpp on AGX Thor: 18.1 tok/s
    AGX Thor128 GB
  • 13.2 vLLM on AGX Orin: 13.2 tok/s 11 llama.cpp on AGX Orin: 11 tok/s
    AGX Orin64 GB

Output tokens/s