New Multimodal

Gemma 4 12B

Google's mid-size dense Gemma 4 model — strong general reasoning and multimodal understanding for Jetson Thor and AGX Orin

Command to Run on Jetson Benchmark Model Details

Parameters 12B

Modalities

Text Image

Context Length 256K

License Apache 2.0

Precision

NVFP4 Q4_0 QAT GGUF

Serve the model

Start server

Choose module, then engine and optional parameters on the left, then copy the serve command by clicking the button on the right.

Command

Call the model over Web API

Copy a client command below and paste it into your terminal to make a Web API request to the model you just served.

curl -s http://${JETSON_HOST}:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "RedHatAI/gemma-4-12B-it-NVFP4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://${JETSON_HOST}:8000/v1",
    api_key="not-needed",  # vLLM / llama.cpp typically do not enforce a key
)

completion = client.chat.completions.create(
    model="RedHatAI/gemma-4-12B-it-NVFP4",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)

llama.cpp server (OpenAI-compatible API)

After llama-server is running with --network host, call it from another machine on the LAN (set ${JETSON_HOST} or use the field). Default port is often 8080 unless you set --port.

curl -s http://${JETSON_HOST}:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my_model",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://${JETSON_HOST}:8080/v1",
    api_key="not-needed",  # vLLM / llama.cpp typically do not enforce a key
)

completion = client.chat.completions.create(
    model="my_model",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)

Benchmark

Gemma 4 12B · vLLM · NVFP4 · ISL 2048 / OSL 128

Engine

Concurrency

C = concurrent requests. Results will vary with image, clocks, and workload.

Model Details

View on HuggingFace

If you want to learn more about the Gemma 4 family and the different ways to run it on Jetson, check out the Gemma 4 on Jetson tutorial .

Gemma 4 12B is Google’s mid-size dense Gemma 4 model — the step up from the edge-sized E2B/E4B variants for workloads that need stronger reasoning while fitting on a single Jetson. This page covers the NVFP4 checkpoint for vLLM on Thor (efficient 4-bit inference on Blackwell) and Google’s official quantization-aware-trained Q4_0 GGUF for llama.cpp on Thor and AGX Orin.

Local assistants and RAG that outgrow the E-series models
Document, chart, and image understanding workloads
Coding help and repository Q&A on Thor- and Orin-class devices
General-purpose reasoning where MoE routing overhead isn’t wanted

Inputs and Outputs

Input: Text and image

Output: Text

Supported Platforms

Jetson Thor (vLLM NVFP4, llama.cpp GGUF)
Jetson AGX Orin 64GB (llama.cpp GGUF)

Inference Engine

This model is configured to run on Jetson with vLLM and llama.cpp.

Official Highlights

Google positions 12B as the dense mid-size option in the Gemma 4 family — a balance point between the edge-sized E2B/E4B and the frontier 26B-A4B/31B models.
Supports 256K context, text/image input, and the Gemma 4 function-calling and long-context reasoning features.
The official QAT (quantization-aware trained) Q4_0 release preserves near-BF16 quality at 4-bit, making it the recommended GGUF for llama.cpp deployment.

Gemma 4 Family

Model	Parameters	Memory	Best For
Gemma 4 E2B	2.3B effective (5.1B with embeddings)	8GB RAM	Lightweight edge deployment
Gemma 4 E4B	4.5B effective (8B with embeddings)	8GB RAM	Edge multimodal assistants
Gemma 4 12B	12B dense	16GB RAM	Mid-size reasoning and multimodal
Gemma 4 26B-A4B	25.8B total / 3.8B active	24GB RAM	High-end MoE reasoning
Gemma 4 31B	31B dense	32GB RAM	Maximum quality in the family