Gemma 4 on Jetson

Gemma 4 is Google’s family of open models for text generation, visual understanding, reasoning, and tool calling. On Jetson, you can use it to build local assistants, interpret images, and connect language models to your applications.

The family includes four models:

All four generate text. E2B and E4B are the smaller models for on-device applications, while 26B-A4B and 31B need more memory. Their capabilities include answering questions about images, generating code, and producing tool calls that an application can execute. E2B and E4B also support audio tasks such as transcription and translation. Available input modes depend on the runtime and model files used.

On Jetson, E2B and E4B have llama.cpp configurations for Orin Nano 8GB, Orin NX 16GB, AGX Orin 64GB, and Thor. The larger models have configurations for AGX Orin 64GB and Thor.

Prerequisites

1. Run Gemma 4 with llama.cpp

We’ll use E2B and E4B as examples. Choose one model below and run the command for your Jetson in its terminal.

The Orin commands apply to Orin Nano 8GB, Orin NX 16GB, and AGX Orin 64GB. The Thor commands apply to T4000 and T5000. Run one model at a time.

Gemma 4 E2B

docker run --rm -it --pull always --runtime nvidia --network host \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
llama-server \
-hf google/gemma-4-E2B-it-qat-q4_0-gguf \
--hf-file gemma-4-E2B_q4_0-it.gguf \
--host 0.0.0.0 \
--port 8000 \
--ctx-size 36864 \
--parallel 1 \
--n-gpu-layers 999 \
--jinja \
--lazy-mode on \
--load-mode mmap \
--cache-ram 0

docker run --rm -it --pull always --runtime nvidia --network host \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-thor \
llama-server \
-hf google/gemma-4-E2B-it-qat-q4_0-gguf \
--hf-file gemma-4-E2B_q4_0-it.gguf \
--host 0.0.0.0 \
--port 8000 \
--ctx-size 36864 \
--parallel 1 \
--n-gpu-layers 999 \
--jinja \
--lazy-mode on \
--load-mode mmap \
--cache-ram 0

Gemma 4 E4B

docker run --rm -it --pull always --runtime nvidia --network host \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
llama-server \
-hf google/gemma-4-E4B-it-qat-q4_0-gguf \
--hf-file gemma-4-E4B_q4_0-it.gguf \
--host 0.0.0.0 \
--port 8000 \
--ctx-size 36864 \
--parallel 1 \
--n-gpu-layers 999 \
--jinja \
--lazy-mode on \
--load-mode mmap \
--cache-ram 0

docker run --rm -it --pull always --runtime nvidia --network host \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-thor \
llama-server \
-hf google/gemma-4-E4B-it-qat-q4_0-gguf \
--hf-file gemma-4-E4B_q4_0-it.gguf \
--host 0.0.0.0 \
--port 8000 \
--ctx-size 36864 \
--parallel 1 \
--n-gpu-layers 999 \
--jinja \
--lazy-mode on \
--load-mode mmap \
--cache-ram 0

The first run downloads the container and model. Wait for the model to finish loading, and leave the server terminal running while you use the browser or API.

2. Chat in Your Browser

On your laptop, open http://JETSON_IP:8000, replacing JETSON_IP with your Jetson’s network address. If you’re connected over USB, open http://192.168.55.1:8000. If your browser is running on the Jetson itself, open http://localhost:8000.

To find the Jetson’s network address, run hostname -I in a terminal on the Jetson. Use its Wi-Fi or Ethernet address for network access; 192.168.55.1 is the USB address. Your laptop must be able to reach the Jetson over that connection.

Type a message in the chat box and send it. Keep the server terminal running while you chat.

3. Use the Model in Your App

The same server exposes an OpenAI-compatible API. This request uses E2B as the example. If you started E4B, change the model ID to google/gemma-4-E4B-it-qat-q4_0-gguf. Open a second terminal on the Jetson and send the request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/gemma-4-E2B-it-qat-q4_0-gguf",
    "messages": [
      {"role": "user", "content": "Suggest three things a plant-watering robot should check before watering."}
    ],
    "max_tokens": 512
  }'

The response is JSON. Read the reply from choices[0].message.content. From your laptop, replace localhost with the Jetson’s network IP address or 192.168.55.1 over USB.

Troubleshooting

If the chat page does not open, check that the server has finished loading and that your laptop can reach the Jetson over USB or the network. Use the port from the launch command.

If a model fails to load, check the error in the server terminal. Make sure the container and model downloads completed and that there is enough free storage and memory. Stop any previous model server before starting another one.

Next Steps

To try a larger Gemma 4 model on AGX Orin 64GB or Thor, see 26B-A4B or 31B. Use the command for your module and check its port and model ID when adapting the browser and API examples above.