New Multimodal

Cosmos3 Edge

NVIDIA's edge-optimized omnimodal world model (4B) — multimodal reasoning, video generation, and robot action policies on Jetson.

Command to Run on Jetson Benchmark Model Details

Parameters 4B (2.4B reasoner + generator)

Modalities

Text Image Video Action

License OpenMDW 1.1

Precision

BF16

Serve the model

Start server

Choose module, then engine and optional parameters on the left, then copy the serve command by clicking the button on the right.

Command

Call the model over Web API

Copy a client command below and paste it into your terminal to make a Web API request to the model you just served.

curl -s http://${JETSON_HOST}:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Cosmos3-Edge",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://${JETSON_HOST}:8000/v1",
    api_key="not-needed",  # vLLM / llama.cpp typically do not enforce a key
)

completion = client.chat.completions.create(
    model="nvidia/Cosmos3-Edge",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)

Benchmark

Cosmos3 Edge (Reasoner) · vLLM · BF16 · ISL 1705 / OSL 128

Engine

Concurrency

C = concurrent requests. Results will vary with image, clocks, and workload.

Model Details

View on HuggingFace

Cosmos3 Edge is the edge-optimized member of the NVIDIA Cosmos3 family of omnimodal world models: a 2.4B multimodal reasoner paired with a diffusion-based generative tower (~4B total). It understands text, images, and video; generates images and video; and produces chunked robot action trajectories — designed for embedded deployment from Jetson Thor down to Jetson Orin-class devices.

One checkpoint, two servers: the vLLM container serves the reasoner for text, image, and video understanding through the standard OpenAI chat API (it loads only the 2.4B reasoner from the full checkpoint — verified byte-identical to the standalone Cosmos3-Edge-Reasoner release), while the vLLM-Omni container serves generation and action through its videos API. Deploy either or both depending on your workload.

Key Capabilities

Multimodal Reasoning: text, image, and video understanding on-device — served with vLLM on Jetson Thor and AGX Orin, and validated down to the 8GB Orin Nano with Hugging Face Transformers (BF16)
Robot Action Policy: chunked action inference (32 actions per call) via vLLM-Omni, with a DROID-post-trained checkpoint available (Cosmos3-Edge-Policy-DROID)
World Generation: text-to-video, image-to-video, and text-to-image via the diffusion tower
Forward / Inverse Dynamics: action-conditioned world modeling for Physical AI pipelines

Reasoner Performance on Jetson

Text, image, and video understanding through the vLLM chat API (BF16, batch 1, streaming chat API, up to 128 output tokens, greedy; decode tok/s):

Module	Text	Image	Video
Thor T5000	68.2	67.2	67.1
Thor T4000	63.9	65.6	65.5
AGX Orin 64GB	44.1	45.2	45.1

Single-stream decode is memory-bandwidth-bound, so the Thor modules land close together. At 8 concurrent streams the same server sustains ~293 tok/s aggregate on Thor T5000 and ~171 tok/s on AGX Orin.

Inputs and Outputs

Input: text prompts; images; video clips; robot observations (ego-view) with optional action conditioning
Output: text (reasoning) via vLLM; images, video, and action chunks (16 × action-dim) via vLLM-Omni

Intended Use Cases

On-device multimodal reasoning for robotics, smart spaces, and driving scenes
Robot action policies on Jetson Thor
World simulation and future prediction (video generation)
Forward / inverse dynamics modeling

Cosmos Family

Model	Parameters	Memory	Best For
Cosmos3 Edge	4B	32GB RAM	Omnimodal reasoning + generation + action at the edge
Cosmos3 Nano	16B	16GB RAM	Higher-accuracy reasoning on Thor (NVFP4)
Cosmos Reason 2 2B	2B	8GB RAM	Lightweight vision-language reasoning
Cosmos Reason 2 8B	8B	18GB RAM	Higher-accuracy vision-language reasoning
Cosmos Reason1 7B	7B	16GB RAM	Previous-generation physical reasoning