Text

Qwen3 8B

Alibaba's powerful 8 billion parameter instruction-tuned language model

Run on Jetson
Memory Requirement 8GB RAM
Precision W4A16
Size 4.5GB

Jetson Inference - Supported Inference Engines

🚀
Container 
# Run Command
sudo docker run -it --rm --pull always --runtime=nvidia --network host ghcr.io/nvidia-ai-iot/vllm:latest-jetson-orin vllm serve RedHatAI/Qwen3-8B-quantized.w4a16

Performance on Jetson Orin with vLLM

26.53
output tokens/sec
Single request (concurrency=1)
142.99
output tokens/sec
8 parallel requests (concurrency=8)

Model Details

HuggingFace View on HuggingFace

Qwen3 8B is a more powerful variant in Alibaba Cloud’s latest generation of large language models. With 8 billion parameters, it offers enhanced capabilities while remaining deployable on edge devices.

Inputs and Outputs

Input: Text

Output: Text

Intended Use Cases

  • Reasoning: Advanced logical and analytical reasoning tasks
  • Function Calling: Native support for tool use and function calling
  • Subject Matter Experts: Fine-tuning for domain-specific expertise
  • Multilingual Instruction Following: Following instructions across 100+ languages
  • Translation: High-quality translation between supported languages