Models

 

 

Model Performance Data for Intel® Gaudi® 3 AI Accelerators


These performance numbers are measured using the latest SynapseAI* software release version 1.19.0, unless otherwise noted.

Note All models for both training and inference are using the PyTorch* 2.5.1 framework. Other applicable frameworks used for training or inference are noted for each model.

Explore Intel Gaudi 2 Accelerator Performance Data

INFERENCE

Large Language Models (LLM) for Throughput with Intel Gaudi 3 Accelerator

 

Model# HPUPrecisionInput Length Output Length Batch SizeThroughput (tokens/sec)
LLaMA 2 7b1fp8 128 128 1536 20381
LLaMA 2 7b1fp8 128 2048 217 7476
LLaMA 2 7b1fp8 2048 128 153 2137
LLaMA 2 7b1fp8 2048 2048 117 2977
LLaMA 2 70b2fp8 128 128 1750 4562
LLaMA 2 70b2fp8 128 2048 512 6590
LLaMA 2 70b2fp8 2048 128 242 486
LLaMA 2 70b2fp8 2048 2048 241 2736
LLaMA 3.1 8B1fp8 128 128 1536 24364
LLaMA 3.1 8B1fp8 128 2048 768 18063
LLaMA 3.1 8B1fp8 2048 128 256 2590
LLaMA 3.1 8B1fp8 2048 2048 371 8335
LLaMA 3.1 70B2fp8 128 128 2048 4562
LLaMA 3.1 70B2fp8 128 2048 450 6278
LLaMA 3.1 70B2fp8 2048 128 223 499
LLaMA 3.1 70B2fp8 2048 2048 175 2796
LLaMA 3.1 70B8fp8 128 128 4000 15377
LLaMA 3.1 70B8fp8 128 2048 600 16891
LLaMA 3.1 70B8fp8 2048 128 512 1594
LLaMA 3.1 70B8fp8 2048 2048 600 9467
LLaMA 3.1 405B8fp8 128 128 2996 3306
LLaMA 3.1 405B8fp8 128 2048 460 4793
LLaMA 3.1 405B8fp8 2048 128 195 371
LLaMA 3.1 405B8fp8 2048 2048 180 2143

System Configuration

Intel Gaudi 3 Platform

System: HLS-Gaudi3 with eight Intel Gaudi 3 platform HL-325L mezzanine cards, two Intel Xeon Platinum 8480+ CPUs at 2.0 GHz, and 1 TB of system memory

 

Common Software

Stay Informed


Register for the latest Intel Gaudi AI accelerator developer news, events, training, and updates.

Sign Up