Performance Data for Intel® AI Data Center Products

Find the latest AI benchmark performance data for Intel Data Center products, including detailed hardware and software configurations.


 

Pretrained models, sample scripts, best practices, and tutorials

 

Measurements were taken using:

 

3rd Generation Intel® Xeon® Scalable Processors

Intel® Xeon® Platinum 8380 Processor (40 Cores)

Download Tables (.xls)

Deep Learning Inference

 

Framework Version Model/Dataset Usage Precision Throughput Perf/Watt Accuracy Latency Batch size
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition fp32 965.26 img/s 1.27 76.13(%)   64
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition int8 3428.21 img/s 4.59 75.99(%)   116
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition int8 3899.15 img/s 5.17 76.02(%)   68
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition fp32 961.96 img/s 1.26 76.48(%)   64
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition fp32 938.09 img/s 1.27   21.31992 1
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition int8 3121.72 img/s 4.26   6.406724 1
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition int8 3527.74 img/s 4.87   5.669352 1
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition fp32 908.29 img/s 1.32   22.0194 1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition fp32 932.07 img/s 1.27 76.46(%)   1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition int8 3675.76 img/s 5.03 76.47(%)   1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition fp32 891.9 img/s 1.17     64
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition int8 3749.98 img/s 5.01     116
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing fp32 27.6 sent/s 0.035 93.15 (F1)   56
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing int8 80.26 sent/s 0.105 92.92 (F1)   56
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing fp32 24.7 sent/s 0.03   809.7166 1
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing int8 98.3 sent/s 0.129   203.4588 1
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing int8 69.56 sent/s 0.089 92.47 (F1)   16
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing fp32 27.75 sent/s 0.036 92.98(F1)   32
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing int8 78.35 sent/s 0.1   255.2648 1
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing fp32 26.94 sent/s 0.036   742.3905 1
OpenVINO 2022.3 BERTLarge Natural Language Processing fp32 30.42 sent/s 0.04 93.25(F1)   1
OpenVINO 2022.3 BERTLarge Natural Language Processing int8 95.91 sent/s 0.13 92.65(F1)   1
OpenVINO 2022.3 BERTLarge Natural Language Processing fp32 27.15 sent/s 0.036     16
OpenVINO 2022.3 BERTLarge Natural Language Processing int8 95.42 sent/s 0.13     16
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection fp32 21.37 img/s 0.03 20.003(mAP)   112
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection int8 86 img/s 0.137 19.9(mAP)   112
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection int8 86.98 img/s 0.116   229.9379 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection fp32 21.71 img/s 0.029   921.2345 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection int8 84.78 img/s 0.11     56
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection fp32 21.4 img/s 0.028     56
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection int8 84.78 img/s 0.11 21.4 (mAP) 235.9047 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection fp32 21.4 img/s 0.028 22.4(mAP) 934.5794 1
OpenVINO 2022.3 SSD-ResNet34 Object Detection fp32 21.58 0.029 20(mAP)   1
OpenVINO 2022.3 SSD-ResNet34 Object Detection int8 88.82 0.12 19.9(mAP)   1
OpenVINO 2022.3 SSD-ResNet34 Object Detection fp32 20.96 0.028     64
OpenVINO 2022.3 SSD-ResNet34 Object Detection int8 85.95 0.115     64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition fp32 269.44 fps 0.37 7.31 (WER)   64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition fp32 35.56 fps 0.0449   562.4297 1
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification fp32 110.13 fps 0.14 84.18(%)   64
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification int8 410.65 fps 0.54 84.05(%)   116
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification fp32 111.55 fps 0.148   179.2918 1
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification int8 385.95 fps 0.52   51.82018 1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification fp32 110.66 0.15 84.17(%)   1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification int8 1220.23 1.21 84.12(%)   1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification fp32 109.59 0.14     64
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification int8 1732.23 1.68     64
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection fp32 17.58 img/s 0.024 37.8234.23 (bbox/segm)   112
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection fp32 20.7 img/s 0.027   966.1836 1
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender fp32 1546948 rec/s 2080 80.27(AUC)   128
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender int8 5413627 rec/s 7351 80.24(AUC)   128
Intel TensorFlow 2.11 Transformer MLPerf Language Translation int8 98.1 sent/s 0.126 26.96(%)   448
Intel TensorFlow 2.11 Transformer MLPerf Language Translation fp32 79.34 sent/s 0.104 27.16(%)   448
Intel TensorFlow 2.11 Transformer MLPerf Language Translation int8 21.35 sent/s 0.028   936.7681 1
Intel TensorFlow 2.11 Transformer MLPerf Language Translation fp32 13.51 sent/s 0.018   1480.385 1
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender fp32 83852 rec/s 135   0.238515 1
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender fp32 309138 rec/s 461 75.39(%)   128
Intel TensorFlow 2.11 3D-UNet Image Segmentation int8 5.12 samp/s 0.006 85.09 (mean) 3906.25 1
Intel TensorFlow 2.11 3D-UNet Image Segmentation fp32 2.06 samp/s 0.002 5.30 (mean) 9708.738 1
Intel TensorFlow 2.11 3D-UNet Image Segmentation int8 4.43 samp/s 0.005 85.09 (mean)   6
Intel TensorFlow 2.11 3D-UNet Image Segmentation fp32 1.84 samp/s 0.002 85.3 (mean)   6
OpenVINO 2022.3 3D-UNet Image Segmentation fp32 2 samp/s 0.003 0.85 (mean)   1
OpenVINO 2022.3 3D-UNet Image Segmentation int8 6.99 samp/s 0.01 0.85 (mean)   1
OpenVINO 2022.3 3D-UNet Image Classification fp32 1.83 samp/s 0.002     6
OpenVINO 2022.3 3D-UNet Image Segmentation int8 19.64 samp/s 0.018     6

Hardware and software configuration (measured October 24, 2022):

  • Hardware configuration for Intel® Xeon® Platinum 8380 processor (formerly code named Ice Lake): 2 sockets, 40 cores, 270 watts, 16 x 64 GB DDR5 3200 memory, BIOS version SE5C620.86B.01.01.0005.2202160810, operating system: Ubuntu 22.04.1 LTS, int8 with Intel® oneAPI Deep Neural Network Library (oneDNN) v2.6.0 optimized kernels integrated into Intel® Extension for PyTorch* v1.12, Intel® Extension for TensorFlow* v2.10, and Intel® oneAPI Data Analytics Library (oneDAL) 2021.2 optimized kernels integrated into Intel® Extension for Scikit-learn* v2021.2. XGBoost v1.6.2, Intel® Distribution of Modin* v0.16.2, Intel oneAPI Math Kernel Library (oneMKL) v2022.2, and Intel® Distribution of OpenVINO™ toolkit v2022.3. Measurements may vary.
  • If the dataset is not listed, a synthetic dataset was used to measure performance. Accuracy (if listed) was validated with the specified dataset.