Performance Data for Intel® AI Data Center Products

                                                                               

Find the latest performance data for 4th gen Intel® Xeon® Scalable processors and 3rd gen Intel® Xeon® processors, including detailed hardware and software configurations.

For pretrained models, sample scripts, best practices, and tutorials used, see:

Measurements were taken using:

 

 

4th Gen Intel® Xeon® Scalable Processors for Deep Learning Inference

 

Framework Version Model/Dataset Usage Part Precision Throughput Wall Power(Watt) Perf/Watt Accuracy Latency(ms) Batch size
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 1338.80 img/s 1067.19 1.25 76.13(%)   64
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt int8 13012.99 img/s 1018.38 12.78 75.99(%)   116
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 7002.92 img/s 1028.97 6.81 76.14(%)   68
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf32 2068.72 img/s     76.13 (%)   64
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 1293.33 img/s 1028.96 1.25   21.65 1
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt int8 9680.59 img/s 1023.13 9.46   2.89 1
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 5805.89 img/s 1030.95 5.63   4.82 1
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 1294.17 img/s 1034.87 1.25 76.48(%)   64
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt int8 12390.26 img/s 1055.36 11.74 76.02(%)   116
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 6299.25 img/s 1051.71 5.99 76.75(%)   80
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf32 1984.48 img/s     76.47(%)   64
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 1238.55 img/s 1031.19 1.2   22.61 1
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt int8 8221.7 img/s 964.19 8.52   3.41 1
Intel TensorFlow 2.11 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 5451.4 img/s 1003.79 5.43   5.14 1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 1252.31 img/s 1017.9 1.23 76.46(%)   1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition 56 core 350 Watt int8 8982.73 img/s 1006.58 8.92 76.36(%)   1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 5719.37 img/s 1011.18 5.66 76.47(%)   1
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 1248.72 img/s 1034.33 1.21     64
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition 56 core 350 Watt int8 11951.34 img/s 1030.59 11.6     64
OpenVINO 2022.3 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 6087.33 img/s 1027.35 5.93     116
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt fp32 40.15 sent/s 1097.96 0.04 93.15(%)   56
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt int8 212.71 sent/s 1089.53 0.2 92.78(%)   56
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt bf16 162.24 sent/s 1009.88 0.16 93.2(%)   16
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt fp32 65.41 sent/s     93.15(%)   16
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt fp32 34.68 sent/s 1074.68 0.03   807.38 1
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt int8 254.06 sent/s 976.32 0.26   110.21 1
Intel PyTorch 1.13 BERTLarge SQuAD1.1 seq_len=384 Natural Language Processing 56 core 350 Watt bf16 154.28 sent/s 1036.73 0.14   181.49 1
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt fp32 37.00 sent/s 989.91 0.04 92.98(%)   16
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt int8 232.61 sent/s 989.57 0.24 92.32(%)   32
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt bf16 160.27 sent/s 1078.51 0.15 93.01(%)   16
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt bf32 65.78 sent/s     93.0(%)   16
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt fp32 37.23 sent/s 1069.21 0.03   752.08 1
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt int8 246.85 sent/s 1027.49 0.24   113.43 1
Intel TensorFlow 2.11 BERTLarge seq_len=384 Natural Language Processing 56 core 350 Watt bf16 157.65 sent/s 999.12 0.15   177.61 1
OpenVINO 2022.3 BERTLarge Natural Language Processing 56 core 350 Watt fp32 44.4 sent/s 1042.17 0.04 93.25(F1)   1
OpenVINO 2022.3 BERTLarge Natural Language Processing 56 core 350 Watt int8 286.24 sent/s 1003.52 0.29 92.65(F1)   1
OpenVINO 2022.3 BERTLarge Natural Language Processing 56 core 350 Watt bf16 168.36 sent/s 1007.01 0.17 93.29(F1)   1
OpenVINO 2022.3 BERTLarge Natural Language Processing 56 core 350 Watt fp32 41.81 sent/s 1030.75 0.04     16
OpenVINO 2022.3 BERTLarge Natural Language Processing 56 core 350 Watt int8 288.34 sent/s 1005.31 0.29     16
OpenVINO 2022.3 BERTLarge Natural Language Processing 56 core 350 Watt bf16 177.3 sent/s 1005.46 0.18     16
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt fp32 31.27 img/s 999.88 0.03 20 mAP   112
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt int8 377.12 img/s 782.1 0.48 19.9 mAP   112
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf16 206.27 img/s 864.55 0.24 19.98 mAP   112
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf32 32.53 img/s     20 mAP   112
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt fp32 31.17 img/s 1071.73 0.02   898.3 1
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt int8 425.89 img/s 1035.36 0.41   65.74 1
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf16 215.52 img/s 1073.23 0.2   129.92 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection 56 core 350 Watt fp32 30.31 img/s 1039.8 0.03 22.40 mAP 923.79 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection 56 core 350 Watt int8 412.59 img/s 1036.38 0.4 21.40 mAP 67.86 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection 56 core 350 Watt bf16 209.05 img/s 1091.37 0.19 22.50 mAP 133.94 1
Intel TensorFlow 2.11 SSD-ResNet34 Object Detection 56 core 350 Watt bf32 31.85 img/s     22.40 mAP 879.12 1
OpenVINO 2022.3 SSD-ResNet34 Object Detection 56 core 350 Watt fp32 30.64 img/s 1033.36 0.03 20 mAP   1
OpenVINO 2022.3 SSD-ResNet34 Object Detection 56 core 350 Watt int8 466.78 img/s 1053.6 0.44 19.9 mAP   1
OpenVINO 2022.3 SSD-ResNet34 Object Detection 56 core 350 Watt bf16 213.92 img/s 1067.98 0.2 20 mAP   1
OpenVINO 2022.3 SSD-ResNet34 Object Detection 56 core 350 Watt fp32 30.54 img/s 1039.82 0.03     64
OpenVINO 2022.3 SSD-ResNet34 Object Detection 56 core 350 Watt int8 413.69 img/s 1075.36 0.38     64
OpenVINO 2022.3 SSD-ResNet34 Object Detection 56 core 350 Watt bf16 198.93 img/s 1076.76 0.18     64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt fp32 410.58 fps 953 0.43 7.31 WER   64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt bf16 1663.41 fps 978.13 1.7 7.30 WER   64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt bf32 1103.73 fps     7.32 WER   64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt fp32 57.66 fps 1063 0.05   485.61 1
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt bf16 306.31 fps 1055.75 0.29   91.41 1
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt fp32 158.28 fps 1051.99 0.15 84.18(%)   64
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt int8 1870.01 fps 1064.08 1.76 84.05(%)   116
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt bf16 865.86 fps 1053.21 0.82 84.18(%)   64
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt bf32 241.77 fps     84.18(%)   64
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt fp32 148.2 fps 1063.83 0.13   188.93 1
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt int8 1206.45 fps 989.23 1.21   23.21 1
Intel PyTorch 1.13 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt bf16 636.1 fps 1010.28 0.62   44.02 1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt fp32 145.76 fps 1023.5 0.14 84.17(%)   1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt int8 1220.23 fps 1009.7 1.21 84.2(%)   1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt bf16 644.92 fps 1013.11 0.64 84.16(%)   1
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt fp32 152.25 fps 1043.98 0.15     64
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt int8 1732.23 fps 1029.84 1.68     64
OpenVINO 2022.3 ResNeXt101 32x16d ImageNet Image Classification 56 core 350 Watt bf16 826.85 fps 1050.67 0.79     64
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt fp32 25.71 img/s 1051.27 0.02 37.82/34.23 bbox/segm   112
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt bf16 112.54 img/s 967.22 0.11 37.75/34.33 bbox/segm   112
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt bf32 34.19 img/s     37.78/34.22 bbox/segm   112
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt fp32 28.43 img/s 1081.51 0.02   984.88 1
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt bf16 129.14 img/s 1081.93 0.11   216.82 1
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt fp32 2321626 rec/s 1000.02 2321 80.27 AUC   128
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt int8 19404011 rec/s 947.08 20488 80.24 AUC   128
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt bf16 9818003 rec/s 1037.83 9460 80.27 AUC   128
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt bf32 3875003 rec/s     80.27 AUC   128
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt fp32 20.34 sent/s 1098.8 0.01   1376 1
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt fp32 101.58 sent/s 1054.17 0.1 27.60 BLEU   448
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt int8 265.23 sent/s 1067.9 0.25 27.11 BLEU   448
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt bf16 237.23 sent/s 1026.85 0.23 27.13 BLEU   448
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt bf32 110.66 sent/s     27.14 BLEU   448
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt int8 64.61 sent/s 1094.1 0.05   433.37 1
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt bf16 40.69 sent/s 1139.17 0.03   688.13 1
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender 56 core 350 Watt fp32 423825 rec/s 957.92 442.44 77.18(%)   65536
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender 56 core 350 Watt bf16 572868 rec/s 871.09 657.65 77.12(%)   65536
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender 56 core 350 Watt bf32 436195 rec/s     77.19(%)   65536
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender 56 core 350 Watt fp32 129846 rec/s 1000.84 129.73     16
Intel TensorFlow 2.11 DIEN Amazon Books Data Recommender 56 core 350 Watt bf16 156727 rec/s 977.44 160.34     16
Intel TensorFlow 2.11 3D-UNet Image Segmentation 56 core 350 Watt fp32 2.61 samp/s 1055.99   85.30 mean   6
Intel TensorFlow 2.11 3D-UNet Image Segmentation 56 core 350 Watt int8 11.05 samp/s 1103.99 0.01 85.09 mean   1
Intel TensorFlow 2.11 3D-UNet Image Segmentation 56 core 350 Watt bf16 11.60 samp/s 1038.03 0.01 85.31 mean   6
Intel TensorFlow 2.11 3D-UNet Image Segmentation 56 core 350 Watt bf32 3.73 samp/s     85.30 mean   1
Intel TensorFlow 2.11 3D-UNet Image Segmentation 56 core 350 Watt fp32 2.88 samp/s 1083.78 0.0026   9722 1
Intel TensorFlow 2.11 3D-UNet Image Segmentation 56 core 350 Watt bf16 11.01 samp/s 1176.66 0.009   2543 1
OpenVINO 2022.3 3D-UNet Image Segmentation 56 core 350 Watt fp32 2.81 samp/s 1073.97 0.003 0.85 mean   1
OpenVINO 2022.3 3D-UNet Image Segmentation 56 core 350 Watt int8 21.28 samp/s 1097.65 0.019 0.85 mean   1
OpenVINO 2022.3 3D-UNet Image Segmentation 56 core 350 Watt bf16 13.11 samp/s 1122.76 0.012 0.85 mean   1
OpenVINO 2022.3 3D-UNet Image Segmentation 56 core 350 Watt fp32 2.58 samp/s 1065.65 0.002     6
OpenVINO 2022.3 3D-UNet Image Segmentation 56 core 350 Watt int8 19.64 samp/s 1091.12 0.018     6
OpenVINO 2022.3 3D-UNet Image Segmentation 56 core 350 Watt bf16 12.18 samp/s 1112.3 0.011     6

 

4th Gen Intel® Xeon® Processors for Deep Learning Training

 

Framework Version Model/Dataset Usage Part Precision Throughput Power(Watts) Perf/Watt Batch size
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt fp32 128.33 img/s 764.02 0.16 128
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf16 302.66 img/s 721.43 0.41 128
Intel PyTorch 1.13 ResNet50 v1.5 Image Recognition 56 core 350 Watt bf32 145.73 img/s     128
Intel TensorFlow 2.11 ResNet50 v1.5 ImageNet (224 x224) Image Recognition 56 core 350 Watt fp32 131.54 img/s 816.62 0.16 1024
Intel TensorFlow 2.11 ResNet50 v1.5 ImageNet (224 x224) Image Recognition 56 core 350 Watt bf16 292.48 img/s 818.16 0.35 1024
Intel TensorFlow 2.11 ResNet50 v1.5 ImageNet (224 x224) Image Recognition 56 core 350 Watt bf32 148.07 img/s     1024
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt fp32 262891 rec/s 807.22 325.67 32768
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt bf16 789677 rec/s 797.31   32768
Intel PyTorch 1.13 DLRM Criteo Terabyte Recommender 56 core 350 Watt bf32 349616 rec/s     32768
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt fp32 55.89 img/s 729.24 0.07 224
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf16 206.45​ img/s 681.54   224
Intel PyTorch 1.13 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf32 74.12 img/s     224
Intel TensorFlow 2.11 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt fp32 47.75 img/s 776.25 0.06 896
Intel TensorFlow 2.11 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf16 183.35 img/s 698.98 0.26 896
Intel TensorFlow 2.11 SSD-ResNet34 COCO 2017 (1200 x1200) Object Detection 56 core 350 Watt bf32 62.17 img/s     896
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt fp32 3.35 fps 781.65 0.004 64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt bf16 26.43 fps 546.77 0.048 64
Intel PyTorch 1.13 RNNT LibriSpeech Speech Recognition 56 core 350 Watt bf32 10.48 fps     64
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt fp32 3.71 img/s 799.53 0.0046 112
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt bf16 11.63 img/s 792.74 0.0146 112
Intel PyTorch 1.13 MaskR-CNN COCO 2017 Object Detection 56 core 350 Watt bf32 4.30 img/s     112
Intel PyTorch 1.13 BERTLarge Wikipedia 2020/01/01 seq len=512 Natural Language Processing 56 core 350 Watt fp32 3.72 sent/s 817.6 0.0045 28
Intel PyTorch 1.13 BERTLarge Wikipedia 2020/01/01 seq len=512 Natural Language Processing 56 core 350 Watt bf16 10.01 sent/s 823.3 0.0121 56
Intel PyTorch 1.13 BERTLarge Wikipedia 2020/01/01 seq len=512 Natural Language Processing 56 core 350 Watt bf32 4.39 sent/s     28
Intel TensorFlow 2.11 BERTLarge Wikipedia 2020/01/01 seq len=512 Natural Language Processing 56 core 350 Watt fp32 3.38 sent/s 801.63 0.004 128
Intel TensorFlow 2.11 BERTLarge Wikipedia 2020/01/01 seq len=512 Natural Language Processing 56 core 350 Watt bf16 10.02 sent/s 810.56 0.012 128
Intel TensorFlow 2.11 BERTLarge Wikipedia 2020/01/01 seq len=512 Natural Language Processing 56 core 350 Watt bf32 3.82 sent/s     128
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt fp32 7055.54 sent/s 770.18 9.16 12K
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt bf16 15184.80 sent/s ​ 754.74   12K
Intel TensorFlow 2.11 Transformer MLPerf Language Translation 56 core 350 Watt bf32 7190.51 sent/s ​     12K

Hardware and software configuration (measured October 24, 2022):

  • Hardware configuration for Intel® Xeon® Platinum 8480+ processor (formerly code named Sapphire Rapids): 2 sockets, 56 cores, 350 watts, 16 x 64 GB DDR5 4800 memory, BIOS version EGSDCRB1.SYS.8901.P01.2209200243, operating system: CentOS* Stream 8, using Intel® Advanced Matrix Extensions (Intel® AMX) int8 and bf16 with Intel® oneAPI Deep Neural Network Library (oneDNN) v2.7 optimized kernels integrated into Intel® Extension for PyTorch* v1.13, Intel® Extension for TensorFlow* v2.12, and Intel® Distribution of OpenVINO™ toolkit v2022.3. Measurements may vary.
  • Wall power refers to platform power consumption.
  • If the dataset is not listed, a synthetic dataset was used to measure performance. Accuracy (if listed) was validated with the specified dataset.