Framework Version Model Usage Precision Throughput Perf/Watt Accuracy Latency(ms) Batch size
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionint87015.80 img/s 75.99(%) with BS=128 1
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionbf163609.80 img/s 76.14(%) with BS=128 1
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionbf321153.05 img/s 76.13(%) with BS=128 1
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionfp32894.15 img/s   64
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionint89117.04 img/s   116
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionbf164851.59 img/s   68
Intel PyTorch 1.13ResNet50 v1.5Image Recognitionbf321576.28 img/s   68
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionfp32901.70 img/s 76.48(%) with BS=100 1
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionint86243.78 img/s 76.02(%) with BS=101 1
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionbf163417.28 img/s 76.75(%) with BS=102 1
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionbf321120.95 img/s 76.47(%) with BS=103 1
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionfp32901.40 img/s  22.6164
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionint88908.44 img/s  3.41116
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionbf164606.38 img/s  5.1480
Intel TensorFlow 2.11ResNet50 v1.5Image Recognitionbf321475.14 img/s   64
OpenVINOResNet50 v1.5Image Recognitionfp32885.78 img/s 76.46(%)  
OpenVINOResNet50 v1.5Image Recognitionint86495.75 img/s 76.36(%)  
OpenVINOResNet50 v1.5Image Recognitionbf163531.29 img/s 76.47(%)  
OpenVINOResNet50 v1.5Image Recognitionfp32887.29 img/s    
OpenVINOResNet50 v1.5Image Recognitionint88562.83 img/s    
OpenVINOResNet50 v1.5Image Recognitionbf164269.57 img/s    
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingfp3225.48 sent/s 93.15(F1) with BS=8 1
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingint8181.72 sent/s 92.78(F1) with BS=8 1
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingbf16114.62 sent/s 93.2(F1) with BS=8 1
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingbf3247.52 sent/s 93.15(F1) with BS=8 1
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingfp3228.20 sent/s   56
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingint8154.42 sent/s   56
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingbf16110.94 sent/s   16
Intel PyTorch 1.13BERTLarge SQuAD1.1 seq_len=384Natural Language Processingbf3245.37 sent/s   16
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingfp3225.26 sent/s 92.98(F1) with BS=32 1
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingint8173.81 sent/s 92.32(F1) with BS=32 1
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingbf16113.56 sent/s 93.01(F1) with BS=32 1
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingbf3248.19 sent/s 93.00(F1) with BS=32 1
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingfp3226.02 sent/s   16
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingint8162.11 sent/s   16
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingbf16113.03 sent/s   128
Intel TensorFlow 2.11BERTLarge seq_len=384Natural Language Processingbf3244.77 sent/s   16
OpenVINOBERTLargeNatural Language Processingfp3230.75 sent/s 93.25(F1) 1
OpenVINOBERTLargeNatural Language Processingint8207.64 sent/s 92.65(F1) 1
OpenVINOBERTLargeNatural Language Processingbf16122.66 sent/s 93.29(F1) 1
OpenVINOBERTLargeNatural Language Processingfp3228.37 sent/s   16
OpenVINOBERTLargeNatural Language Processingint8205.7 sent/s   16
OpenVINOBERTLargeNatural Language Processingbf16121.2 sent/s   16
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionfp3220.88 img/s 20 mAP with BS=16 1
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionint8301.04 img/s 19.9 mAP with BS=16 1
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionbf16147.99 img/s 19.98 mAP with BS=16 1
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionbf3221.77 img/s 20 mAP with BS=16 1
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionfp3220.82 img/s   112
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionint8278.59 img/s   112
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionbf16151.04 img/s   112
Intel PyTorch 1.13SSD-ResNet34 COCO 2017 (1200 x1200)Object Detectionbf1621.82 img/s    
Intel TensorFlow 2.11SSD-ResNet34Object Detectionfp3220.81 img/s 22.40 mAP 1
Intel TensorFlow 2.11SSD-ResNet34Object Detectionint8290.47 img/s 21.40 mAP 1
Intel TensorFlow 2.11SSD-ResNet34Object Detectionbf16148.50 img/s 22.50 mAP 1
Intel TensorFlow 2.11SSD-ResNet34Object Detectionbf3221.69 img/s 22.40 mAP 1
Intel TensorFlow 2.11SSD-ResNet34Object Detectionfp3220.73 img/s   56
Intel TensorFlow 2.11SSD-ResNet34Object Detectionint8265.92 img/s   56
Intel TensorFlow 2.11SSD-ResNet34Object Detectionbf16142.63 img/s   56
Intel TensorFlow 2.11SSD-ResNet34Object Detectionbf3221.60 img/s   56
OpenVINOSSD-ResNet34Object Detectionfp3220.51 img/s 20 mAP 1
OpenVINOSSD-ResNet34Object Detectionint8322.16 img/s 19.9 mAP 1
OpenVINOSSD-ResNet34Object Detectionbf16147.37 img/s 20 mAP 1
OpenVINOSSD-ResNet34Object Detectionfp3220.69 img/s   64
OpenVINOSSD-ResNet34Object Detectionint8303.29 img/s   64
OpenVINOSSD-ResNet34Object Detectionbf16144.55 img/s   64
Intel PyTorch 1.13RNNT LibriSpeechSpeech Recognitionfp3243.11 fps 7.31 WER with BS=64 1
Intel PyTorch 1.13RNNT LibriSpeechSpeech Recognitionbf16213.21 fps 7.30 WER with BS=64 1
Intel PyTorch 1.13RNNT LibriSpeechSpeech Recognitionbf3294.06 fps 7.32 WER with BS=64 1
Intel PyTorch 1.13RNNT LibriSpeechSpeech Recognitionfp32312.95 fps   448
Intel PyTorch 1.13RNNT LibriSpeechSpeech Recognitionfp161345.69 fps   448
Intel PyTorch 1.13RNNT LibriSpeechSpeech Recognitionbf32940.18 fps   448
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationfp32105.21 fps 84.18(%) at BS=128 1
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationint8921.42 fps 84.05(%) at BS=128 1
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationbf16506.98 fps 84.18(%) at BS=128 1
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationbf32159.27 fps 84.18(%) at BS=128 1
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationfp32104.06 fps   64
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationint81361.66 fps   116
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationbf16614.73 fps   64
Intel PyTorch 1.13ResNeXt101 32x16d ImageNetImage Classificationbf32183.57 fps   116
OpenVINOResNeXt101 32x16d ImageNetImage Classificationfp32103.2 fps 84.17(%) 1
OpenVINOResNeXt101 32x16d ImageNetImage Classificationint8922.61 fps 84.2(%) 1
OpenVINOResNeXt101 32x16d ImageNetImage Classificationbf16498.71 fps 84.16(%) 1
OpenVINOResNeXt101 32x16d ImageNetImage Classificationfp32102.46 fps   64
OpenVINOResNeXt101 32x16d ImageNetImage Classificationint81248.03 fps   64
OpenVINOResNeXt101 32x16d ImageNetImage Classificationbf16603.51 fps   64
Intel PyTorch 1.13MaskR-CNN COCO 2017Object Detectionfp3219.30 img/s   1
Intel PyTorch 1.13MaskR-CNN COCO 2017Object Detectionbf1696.30 img/s   1
Intel PyTorch 1.13MaskR-CNN COCO 2017Object Detectionbf3226.62 img/s   1
Intel PyTorch 1.13MaskR-CNN COCO 2017Object Detectionfp3217.48 img/s 37.82/34.23 bbox/segm 112
Intel PyTorch 1.13MaskR-CNN COCO 2017Object Detectionbf1689.11 img/s 37.75/34.33 bbox/segm 112
Intel PyTorch 1.13MaskR-CNN COCO 2017Object Detectionbf3225.64 img/s  37.78/34.22​bbox/segm 112
Intel PyTorch 1.13DLRM Criteo TerabyteRecommenderfp321564836.01 rec/s 80.27 AUC 128
Intel PyTorch 1.13DLRM Criteo TerabyteRecommenderint813793657.22 rec/s 80.27 AUC 128
Intel PyTorch 1.13DLRM Criteo TerabyteRecommenderbf166942136.72 rec/s 80.27 AUC 128
Intel PyTorch 1.13DLRM Criteo TerabyteRecommenderbf322648795.53 rec/s 80.27 AUC 128
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationfp3218.64 sent/s 27.16 BLEU with BS=64 1
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationint851.51 sent/s 27.11 BLEU with BS=64 1
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationbf1634.24 sent/s 27.13 BLEU with BS=64 1
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationbf3218.67 sent/s 27.14 BLEU with BS=64 1
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationfp3290.49 sent/s   448
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationint8239.95 sent/s   448
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationbf16217.82 sent/s   448
Intel TensorFlow 2.11Transformer MLPerfLanguage Translationbf32103.14 sent/s   448
Intel TensorFlow 2.11DIEN Amazon Books DataRecommenderfp3289221.46 rec/s 77.18(%) with BS=128 16
Intel TensorFlow 2.11DIEN Amazon Books DataRecommenderbf16104481.13 rec/s 77.11(%) with BS=128 16
Intel TensorFlow 2.11DIEN Amazon Books DataRecommenderbf3290065.53 rec/s 77.19(%) with BS=128 16
Intel TensorFlow 2.11DIEN Amazon Books DataRecommenderfp32359324.72 rec/s   65536
Intel TensorFlow 2.11DIEN Amazon Books DataRecommenderbf16466339.98 rec/s   65536
Intel TensorFlow 2.11DIEN Amazon Books DataRecommenderbf16376498.95 rec/s   65536
Intel TensorFlow 2.113D-UNet Image Segmentationfp322.04 samp/s 85.30 mean 1
Intel TensorFlow 2.113D-UNet Image Segmentationint89.32 samp/s 85.08 mean 1
Intel TensorFlow 2.113D-UNet Image Segmentationbf169.04 samp/s 85.31 mean 1
Intel TensorFlow 2.113D-UNet Image Segmentationbf323.12 samp/s 85.30 mean 1
Intel TensorFlow 2.113D-UNet Image Segmentationfp321.90 samp/s   6
Intel TensorFlow 2.113D-UNet Image Segmentationint810.29 samp/s   6
Intel TensorFlow 2.113D-UNet Image Segmentationbf169.35 samp/s   6
Intel TensorFlow 2.113D-UNet Image Segmentationbf323.15 samp/s   6
OpenVINO3D-UNet Image Segmentationfp321.99 samp/s 0.85 mean 1
OpenVINO3D-UNet Image Segmentationint815.5 samp/s 0.85 mean 1
OpenVINO3D-UNet Image Segmentationbf1610.34 samp/s 0.85 mean 1
OpenVINO3D-UNet Image Segmentationfp321.88 samp/s   6
OpenVINO3D-UNet Image Segmentationint814.3 samp/s   6
OpenVINO3D-UNet Image Segmentationbf169.68 samp/s   6

Hardware and software configuration (measured January 10, 2023):

  • Hardware configuration for Intel® Xeon® Platinum 6448Y processor (formerly code named Sapphire Rapids): 2 sockets, 32 cores, 225 watts, 16 x 32 GB DDR5 4800 memory, BIOS version EGSDCRB1.SYS.8901.P01.2209200243, operating system: CentOS* Stream 8, using Intel® Advanced Matrix Extensions (Intel® AMX) int8 and bf16 with Intel® oneAPI Deep Neural Network Library (oneDNN) v2.7 optimized kernels integrated into Intel® Extension for PyTorch* v1.13, Intel® Extension for TensorFlow* v2.12, and Intel® Distribution of OpenVINO™ toolkit v2022.3. Measurements may vary.
  • If the dataset is not listed, a synthetic dataset was used to measure performance. Accuracy (if listed) was validated with the specified dataset.