Performance Data for Intel® AI Data Center Products
Find the latest AI benchmark performance data for Intel Data Center products, including detailed hardware and software configurations.
Pretrained models, sample scripts, best practices, and tutorials
- Intel® Developer Cloud
- Intel® AI Reference Models and Jupyter Notebooks*
- AI-Optimized CPU Containers from Intel
- AI-Optimized GPU Containers from Intel
- Open Model Zoo for OpenVINO™ toolkit
- Jupyter Notebook tutorials for OpenVINO™
- AI Performance Debugging on Intel® CPUs
Measurements were taken using:
- PyTorch* Optimizations from Intel
- TensorFlow* Optimizations from Intel
- Intel® Distribution of OpenVINO™ Toolkit
3rd Generation Intel® Xeon® Scalable Processors
3rd Gen Intel® Xeon® Platinum 8352Y Processor, 32 Cores for Deep Learning Inference
Deep Learning Inference
| Framework Version | Model | Usage | Precision | Throughput | Perf/Watt | Accuracy | Latency(ms) | Batch size |
|---|---|---|---|---|---|---|---|---|
| Intel PyTorch 1.13 | ResNet50 v1.5 | Image Recognition | fp32 | 640.06 img/s | 76.13(%) with BS=128 | 1 | ||
| Intel PyTorch 1.13 | ResNet50 v1.5 | Image Recognition | int8 | 2139.92 img/s | 75.99(%) with BS=128 | 1 | ||
| Intel PyTorch 1.13 | ResNet50 v1.5 | Image Recognition | fp32 | 670.56 img/s | 64 | |||
| Intel PyTorch 1.13 | ResNet50 v1.5 | Image Recognition | int8 | 2414.29 img/s | 116 | |||
| Intel TensorFlow 2.11 | ResNet50 v1.5 | Image Recognition | fp32 | 643.11 img/s | 76.48(%) with BS=100 | 1 | ||
| Intel TensorFlow 2.11 | ResNet50 v1.5 | Image Recognition | int8 | 2396.80 img/s | 76.02(%) with BS=100 | 1 | ||
| Intel TensorFlow 2.11 | ResNet50 v1.5 | Image Recognition | fp32 | 663.59 img/s | 64 | |||
| Intel TensorFlow 2.11 | ResNet50 v1.5 | Image Recognition | int8 | 2723.73 img/s | 116 | |||
| OpenVINO | ResNet50 v1.5 | Image Recognition | fp32 | 653.36 img/s | 76.46(%) | 1 | ||
| OpenVINO | ResNet50 v1.5 | Image Recognition | int8 | 2517.58 img/s | 76.36(%) | 1 | ||
| OpenVINO | ResNet50 v1.5 | Image Recognition | fp32 | 666.39 img/s | 64 | |||
| OpenVINO | ResNet50 v1.5 | Image Recognition | int8 | 2679.54 img/s | 116 | |||
| Intel PyTorch 1.13 | BERTLarge SQuAD1.1 seq_len=384 | Natural Language Processing | fp32 | 17.42 sent/s | 93.15(F1) with BS=8 | 1 | ||
| Intel PyTorch 1.13 | BERTLarge SQuAD1.1 seq_len=384 | Natural Language Processing | int8 | 68.52 sent/s | 92.92(F1) with BS=8 | 1 | ||
| Intel PyTorch 1.13 | BERTLarge SQuAD1.1 seq_len=384 | Natural Language Processing | fp32 | 19.77 sent/s | 56 | |||
| Intel PyTorch 1.13 | BERTLarge SQuAD1.1 seq_len=384 | Natural Language Processing | int8 | 58.05 sent/s | 56 | |||
| Intel TensorFlow 2.11 | BERTLarge seq_len=384 | Natural Language Processing | fp32 | 19.24 sent/s | 92.98(F1) with BS=32 | 1 | ||
| Intel TensorFlow 2.11 | BERTLarge seq_len=384 | Natural Language Processing | int8 | 44.20 sent/s | 92.24(F1) with BS=32 | 1 | ||
| Intel TensorFlow 2.11 | BERTLarge seq_len=384 | Natural Language Processing | fp32 | 19.00 sent/s | 16 | |||
| Intel TensorFlow 2.11 | BERTLarge seq_len=384 | Natural Language Processing | int8 | 42.27 sent/s | 16 | |||
| OpenVINO | BERTLarge | Natural Language Processing | fp32 | 21.11 sent/s | 93.25(F1) | 1 | ||
| OpenVINO | BERTLarge | Natural Language Processing | int8 | 65.77 sent/s | 92.65(F1) | 1 | ||
| OpenVINO | BERTLarge | Natural Language Processing | fp32 | 20.17 sent/s | 16 | |||
| OpenVINO | BERTLarge | Natural Language Processing | int8 | 62.83 sent/s | 16 | |||
| Intel PyTorch 1.13 | SSD-ResNet34 COCO 2017 (1200 x1200) | Object Detection | fp32 | 15.04 img/s | 20 mAP with BS=16 | 1 | ||
| Intel PyTorch 1.13 | SSD-ResNet34 COCO 2017 (1200 x1200) | Object Detection | int8 | 61.19 img/s | 19.9 mAP with BS=16 | 1 | ||
| Intel PyTorch 1.13 | SSD-ResNet34 COCO 2017 (1200 x1200) | Object Detection | fp32 | 14.95 img/s | 112 | |||
| Intel PyTorch 1.13 | SSD-ResNet34 COCO 2017 (1200 x1200) | Object Detection | int8 | 57.64 img/s | 112 | |||
| Intel TensorFlow 2.11 | SSD-ResNet34 | Object Detection | fp32 | 15.20 img/s | 22.40 mAP | 1 | ||
| Intel TensorFlow 2.11 | SSD-ResNet34 | Object Detection | int8 | 60.93 img/s | 21.40 mAP | 1 | ||
| Intel TensorFlow 2.11 | SSD-ResNet34 | Object Detection | fp32 | 15.11 img/s | 56 | |||
| Intel TensorFlow 2.11 | SSD-ResNet34 | Object Detection | int8 | 59.70 img/s | 56 | |||
| OpenVINO | SSD-ResNet34 | Object Detection | fp32 | 76.85 img/s | 20 mAP | 1 | ||
| OpenVINO | SSD-ResNet34 | Object Detection | int8 | 307.83 img/s | 19.9 mAP | 1 | ||
| OpenVINO | SSD-ResNet34 | Object Detection | fp32 | 76.43 img/s | 64 | |||
| OpenVINO | SSD-ResNet34 | Object Detection | int8 | 317.44 img/s | 64 | |||
| Intel PyTorch 1.13 | RNNT LibriSpeech | Speech Recognition | fp32 | 28.32 fps | 7.31 WER with BS=64 | 1 | ||
| Intel PyTorch 1.13 | RNNT LibriSpeech | Speech Recognition | fp32 | 187.32 fps | 448 | |||
| Intel PyTorch 1.13 | ResNeXt101 32x16d ImageNet | Image Classification | fp32 | 77.49 fps | 84.18(%) at BS=128 | 1 | ||
| Intel PyTorch 1.13 | ResNeXt101 32x16d ImageNet | Image Classification | int8 | 268.56 fps | 84.05(%) at BS=128 | 1 | ||
| Intel PyTorch 1.13 | ResNeXt101 32x16d ImageNet | Image Classification | fp32 | 76.79 fps | 64 | |||
| Intel PyTorch 1.13 | ResNeXt101 32x16d ImageNet | Image Classification | int8 | 290.77 fps | 116 | |||
| OpenVINO | ResNeXt101 32x16d ImageNet | Image Classification | fp32 | 15.18 fps | 84.17(%) | 1 | ||
| OpenVINO | ResNeXt101 32x16d ImageNet | Image Classification | int8 | 61.68 fps | 84.2(%) | 1 | ||
| OpenVINO | ResNeXt101 32x16d ImageNet | Image Classification | fp32 | 15.09 fps | 64 | |||
| OpenVINO | ResNeXt101 32x16d ImageNet | Image Classification | int8 | 61.15 fps | 64 | |||
| Intel PyTorch 1.13 | MaskR-CNN COCO 2017 | Object Detection | fp32 | 14.37 img/s | 1 | |||
| Intel PyTorch 1.13 | MaskR-CNN COCO 2017 | Object Detection | fp32 | 12.39 img/s | 37.82/34.23 bbox/segm | 112 | ||
| Intel PyTorch 1.13 | DLRM Criteo Terabyte | Recommender | fp32 | 1062552.54 rec/s | 80.27 AUC | 128 | ||
| Intel PyTorch 1.13 | DLRM Criteo Terabyte | Recommender | int8 | 3712574.64 rec/s | 80.24 AUC | 128 | ||
| Intel TensorFlow 2.11 | Transformer MLPerf | Language Translation | fp32 | 12.06 sent/s | 27.16 BLEU with BS=64 | 1 | ||
| Intel TensorFlow 2.11 | Transformer MLPerf | Language Translation | int8 | 24.96 sent/s | 27.11 BLEU with BS=64 | 1 | ||
| Intel TensorFlow 2.11 | Transformer MLPerf | Language Translation | fp32 | 60.40 sent/s | 448 | |||
| Intel TensorFlow 2.11 | Transformer MLPerf | Language Translation | int8 | 48.96 sent/s | 448 | |||
| Intel TensorFlow 2.11 | DIEN Amazon Books Data | Recommender | fp32 | 69547.37 rec/s | 77.18(%) with BS=128 | 16 | ||
| Intel TensorFlow 2.11 | DIEN Amazon Books Data | Recommender | fp32 | 236578.43 rec/s | 65536 | |||
| Intel TensorFlow 2.11 | 3D-UNet | Image Segmentation | fp32 | 1.45 samp/s | 85.30 mean | 1 | ||
| Intel TensorFlow 2.11 | 3D-UNet | Image Segmentation | int8 | 3.63 samp/s | 85.08 mean | 1 | ||
| Intel TensorFlow 2.11 | 3D-UNet | Image Segmentation | fp32 | 1.35 samp/s | 6 | |||
| Intel TensorFlow 2.11 | 3D-UNet | Image Segmentation | int8 | 3.30 samp/s | 6 | |||
| OpenVINO | 3D-UNet | Image Segmentation | fp32 | 1.44 samp/s | 0.85 mean | 1 | ||
| OpenVINO | 3D-UNet | Image Segmentation | int8 | 4.91 samp/s | 0.85 mean | 1 | ||
| OpenVINO | 3D-UNet | Image Segmentation | fp32 | 1.34 samp/s | 6 | |||
| OpenVINO | 3D-UNet | Image Segmentation | int8 | 4.47 samp/s | 6 |
Hardware and software configuration (measured January 10, 2023):
- Hardware configuration for Intel® Xeon® Platinum 8352Y processor (formerly code named Ice Lake): 2 sockets, 32 cores, 205 watts, 16 x 32 GB DDR4 3200 memory, BIOS version SE5C620.86B.01.01.0006.2207150335, operating system: Ubuntu* 22.04 LTS, using Intel® Advanced Vector Extensions 512 (Intel® AVX-512) with Intel® oneAPI Deep Neural Network Library (oneDNN) v2.7 optimized kernels integrated into Intel® Extension for PyTorch* v1.13, Intel® Extension for TensorFlow* v2.12, and Intel® Distribution of OpenVINO™ toolkit v2022.3. Measurements will vary. Wall power refers to platform power consumption.
- If the dataset is not listed, a synthetic dataset was used to measure performance. Accuracy (if listed) was validated with the specified dataset.