Framework VersionModelUsagePrecisionThroughputLatency(ms)Batch sizeCluster ModeConfig
Intel PyTorch 2.0 (master)GPT-J 6B LAMBADA (32 tokens input)LLM / chatGPTINT823.2 tokens/s431QUAD1 instance per socket, 1 socket
Intel PyTorch 2.0 (master)GPT-J 6B LAMBADA (32 token input)LLM / chatGPTBF1620.2 tokens/s491QUAD1 instance per socket, 1 socket
Intel PyTorch 2.0 (master)GPT-J 6B LAMBADA (32 tokens input)LLM / chatGPTBF1684.5 tokens/s16516QUAD1 instance per socket, 1 socket
Intel PyTorch 2.0 (master)T5-3B Samsum (32 tokens input)LLM / chatGPTBF1641.9 tokens/s281QUAD1 instance per socket, 1 socket
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint812603 img/s 1SNC41 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint86902 img/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint81542 img/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint88612 img/s 1SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint82953 img/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint810560 img/s 1SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint85079 img/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint811804 img/s 1QUAD1 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint86261 img/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint81834 img/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint87786 img/s 1QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint82948 img/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint89721 img/s 1QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint84573 img/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint813316 img/s 16SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint86174 img/s 16SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint813773 img/s 16SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint89439 img/s 16SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint813249 img/s 16SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint812012 img/s 16SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint812354 img/s 16QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint88397 img/s 16QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint812804 img/s 16QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint810817 img/s 16QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint812044 img/s 16QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionint811691 img/s 16QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167147 img/s 1SNC41 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf164325 img/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf16966 img/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf165583 img/s 1SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf161798 img/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf166778 img/s 1SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf163002 img/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf166610 img/s 1QUAD1 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf163944 img/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf161374 img/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf165076 img/s 1QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf161938 img/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf166132 img/s 1QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf162793 img/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167871 img/s 16SNC41 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf168016 img/s 16SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf163889 img/s 16SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167790 img/s 16SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf165034 img/s 16SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167763 img/s 16SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167464 img/s 16SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167170 img/s 16QUAD1 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167197 img/s 16QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf165184 img/s 16QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167095 img/s 16QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf165581 img/s 16QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf167044 img/s 16QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionbf166927 img/s 16QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321404 img/s 1SNC41 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321201 img/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp32506 img/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321337 img/s 1SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp32584 img/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321371 img/s 1SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp32875 img/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321325 img/s 1QUAD1 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321153 img/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp32686 img/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321279 img/s 1QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp32727 img/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321304 img/s 1QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp32900 img/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321455 img/s 16SNC41 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321366 img/s 16SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321008 img/s 16SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321422 img/s 16SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321181 img/s 16SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321400 img/s 16SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321398 img/s 16SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321378 img/s 16QUAD1 core per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321320 img/s 16QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321150 img/s 16QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321378 img/s 16QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321241 img/s 16QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321359 img/s 16QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0ResNet50 v1.5 Image Recognitionfp321316 img/s 16QUAD14 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint811286 img/s 1SNC41 core per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint84753 img/s 1SNC47 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint8954 img/s 1SNC456 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint86793 img/s 1SNC44 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint81773 img/s 1SNC428 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint89110 img/s 1SNC42 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint83309 img/s 1SNC414 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint810706 img/s 1QUAD1 core per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint85185 img/s 1QUAD7 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint81060 img/s 1QUAD56 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint86474 img/s 1QUAD4 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint81810 img/s 1QUAD28 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint88288 img/s 1QUAD2 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint82895 img/s 1QUAD14 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint814450 img/s 16SNC41 core per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint812750 img/s 16SNC47 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint85217 img/s 16SNC456 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint813469 img/s 16SNC44 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint87477 img/s 16SNC428 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint813400 img/s 16SNC42 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint810928 img/s 16SNC414 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint813104 img/s 16QUAD1 core per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint811776 img/s 16QUAD7 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint86803 img/s 16QUAD56 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint812695 img/s 16QUAD4 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint88380 img/s 16QUAD28 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint812342 img/s 16QUAD2 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionint810216 img/s 16QUAD14 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf163872 img/s 1SNC47 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf16841 img/s 1SNC456 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf165107 img/s 1SNC44 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf161470 img/s 1SNC428 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf166450 img/s 1SNC42 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf162519 img/s 1SNC414 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf163369 img/s 1QUAD7 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf16982 img/s 1QUAD56 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf164612 img/s 1QUAD4 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf161503 img/s 1QUAD28 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf165359 img/s 1QUAD2 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf162291 img/s 1QUAD14 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167866 img/s 16SNC47 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf162571 img/s 16SNC456 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167853 img/s 16SNC44 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf164271 img/s 16SNC428 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167835 img/s 16SNC42 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167037 img/s 16SNC414 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167183 img/s 16QUAD7 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf164601 img/s 16QUAD56 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167082 img/s 16QUAD4 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf165186 img/s 16QUAD28 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf167119 img/s 16QUAD2 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionbf166692 img/s 16QUAD14 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321167 img/s 1SNC47 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp32371 img/s 1SNC456 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321258 img/s 1SNC44 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp32587 img/s 1SNC428 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321327 img/s 1SNC42 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp32869 img/s 1SNC414 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321084 img/s 1QUAD7 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp32585 img/s 1QUAD56 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321214 img/s 1QUAD4 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp32685 img/s 1QUAD28 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321265 img/s 1QUAD2 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp32859 img/s 1QUAD14 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321369 img/s 16SNC47 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321020 img/s 16SNC456 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321376 img/s 16SNC44 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321191 img/s 16SNC428 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321389 img/s 16SNC42 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321352 img/s 16SNC414 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321319 img/s 16QUAD7 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321123 img/s 16QUAD56 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321324 img/s 16QUAD4 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321231 img/s 16QUAD28 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321332 img/s 16QUAD2 cores per instance, 2 sockets
Intel TensorFlow 2.12.0ResNet50 v1.5 Image Recognitionfp321299 img/s 16QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8211 sent/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint847 sent/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint875 sent/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8131 sent/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8208 sent/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint870 sent/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint896 sent/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8138 sent/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8194 sent/s 16SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8329 sent/s 16SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8287 sent/s 16QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingint8340 sent/s 16QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16155 sent/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf1637 sent/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf1666 sent/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16101 sent/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16151 sent/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf1660 sent/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf1680 sent/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16102 sent/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16230 sent/s 16SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16115 sent/s 16SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16187 sent/s 16SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16225 sent/s 16SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16213 sent/s 16QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16176 sent/s 16QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16201 sent/s 16QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0BERT-Large SQuAD v1.1 seq_len=384Natural Language Processingbf16211 sent/s 16QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint82354 sent/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint8354 sent/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint83442 sent/s 1SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint8827 sent/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint84998 sent/s 1SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint81616 sent/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint82442 sent/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint8557 sent/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint83231 sent/s 1QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint81029 sent/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint84600 sent/s 1QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint81689 sent/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint86675 sent/s 420SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint88536 sent/s 420SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint88288 sent/s 420QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingint88736 sent/s 420QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf162716 sent/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf16566 sent/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf163319 sent/s 1SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf16880 sent/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf163809 sent/s 1SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf161914 sent/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf162586 sent/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf16706 sent/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf163036 sent/s 1QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf161115 sent/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf163593 sent/s 1QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf161810 sent/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167854 sent/s 420SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf165191 sent/s 420SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167837 sent/s 420SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167308 sent/s 420SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167888 sent/s 420SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167224 sent/s 420QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf166860 sent/s 420QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167237 sent/s 420QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167353 sent/s 420QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingbf167323 sent/s 420QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32801 sent/s 1SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32306 sent/s 1SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32880 sent/s 1SNC44 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32492 sent/s 1SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32926 sent/s 1SNC42 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32634 sent/s 1SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32748 sent/s 1QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32364 sent/s 1QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32844 sent/s 1QUAD4 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32526 sent/s 1QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32909 sent/s 1QUAD2 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp32614 sent/s 1QUAD14 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321471 sent/s 420SNC47 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321328 sent/s 420SNC456 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321386 sent/s 420SNC428 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321446 sent/s 420SNC414 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321410 sent/s 420QUAD7 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321387 sent/s 420QUAD56 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321404 sent/s 420QUAD28 cores per instance, 2 sockets
Intel PyTorch 2.0.0DistilBERT SST-2Natural Language Processingfp321412 sent/s 420QUAD14 cores per instance, 2 sockets

Deep Learning Configuration:

  • Hardware configuration for Intel® Xeon® CPU Max Series 9480: 2 sockets, 56 cores, 350 watts, 128 GB 8 x 16 GB HBM 3200 MT/s, CentOS* Stream 8 with BIOS version EGSDCRB1.86B.9409.P28.2302110324, Intel® Hyper-Threading Technology on, turbo on, HBM_only mode with either SNC4 or QUAD mode as specified above, Power and Performance Policy=Performance, Frequency Governor=Performance, Frequency Driver=intel_pstate, THP=always, Auto NUMA Balancing=enabled. PyTorch* measurements taken using Intel® Advanced Matrix Extensions (Intel® AMX) int8 and bf16 with Intel® oneAPI Deep Neural Network Library (oneDNN) optimized kernels integrated into Intel® Extension for PyTorch* v2.0.0. Tensorflow measurements taken using Intel AMX int8 and bf16 with oneDNN v2.7.3 optimized kernels integrated into Intel® Extension for TensorFlow* v2.12.0. Measurements may vary.
  • If the dataset is not listed, a synthetic dataset was used to measure performance.