Tuning Guide for BERT-Based AI Inference with Intel® Advanced Matrix Extensions with 4th Generation Intel® Xeon® Scalable Processors

Introduction

This guide is for users who are already familiar with the BERT model. It provides recommendations for configuring hardware and software that provide the best performance in most situations. However, note that we rely on the users to carefully consider these settings for their specific scenarios, since the workload can be deployed in multiple ways.

In this tuning guide, we use the Light Model Transformer as main optimization tool. Light Model Transformer is a lightweight software tool that transforms a trained TensorFlow* model into C++ code. The generated code is based on Intel^® Math Kernel Library for Deep Neural Networks (Intel^® MKL-DNN) and provides interface to do inference without any framework, intent to accelerate the inference performance.

4th generation Intel^® Xeon^® Scalable processors deliver workload-optimized performance with built-in acceleration for AI, encryption, HPC, storage, database systems, and networking. They feature unique security technologies to help protect data on-premises or in the cloud.
 

• New built-in accelerators for AI, HPC, networking, security, storage and analytics
• Intel® Ultra Path Interconnect (Intel® UPI)
• Intel® Speed Select
• Hardware-enhanced security
• New flex bus I/O interface (PCIe* 5.0 and CXL)
• New flexible I/O interface up to 20 high-speed I/O (HSIO) lanes (PCI 3.0)
• Increased I/O bandwidth with PCIe 5.0 (up to 80 lanes)
• Increased memory bandwidth with DDR5
• Increased multisocket bandwidth with Intel UPI 2.0 (up to 16 GT/s)
• Support for Intel^® Optane™ PMem 300 series

Intel^® Advanced Matrix Extensions (Intel^® AMX) architecture is shown in Figure 1.

Figure 1.

Tiles are two-dimensional registers, and Tile Matrix Multiply Unite (TMUL) is an accelerator, which carries out operations on the tiles. Specifically, TMUL is a grid of fused multiply-add units that can read and write tiles. TMUL reads from two source tiles, multiplies their values in a systolic pattern, and adds the result into a destination tile.

For more information on Intel AMX, see LWN.net.

Server Configuration

Hardware

The configuration described in this article is based on 4th generation Intel Xeon processors. The server platform, memory, hard drives, and network interface cards can be determined according to your usage requirements.

Software
 

Hardware Tuning

This guide targets using BERT optimizations on 4th gen Intel Xeon Scalable processors with Intel^® Advanced Matrix Extensions (Intel^® AMX).

BIOS Settings

Download BIOS binaries from Intel or ODM-released BKC matched firmware binaries and flush them to the board. This document is based on BKC#57.

Software Tuning

Software configuration tuning is essential. From the operating system to workload configuration settings, they are all designed for general purpose applications and default settings are almost never tuned for best performance.

Linux* Kernel Settings

Typically, a CentOS 8 Stream is chosen for the POC environment, but to enable AMX, you need to update its kernel. At the current stage, you can use public version 5.16. In version 5.16 or the newer version, the Intel AMX feature is enabled by default. No other specific settings are needed.

To check the kernel version after rebooting, use the uname command.

oneDNN Settings

In this tool, use oneDNN to call a function related to Intel AMX. For the Linux* kernel version 5.16 and higher, only oneDNN version 2.6 and higher can support Intel AMX instructions.

You can install oneDNN according to this reference. Only oneDNN needs to be installed. No other components are needed.

BERT Tuning with Intel AMX

BERT is a widely used machine learning technique for natural language processing (NLP) that has refreshed lots of records in NLP tasks since its inception. BERT has also shown extremely strong performance in practical applications. For search, machine translation, man-machine interaction, and other NLP services, BERT has been widely adopted across multiple user scenarios. As the performance of BERT directly affects the user experience of applications, engineers have thought about all kinds of ways to optimize the model to improve its performance.

BERT includes large amount of matrix computing that can be accelerated with Intel AMX features. Theoretically, Intel AMX can improve performance by eight times compared with Vector Neural Network Instructions (VNNI) for int8 or bf16 vector computing. If you can make full use of the Intel AMX feature in the workload, you can greatly improve the business performance.

Download the Tool and Compile

git clone https://github.com/intel/light-model-transformer
cd light-model-transformer/BERT
mkdir build
cd build
source /opt/intel/oneapi/setvars.sh # Make sure CMake can find oneDNN
cmake ..
cmake --build . -j 8

tests/benchmark/benchmark

python run_classifier.py -b $encoder_handle -p $preprocessor_handle -e $epochs -l $learning_rate $output_dir

$ cd <repo_root>/python/proto

$ ./compile_proto.sh $tensorflow_include_dir

export PYTHONPATH=<repo_root/python>:$PYTHONPATH

$ cd <repo_root>/util/tf2/bert_en_uncased_L-12_H-768_A-12_4

$ ./replace_full_bert.sh $path_to_bert_encoder $path_to_fine_tuned_model

$ cd $path_to_model

$ mv saved_model.pb original_saved_model.pb

$ ln -s modified_saved_model.pb saved_model.pb

export PYTHONPATH=$PYTHONPATH:$<repo_root>/python

python -m model_modifier.configure_bert_op --no-quantization --no-bfloat16 $path_to_fine_tuned_model

python accuracy.py $model_dir $op_library

Conclusion

According to this test, the 4th generation Intel Xeon Scalable processors can improve the performance in some use cases using the VNNI instructions or Intel AMX instructions. Intel AMX instruction, can greatly promote the competitiveness of Intel CPUs in AI.

Feedback

We value your feedback. If you have comments (positive or negative) on this guide or are seeking something that is not part of this guide, reach out and let us know what you think.