Building Software Acceleration Features in the Intel® Quick Assist Technology (Intel® QAT) Engine for OpenSSL*

ID 658256
Updated 9/16/2022
Version Latest
Public

author-image

By

Intel® Quick Assist Technology (Intel® QAT) has been expanded to provide software-based acceleration of cryptographic operations through instructions in the Intel® Advanced Vector Extensions 512 (Intel® AVX-512) family. This software-based acceleration has been incorporated into the Intel QAT Engine for OpenSSL*, a dynamically loadable module that uses the OpenSSL ENGINE framework, allowing administrators to add this capability to OpenSSL without having to rebuild or replace their existing OpenSSL libraries.

Software acceleration is provided for the following algorithms:

  • RSA with 2048, 3072, and 4096-bit keys
  • ECDH for the Montgomery Curve X25519 and NIST Prime Curves P-256 and P-384
  • ECDSA for the NIST Prime Curves P-256 and P-384
  • AES-GCM with 128, 192, and 256-bit keys

About This Guide

This guide steps you through the process of building the Intel QAT Engine for OpenSSL on the following Linux distributions, but it can be adapted to others.

  • Ubuntu* Server 22.04 LTS
  • Redhat Enterprise Linux 9.2

Building OpenSSL

Two build procedures are provided: one that uses the distribution-provided build of OpenSSL, and one that creates a customized installation using OpenSSL built from source. Each methodology has its pros and cons, and you should choose the procedure that works best for your environment. Using the distribution-provided OpenSSL means less complexity as you are running OpenSSL out of its standard system path, but it ties you to a specific version that is integrated with the OS. Building OpenSSL from source lets you control the version that you deploy independent of the distribution-provided build, and that makes it possible to perform version updates as needed without disrupting system operations. This added flexibility comes at a cost, however, as you'll need to add the OpenSSL binary directory to your PATH and update LD_LIBRARY_PATH to include the shared library directories for OpenSSL and its dependent libraries.

Click the tab for the desired build option to view the procedure.

 

 Using the Distribution-Provided OpenSSL

This section describes how to build the Intel QAT Engine for OpenSSL for your OS distribution's pre-packaged OpenSSL. If you want to build the engine for a custom build of OpenSSL that is made from source code, click the Building OpenSSL from Source tab, above.

Build Requirements

To build the Intel QAT Engine for OpenSSL you'll need to ensure that your distribution's default version of OpenSSL is 1.1.1e or later, as the engine is not compatible with earlier releases. You can check your distribution's OpenSSL version by running:

openssl version

You'll also need some prerequisite software packages to build both the engine and its dependencies.

Ubuntu 22.04 LTS

To build the QAT engine and its dependencies on Ubuntu, you’ll need to install the following packages from apt:

sudo apt install autoconf build-essential libtool cmake cpuid libssl-dev pkg-config_nasm

The libssl-dev package provides the header files for OpenSSL and ensures that the OpenSSL libraries are present.

RHEL 9.2
sudo yum install wget autoconf libtool cmake pkg-config nasm openssl
sudo yum group install "Development Tools"

Runtime Requirements

To make use of the software acceleration features in the Intel QAT Engine for OpenSSL, you’ll need a system that supports Intel® AVX-512 with the following instruction set extensions:

  • AVX512F
  • AVX512_IFMA
  • VAES
  • VPCLMULQDQ

The latter two extensions were introduced with certain 10th Generation Intel® Core™ processors or newer and 3rd Generation Intel® Xeon® Scalable processors (products formerly codenamed Ice Lake) or newer. A quick way to verify that your system supports the necessary features is to run the cpuid command. Run the following and check that the output matches.

$ cpuid -1 | egrep 'VAES|VPCLM|GFNI|AVX512F|AVX512IFMA'
      AVX512F: AVX-512 foundation instructions = true
      AVX512IFMA: fused multiply add           = true
      VAES instructions                        = true
      VPCLMULQDQ instruction                   = true

All features must be present.

These output fields are only present in cpuid version 20211210 or later. This is the default version provided in Ubuntu 22.04.

Building the Intel QAT OpenSSL Engine for Software Acceleration

The software acceleration support in the Intel QAT Engine for OpenSSL depends on the following two libraries. They must be built first, but they may be built in any order:

Once these libraries are installed, you can build the Intel Quick Assist Technology OpenSSL Engine.

We’ll step through how to build each one.

Building Intel® Integrated Performance Primitives Cryptography

First, checkout the source code repository from GitHub*:

git clone https://github.com/intel/ipp-crypto.git
cd ipp-crypto

Ensure you are building against a fixed release of the code, and not the development branch.

git checkout ippcp_2021.7.1

You only need to build the multi-buffer portion of the Intel IPP package, so change to the multi-buffer crypto library subdirectory. Then, prepare the build by running cmake:

cd sources/ippcp/crypto_mb
cmake . -Bbuild -DCMAKE_INSTALL_PREFIX=/usr

This will configure the library to install into /usr. To perform the full build, run:

cd build
make -j
sudo make install

This will put the shared library in /usr/lib, which means we won’t need to set LD_LIBRARY_PATH.

Building the Intel® Multi-Buffer Crypto for IPsec Library

First, checkout the source code repository from GitHub:

git clone https://github.com/intel/intel-ipsec-mb.git
cd intel-ipsec-mb

Ensure you are building against a fixed release of the code, and not a development branch.

git checkout v1.3

There is no configuration step. Build the library using:

make -j

To install:

sudo make install NOLDCONFIG=y

This will place the shared libraries in /usr/lib, which again means no LD_LIBRARY_PATH modifications.

Building the Intel Quick Assist Technology (Intel QAT) Engine for OpenSSL

Checkout the software repository from GitHub:

git clone https://github.com/intel/QAT_Engine.git
cd QAT_Engine

Next, ensure you are building against a fixed release of the code, and not a development branch. 

git checkout v1.2.0

To configure the Intel QAT Engine for OpenSSL for all software acceleration features:

./autogen.sh
./configure --enable-qat_sw

Then build and install:

make -j
sudo make install

Since this makes use of the distribution-provided OpenSSL installation, we won’t need to modify LD_LIBRARY_PATH to use the engine.

After the installation has completed, you should see the engine present in OpenSSL’s engine directory. In Ubuntu 22.04, this is in /usr/lib/x86_64-linux-gnu/engines-3. For non-standard builds, the engine directory can be obtained by running the “openssl version” command.

$ openssl version -e
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"

Verify that the engine is present by running “ls”. You should see qatengine.so in the directory list:

$ ls -l /usr/lib/x86_64-linux-gnu/engines-3
total 796
-rw-r--r-- 1 root root  23000 May 25 01:12 afalg.so
-rw-r--r-- 1 root root  52088 May 25 01:12 loader_attic.so
-rw-r--r-- 1 root root  26840 May 25 01:12 padlock.so
-rwxr-xr-x 1 root root    976 Aug  6 13:03 qatengine.la
-rwxr-xr-x 1 root root 703400 Aug  6 13:03 qatengine.so
 

 Building OpenSSL from Source

This section describes how to build the Intel QAT Engine for OpenSSL when OpenSSL is built from source code. If you want to build the engine using your distribution's pre-packaged version of OpenSSL, click the Using the Distribution-Provided OpenSSL tab, above.

Build Requirements

You'll need some prerequisite software packages to build OpenSSL, the Intel QAT Engine for OpenSSL, and the engine's dependencies.

Ubuntu 22.04 LTS

To build the Intel QAT Engine for OpenSSL and its dependencies on Ubuntu, you’ll need to install the following packages from apt:

sudo apt install autoconf build-essential libtool cmake cpuid pkg-config_nasm

 

RHEL 9.2
sudo yum install autoconf libtool cmake pkg-config nasm openssl perl
Runtime Requirements

To make use of the software acceleration features in the Intel QAT Engine for OpenSSL, you’ll need a system that supports Intel® AVX-512 with the following instruction set extensions:

  • AVX512F
  • AVX512_IFMA
  • VAES
  • VPCLMULQDQ

The latter two extensions were introduced with certain 10th Generation Intel® Core™ processors and 3rd Generation Intel® Xeon® Scalable processors (products formerly codenamed Ice Lake). A quick way to verify that your system supports the necessary features is to run the cpuid command. Run the following and check that the output matches.

$ cpuid -1 | egrep 'VAES|VPCLM|GFNI|AVX512F|AVX512IFMA'
      AVX512F: AVX-512 foundation instructions = true
      AVX512IFMA: fused multiply add           = true
      VAES instructions                        = true
      VPCLMULQDQ instruction                   = true

All features must be present.

These output fields are only present in cpuid version 20211210 or later. This is the default version provided in Ubuntu 22.04.

Choose a Directory Structure

Before we proceed with the build, we need to decide where to install the completed packages and libraries for both OpenSSL and the Intel QAT Engine for OpenSSL. Since the goal of building from source code is to produce a build that can be upgraded as needed without interfering with existing applications, we want a directory hierarchy that allows for parallel installations of multiple versions of the same tool. To keep the filesystem tidy, we'll place these packages in /opt using the following structure:

/opt/tool/version

Building OpenSSL v3.0.9

Example here uses OpenSSL 3.0.1. 

wget https://www.openssl.org/source/openssl-3.0.9.tar.gz 
tar xf openssl-3.0.9.tar.gz
cd openssl-3.0.9

To configure OpenSSL, run the config program and set the --prefix and --openssldir options to our desired installation directory, which will be /opt/openssl/3.0.9

./config --prefix=/opt/openssl/3.0.9 --openssldir=/opt/openssl/3.0.9

Then build and install:

make -j
sudo make install

Because we have installed OpenSSL into a non-standard build directory, we'll need to make some environment changes. To ensure you get this version of OpenSSL and not your system one, prepend the OpenSSL directory to your PATH variable:

export PATH=/opt/openssl/3.0.9/bin:$PATH

You also need to set LD_LIBRARY_PATH in your environment to run the binary, but for the purposes of this document, we will wait until all components are compiled and installed before doing so.

To verify OpenSSL is built and installed correctly, run the following:

$ LD_LIBRARY_PATH=/opt/openssl/3.0.9/lib64 openssl version -v -e
OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023)
ENGINESDIR: "/opt/openssl/3.0.9/lib64/engines-3"

You should see the correct version, and ENGINESDIR should be pointing to your installation in /opt/openssl.

You can set these environment variables in scripts to ensure that OpenSSL from the intended location.

If you don't set LD_LIBRARY_PATH, OpenSSL will load the equivalent libraries from the distribution's default location, resulting in a mismatch of library and binary versions. This will prevent the QAT engine from loading in OpenSSL, and it can also cause sporadic runtime errors in OpenSSL itself.

Building the Intel QAT OpenSSL Engine for Software Acceleration

The software acceleration support in the Intel QAT Engine for OpenSSL depends on the following two libraries. They must be built first, but they may be built in any order:

Once these libraries are installed, you can build the Intel Quick Assist Technology OpenSSL Engine.

We’ll step through how to build each one.

Building Intel® Integrated Performance Primitives Cryptography

First, checkout the source code repository from GitHub*:

git clone https://github.com/intel/ipp-crypto.git
cd ipp-crypto

Ensure you are building against a fixed release of the code, and not the development branch. 

git checkout ippcp_2021.7.1

You only need to build the multi-buffer portion of the Intel IPP package, so change to the multi-buffer crypto library subdirectory. Then, prepare the build by running cmake:

cd sources/ippcp/crypto_mb
cmake . -Bbuild -DCMAKE_INSTALL_PREFIX=/opt/crypto_mb/2021.7.1

Note that we are using crypto_mb as our tool name, since we aren't building the entire IPP package.

This library needs to know where to find your OpenSSL sources, so set OPENSSL_ROOT_DIR.

export OPENSSL_ROOT_DIR=/opt/openssl/3.0.9/

To perform the build, run:

cd build
make -j
sudo make install

We'll also need to update LD_LIBRARY_PATH to include this new library directory and will do so after all components have been compiled and installed.

Building the Intel® Multi-Buffer Crypto for IPsec Library

First, checkout the source code repository from GitHub:

git clone https://github.com/intel/intel-ipsec-mb.git
cd intel-ipsec-mb

Ensure you are building against a fixed release of the code, and not a development branch.

git checkout v1.3

There is no configuration step. Build the library using:

make -j

To install to our destination, define PREFIX when running "make install":

sudo make install NOLDCONFIG=y PREFIX=/opt/ipsec-mb/1.3

We'll also need to update LD_LIBRARY_PATH to include this new library directory and will do so after all components have been compiled and installed.

Building the Intel Quick Assist Technology Engine for OpenSSL

Checkout the software repository from GitHub:

git clone https://github.com/intel/QAT_Engine.git
cd QAT_Engine

Next, ensure you are building against a fixed release of the code, and not a development branch.

git checkout v1.2.0

To configure the Intel QAT OpenSSL engine for all software acceleration features, we need to do the following:

  • Set LDFLAGS and CPPFLAGS to ensure the Intel IPP Crypto and Multi-Buffer Library for IPSec libraries are found by configure. The options --with-qat_sw_crypto_mb_install_dir and --with-qat_sw_ipsec_mb_install_dir are provided for this purpose.
  • Supply the location of our OpenSSL library via the --with-openssl_install_dir option.
  • Add the --with-openssl_dir option, which points to the OpenSSL source code. This will regenerate the error files from OpenSSL's source.

​Here, we assume that you are building from your home directory. Be sure to update the path to match your build environment.

./autogen.sh
LDFLAGS="-L/opt/ipsec-mb/1.3/lib -L/opt/crypto_mb/2021.7.1/lib" CPPFLAGS="-I/opt/ipsec-mb/1.3/include -I/opt/crypto_mb/2021.7.1/include" ./configure --prefix=/opt/openssl/3.0.9 --with-openssl_install_dir=/opt/openssl/3.0.9 --with-openssl_dir=$HOME/openssl-3.0.9 --enable-qat_sw

To build and install, we also need to set PERL5LIB on the command line, so that Perl can find the configdata.pm file in the OpenSSL source directory. This step is necessary due to a bug in the QAT Engine build configuration, and it will be fixed in a future release.

PERL5LIB=$HOME/openssl-3.0.9 make -j
sudo PERL5LIB=$HOME/openssl-3.0.9 make install

After the installation has completed, you should see the engine present in OpenSSL’s engine directory:

ls -l /opt/openssl/3.0.9/lib64/engines-3/qatengine.*
-rwxr-xr-x 1 root root   1029 Aug  7 12:22 /opt/openssl/3.0.9/lib64/engines-3/qatengine.la
-rwxr-xr-x 1 root root 699672 Aug  7 12:22 /opt/openssl/3.0.9/lib64/engines-3/qatengine.so

Remember to set both LD_LIBRARY_PATH and modify your own PATH, or the following examples will fail. This can be done either in the user’s environment or the application’s environment. The example below demonstrates exporting the LD_LIBRARY_PATH in the user’s environment. Note that this method may impact other applications or libraries that are dependent on a specific versions of OpenSSL’s libCrypto or libSSL shared libraries delivered as part of the distribution.

export PATH=/opt/openssl/3.0.9/bin:$PATH
export LD_LIBRARY_PATH=/opt/openssl/3.0.9/lib64:/opt/crypto_mb/2021.7.1/lib:/opt/ipsec-mb/1.3/lib

To avoid impacting other applications or libraries dependent of older version of OpenSSL libraries, the safer option is to set LD_LIBRARY_PATH as part of the application’s environment which is shown in the following example.

LD_LIBRARY_PATH=/opt/openssl/3.0.9/lib64:/opt/crypto_mb/2021.7.1/lib:/opt/ipsec-mb/1.3/lib openssl version -e -v

 

Testing the Engine

Once the engine is in place, you can proceed with functionality tests.

The first test is to ensure the Intel QAT Engine loads correctly.

$ openssl engine -v qatengine
(qatengine) Reference implementation of QAT crypto engine(qat_sw) v1.2.0
     ENABLE_EXTERNAL_POLLING, POLL, ENABLE_HEURISTIC_POLLING,
     GET_NUM_REQUESTS_IN_FLIGHT, INIT_ENGINE, SW_ALGO_BITMAP

If the above command returns errors such as the following:

139667965596992:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:../crypto/dso/dso_dlfcn.c:118:filename(/usr/lib/x86_64-linux-gnu/engines-1.1/qatengine.so): libcrypto_mb.so: cannot open shared object file: No such file or directory
139667965596992:error:25070067:DSO support routines:DSO_load:could not load the shared library:../crypto/dso/dso_lib.c:162:
139667965596992:error:260B6084:engine routines:dynamic_load:dso not found:../crypto/engine/eng_dyn.c:414:
139667965596992:error:2606A074:engine routines:ENGINE_by_id:no such engine:../crypto/engine/eng_list.c:334:id=qatengine

If you are using the distribution-provided OpenSSL

  • Make sure the Intel® Multi-Buffer Crypto for IPsec Library and the Intel IPP CryptoMB Library are both installed into /usr/lib. If you did not set a prefix for the former, it will install into /usr/local and you’ll need to set LD_LIBRARY_PATH in your environment.

If you built OpenSSL from source

  • Make sure LD_LIBRARY_PATH is set to the paths for OpenSSL, the Intel Multi-Buffer Crypto for IPsec Library, and the Intel IPSec Library. All three paths must be present.
  • Verify your installation paths and make sure they are in /opt/tool/version

Assuming the engine loads correctly, you can test the software acceleration for each of the enabled algorithms. To do that, we'll run “openssl speed” on the individual algorithms and compare the engine performance to the baseline.

On Intel® Xeon® Scalable processor families, you must use processor affinity (also known as CPU pinning) to bind these processes to a core. The multi-buffer implementations make use of Intel AVX-512 features that produce internal power transitions, and if the CPU scheduler moves these jobs to other cores during execution then multiple power transitions will occur, countering performance gains. Most server applications support CPU affinity masks in some form, but for “openssl speed” we must rely on the taskset command to do this for us.

RSA

The RSA acceleration makes use of an asynchronous scheduling algorithm which Intel calls multi-buffer, which processes multiple operations in parallel. To test the accelerator performance, you must supply the -async_jobs parameter to “openssl speed”. On current Intel architectures, 8 asynchronous jobs deliver optimal performance.

While both sign and verify operations are accelerated, the largest gains are in signing. This translates to performance gains for servers processing TLS handshakes with RSA certificates.

2048-bit keys

Baseline taskset -c 1 openssl speed rsa2048
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 rsa2048

3072-bit keys

Baseline taskset -c 1 openssl speed rsa3072
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 rsa3072

4096-bit keys

Baseline taskset -c 1 openssl speed rsa4096
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 rsa4096

ECDH

Like RSA, the ECDH acceleration makes use of an asynchronous scheduling algorithm.

Montgomery EC Curve X25519

Baseline taskset -c 1 openssl speed ecdhx25519
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 ecdhx25519

NIST Curve P-256

Baseline taskset -c 1 openssl speed ecdhp256
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 ecdhp256

NIST Curve P-384

Baseline taskset -c 1 openssl speed ecdhp384
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 ecdhp384

ECDSA

The ECDSA algorithms also make use of asynchronous scheduling.

NIST Curve P-256

Baseline taskset -c 1 openssl speed ecdsap256
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 ecdsap256

NIST Curve P-384

Baseline taskset -c 1 openssl speed ecdsap384
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -async_jobs 8 ecdsap384

AES GCM Encryption

The AES GCM encryption acceleration is a purely vectorized implementation of the respective EVP ciphers. Key sizes of 128, 192, and 256 bits are supported but 192-bit encryption is almost never used in practice.

“Openssl speed” reports performance for several block sizes, but real-world applications tend to use buffers that are 8k or larger for increased efficiency and performance. This is also where the largest gains are seen.

128-bit keys

Baseline taskset -c 1 openssl speed -evp aes-128-gcm
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -evp aes-128-gcm

256-bit keys

Baseline taskset -c 1 openssl speed -evp aes-256-gcm
Intel QAT Engine for OpenSSL taskset -c 1 openssl speed -engine qatengine -evp aes-256-gcm