Ceph Deployment in Kubernetes with Intel® QuickAssist Technology

Intel® QuickAssist Technology (Intel® QAT) provides hardware acceleration for offloading security, authentication, and compression services from the CPU, thus significantly increasing the performance and efficiency of standard platform solutions. Its services include symmetric encryption and authentication, asymmetric encryption, digital signatures, Rivest–Shamir–Adleman (RSA), Diffie–Hellman (DH) and Elliptic-Curve Cryptography (ECC), and lossless data compression.

To save cycles, time, and space, Intel QAT can be used in various applications such as networking to enterprise, cloud to storage, and content delivery to database. As a typical example in cloud storage, Ceph is an open-source, distributed storage system that provides excellent performance, reliability, and scalability. Ceph supports object, block, and file storage, all in one unified RADOS cluster. Intel QAT-based encryption and compression have both been enabled in Ceph^[1], which has higher throughput than CPU-based encryption and compression. For more Intel QAT Technology resources please go to Intel QuickAssist Technology^[2].

Liu, Miaomiao

Containers give microservice-based apps an ideal application deployment unit and self-contained execution environment. They make it possible to run multiple parts of an app independently in microservices, on the same hardware, with much greater control over individual pieces and life cycles. By packaging applications in containers that can move across platforms and infrastructure, we can use the application anywhere to move containers because it has everything that successfully runs.

Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for Ceph storage to natively integrate with cloud-native environments. Rook automates deployment and management of Ceph to provide self-managing, self-scaling, and self-healing storage services. The Rook operator does this by building on Kubernetes resources to deploy, configure, provision, scale, upgrade, and monitor Ceph.^[3]

Deployment

This section illustrates an example to deploy Ceph in Kubernetes using Rook to perform an Intel QAT-based compressor test in this Ceph cluster. The deployment architecture is shown in Figure 1.

Figure 1. Architecture of Ceph cluster deployed in Kubernetes

Install the out-of-tree Intel QAT software package on the host

Intel® QuickAssist Accelerators that are integrated in the Intel QAT devices are accessed via Intel QAT APIs. Applications running in user space typically access the cryptography and compression services via the Intel QAT APIs.

There are two ways to build and install the Intel QAT APIs software:

Install the out-of-tree acceleration software package^[4];
Install the upstreamed acceleration software, which refers to the installation^[5] instructions at the Intel QuickAssist Technology Library (QATlib)^[6]repository. QATlib provides user space libraries that allow access to Intel QAT devices and expose the Intel QAT APIs and sample codes. The term in-tree refers to the Intel QAT kernel modules and firmware included in the Linux kernel tree, as opposed to our more common out-of-tree (OOT) driver and firmware, provided directly by Intel at Intel QuickAssist Technology Driver for Linux* - HW Version 2.0^[4].

We choose to install the OOT (out of tree) Intel QAT driver instead of upstreamed acceleration software for the following reasons:

The upstreamed acceleration software qatlib only supports 4xxx (Intel QAT fourth generation devices). Earlier generations of Intel QAT devices (e.g., c62x, dh895xxcc, etc.) are not supported.
The kernel version should be no older than 5.17. The qatlib features^[7] of data compression require a recent kernel driver or firmware version.

For Intel QAT hardware version 2.0, there is a corresponding software version that can be downloaded in Intel QAT drivers for linux-hw-version-2-0^[4].

Create a working directory for the software:

export ICP_ROOT=/the/path/to/QAT
mkdir -p $ICP_ROOT
cd $ICP_ROOT

Transfer the tarball to the system in the $ICP_ROOT directory. Unpack the tarball using the following command:

tar -zxof QAT.L.*.tar.gz

We recommend that you restrict access to the files

chmod -R o-rwx *

To setup the default build configuration, call:

./configure

Build and install:

make
make install

Start the acceleration software:

service qat_service start

Install QATzip

Compile and install QATzip from source code:

export QZ_ROOT=/the/path/to/QATzip
git clone -b v1.0.8 https://github.com/intel/QATzip.git $QZ_ROOT
cd $QZ_ROOT
./autogen.sh
./configure --with-ICP_ROOT=$ICP_ROOT
make clean
make all install
# run performance test with QATzip
cd $QZ_ROOT/test/performance_tests
./run_perf_test.sh

Enable hugepages since QATzip, which is based on the out-of-tree Intel QAT driver, uses hugepages to manage memory for Intel QAT buffers:

echo 9000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
cat /proc/meminfo | grep Huge
rmmod usdm_drv
insmod $ICP_ROOT/build/usdm_drv.ko max_huge_pages=8192 max_huge_pages_per_process=1024
systemctl restart qat_service.service

Then revise Intel QAT configuration files in the directory /etc/, and keep them the same. The section name of the Intel QAT configuration files must be SHIM.

*Figure 2: User Process Instance Section*

Refer to the Programmer’s Guide^[8] for additional information on the configuration files.

Then, set an environment variable to clarify the section name of User Process Instance Section in Intel QAT configuration files:

export QAT_SECTION_NAME=SHIM

Repeat the steps "Install the out-of-tree Intel QAT software package on the host" and "Install QAT.zip" on all nodes where OSDs are running.

Build the Ceph image

To utilize the Intel QAT device for Ceph compressor, we need to build a custom Ceph image used by Rook. Ceph’s CMake option WITH_QAT=ON and WITH_QATZIP=ON must be configured to trigger Intel QAT-based compression when building Ceph with version v17.2.5.

Use Docker build to build the Ceph image, and push it into the local registry:

docker build . -t ceph-1725:1

docker push localhost:5000/ceph-1725:1

Prepare a Kubernetes Cluster.

Here, a simple K8s cluster with two nodes has been deployed:

Deploy a Rook operator in the Kubernetes Cluster

Clone the Rook registry and deploy Rook Operator:

git clone --single-branch --branch v1.10.11 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

Revise the image download location in cluster.yml:

*Figure 4: Revise image download location*

Revise cm-rook-config-override.yaml to edit the Ceph configuration file, e.g.:

*Figure 5: Edit Ceph configuration file*

Create a Ceph cluster using Rook.

Check Rook Ceph Documentation^[9] for details.

kubectl create -f cm-rook-config-override.yaml
kubectl create -f cluster.yaml

*Figure 6: Ceph cluster creation using Rook*

kubectl create -f cm-rook-config-override.yaml
kubectl create -f cluster.yaml

*Figure 7: Override Ceph cluster settings*

Results and Summary

We performed a Ceph block compression test, and by using Ceph with Intel QAT-based compression, we were able to achieve the following:

High throughput. Intel QAT compression bandwidth is more than three times that of CPU software compression
Low CPU usage. CPU usage during Intel QAT compression is lower than CPU software compression