Ceph logo

Ceph Deployment in Kubernetes with Intel® QuickAssist Technology-based Compression



Containers give microservice-based apps an ideal application deployment unit and self-contained execution environment. They make it possible to run multiple parts of an app independently in microservices, on the same hardware, with much greater control over individual pieces and life cycles. By packaging applications in containers that can move across platforms and infrastructure, we can use the application anywhere to move containers because it has everything that successfully runs.

Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for Ceph storage to natively integrate with cloud-native environments. Rook automates deployment and management of Ceph to provide self-managing, self-scaling, and self-healing storage services. The Rook operator does this by building on Kubernetes resources to deploy, configure, provision, scale, upgrade, and monitor Ceph.[3]



This section illustrates an example to deploy Ceph in Kubernetes using Rook to perform an Intel QAT-based compressor test in this Ceph cluster. The deployment architecture is shown in Figure 1.

Figure 1. Architecture of Ceph cluster deployed in Kubernetes



Install the out-of-tree Intel QAT software package on the host

Intel® QuickAssist Accelerators that are integrated in the Intel QAT devices are accessed via Intel QAT APIs. Applications running in user space typically access the cryptography and compression services via the Intel QAT APIs.

There are two ways to build and install the Intel QAT APIs software:

  1. Install the out-of-tree acceleration software package[4];
  2. Install the upstreamed acceleration software, which refers to the installation[5] instructions at the Intel QuickAssist Technology Library (QATlib)[6] repository. QATlib provides user space libraries that allow access to Intel QAT devices and expose the Intel QAT APIs and sample codes. The term in-tree refers to the Intel QAT kernel modules and firmware included in the Linux kernel tree, as opposed to our more common out-of-tree (OOT) driver and firmware, provided directly by Intel at Intel QuickAssist Technology Driver for Linux* - HW Version 2.0[4].

We choose to install the OOT (out of tree) Intel QAT driver instead of upstreamed acceleration software for the following reasons:

  1. The upstreamed acceleration software qatlib only supports 4xxx (Intel QAT fourth generation devices). Earlier generations of Intel QAT devices (e.g., c62x, dh895xxcc, etc.) are not supported.
  2. The kernel version should be no older than 5.17. The qatlib features[7] of data compression require a recent kernel driver or firmware version.

For Intel QAT hardware version 2.0, there is a corresponding software version that can be downloaded in Intel QAT drivers for linux-hw-version-2-0[4].

Create a working directory for the software:

export ICP_ROOT=/the/path/to/QAT
mkdir -p $ICP_ROOT

Transfer the tarball to the system in the $ICP_ROOT directory. Unpack the tarball using the following command:

tar -zxof QAT.L.*.tar.gz

We recommend that you restrict access to the files

chmod -R o-rwx *

To setup the default build configuration, call:


Build and install:

make install

Start the acceleration software:

service qat_service start


Install QATzip


Compile and install QATzip from source code:

export QZ_ROOT=/the/path/to/QATzip
git clone -b v1.0.8 https://github.com/intel/QATzip.git $QZ_ROOT
./configure --with-ICP_ROOT=$ICP_ROOT
make clean
make all install
# run performance test with QATzip
cd $QZ_ROOT/test/performance_tests


Enable hugepages since QATzip, which is based on the out-of-tree Intel QAT driver, uses hugepages to manage memory for Intel QAT buffers:

echo 9000 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
cat /proc/meminfo | grep Huge
rmmod usdm_drv
insmod $ICP_ROOT/build/usdm_drv.ko max_huge_pages=8192 max_huge_pages_per_process=1024
systemctl restart qat_service.service


Then revise Intel QAT configuration files in the directory /etc/, and keep them the same. The section name of the Intel QAT configuration files must be SHIM.

Figure 2: User Process Instance Section



Refer to the Programmer’s Guide[8] for additional information on the configuration files.

Then, set an environment variable to clarify the section name of User Process Instance Section in Intel QAT configuration files:


Repeat the steps "Install the out-of-tree Intel QAT software package on the host" and "Install QAT.zip" on all nodes where OSDs are running.


Build the Ceph image

To utilize the Intel QAT device for Ceph compressor, we need to build a custom Ceph image used by Rook. Ceph’s CMake option WITH_QAT=ON and WITH_QATZIP=ON must be configured to trigger Intel QAT-based compression when building Ceph with version v17.2.5.

Use Docker build to build the Ceph image, and push it into the local registry:

docker build . -t ceph-1725:1

docker push localhost:5000/ceph-1725:1


Prepare a Kubernetes Cluster.

Here, a simple K8s cluster with two nodes has been deployed:

Figure 3: Simple Kubernetes Cluster


Deploy a Rook operator in the Kubernetes Cluster

Clone the Rook registry and deploy Rook Operator:

git clone --single-branch --branch v1.10.11 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml


Revise the image download location in cluster.yml:

Figure 4: Revise image download location


Revise cm-rook-config-override.yaml to edit the Ceph configuration file, e.g.:

Figure 5: Edit Ceph configuration file


Create a Ceph cluster using Rook.

Check Rook Ceph Documentation[9] for details.

kubectl create -f cm-rook-config-override.yaml
kubectl create -f cluster.yaml


Figure 6: Ceph cluster creation using Rook


kubectl create -f cm-rook-config-override.yaml
kubectl create -f cluster.yaml


Figure 7: Override Ceph cluster settings


Results and Summary

We performed a Ceph block compression test, and by using Ceph with Intel QAT-based compression, we were able to achieve the following:

  • High throughput. Intel QAT compression bandwidth is more than three times that of CPU software compression
  • Low CPU usage. CPU usage during Intel QAT compression is lower than CPU software compression



[1] https://docs.ceph.com/en/latest/radosgw/qat-accel/

[2] https://www.intel.com/content/www/us/en/developer/topic-technology/open/quick-assist-technology/overview.html

[3] https://rook.io/


[5] https://github.com/intel/qatlib/blob/main/INSTALL

[6] https://github.com/intel/qatlib

 [7] https://intel.github.io/quickassist/RN/In-Tree/in_tree_firmware_RN.html#qat-kernel-driver-releases-features

[8] https://intel.github.io/quickassist/GSG/2.X/installation.html#configuration-files

[9] https://rook.io/docs/rook/latest-release/Getting-Started/quickstart/