Delivering a performance boost to high-priority applications through core pinning and isolation
Overview
Under normal circumstances, the kernel task scheduler will treat all CPUs as available for scheduling process threads and regularly preempts executing process threads to give CPU time to other applications. The positive side effect is multitasking and more efficient CPU resource utilization. The negative side effect is non-deterministic behavior which makes it unsuitable for latency sensitive workloads. A solution for optimizing these workloads performance is to “isolate” a CPU, or a set of CPUs, from the kernel scheduler, such that it will never schedule a process thread there. Then, the latency sensitive workload process thread(s) can be pinned to execute on that isolated CPU set only, providing them exclusive access to that CPU set. This results in more deterministic behavior due to reduced or eliminated thread preemption and maximizing CPU cache utilization. While beginning to guarantee the deterministic behavior of priority workloads, isolating CPUs also addresses the need to manage resources permitting multiple VNFs to coexist on the same physical server.
In Kubernetes* (as of v1.18), CPU and Memory are the only first class resources managed by Kubernetes with the native CPU manager. CPU is requested in terms of “MilliCPU”, which translates to a guaranteed time slice on a CPU effectively allowing the kernel task scheduler to act as normal. However, as mentioned above, this results in non-deterministic behavior. The Kubernetes community, Intel included, are continuing to work to develop support CPU allocation in order to provide deterministic behavior to priority workloads.
While Kubernetes continues to evolve its support for these capabilities, Intel has created the open source solution CPU Manager for Kubernetes.
Design and Architecture
CPU Manager for Kubernetes is the interim solution to CPU pinning and isolation for Kubernetes while the native CPU Manager is being developed. CPU Manager for Kubernetes contains features that the native CPU Manager does not, specifically isolcpus. It ships with a single multi-use command-line program to perform various functions for host configuration, managing groups of CPUs, and constraining workloads to specific CPUs. CPU Manager for Kubernetes divides up the CPUs on a system into three pools by default with an additional optional pool. In terms of the default setup of three pools and when a certain number of the CPUs are isolated using isolcpus, the CPUs are broken up into pools according to exclusivity. The extra optional pool is used in cases where a user wants a process isolated from other processes on the system but cannot be placed on cores that are a subset of isolcpus. More details about this additional pool can be found in the CPU Management - CPU Pinning and Isolation in Kubernetes* Technology Guide.
More information about the Native CPU Manager can be found at: Feature Highlight: CPU Manager.
Installation and Usage
1. Installing CPU Manager for Kubernetes* starts with cloning the following Intel GitHub link:
#git clone https://github.com/intel/CPU-Manager-for-Kubernetes
2. From inside the cloned repository, the CPU Manager for Kubernetes* Docker images is built:
#cd
#make
Note The CPU Manager for Kubernetes* image needs to be available on each node in the Kubernetes cluster where CPU Manager for Kubernetes* will be deployed. CPU Manager for Kubernetes* uses RBAC and service accounts for authorization.
3. Deploy the following yaml files:
#kubectl create –f cmk-rbac-rules.yaml
#kubectl create –f cmk-serviceaccount.yaml
4. Use the isolcpus boot parameter to ensure exclusive cores in CPU Manager for Kubernetes*are not affected by other system tasks:
#isolcpus=0,1,2,3
a. On an Intel® HT Technology system, fully “isolate” a core by isolating the hyper-thread siblings.
b. At a minimum, the number of fully isolated cores should be equal to the desired number of cores in the exclusive pool.
c. CPU Manager for Kubernetes* will work without the isolcpus set but does not guarantee isolation from system processes being scheduled on the exclusive data plane cores.
5...
Example
The isolate command is used to pin a workload to a core in the requested pool. CPU Manager for Kubernetes* uses the binary installed as part of the deployment process to act as a wrapper to the workload and runs before it. This allows CPU Manager for Kubernetes* to pin the parent process to the allocated core and to clean up on termination.
An example isolate command requesting a core on the exclusive pool:
#/opt/bin/cmk isolate --conf-dir=/et/cmk --pool=exclusive sleep inf
An example isolate command for a pod requesting an exclusive core on the data plane core.
Reference
Enhanced Platform Awareness in Kubernetes Feature Brief
Enhanced Platform Awareness in Kubernetes Application Note
Enabling New Features with Kubernetes for NFV White Paper
Enhanced Platform Awareness in Kubernetes Performance Benchmark Report
CPU Manager for Kubernetes repository
Intel® Speed Select Technology - Base Frequency (Intel® SST-BF) with Kubernetes* Application Note