Introduction
The security and privacy of data has been the top priority for enterprise IT departments and cloud service providers throughout the industry. The datafication trend in the industry, combined with the fact that the compute and data owners may be different entities, has further accelerated efforts in this space, leading to the paradigm of confidential computing. Confidential computing extends the paradigm of protection of data at rest and in transit with protection of data in use where data in memory is protected during runtime of the workload.
Intel has been at the forefront of this effort since 2015 with the release of Intel® Software Guard Extensions (Intel® SGX) that provided applications the ability to protect their data from other platform code including software running at higher privilege levels like operating system and the hypervisor. With the 4th generation Intel® Xeon® Scalable processor (formerly code named Sapphire Rapids), Intel introduced Intel® Trust Domain Extensions (Intel® TDX), an architectural technology to deploy hardware-isolated virtual machines (VMs) called trust domains (TDs). Intel TDX alleviates the need for application changes for data protection and with Intel TDX TD partitioning architecture, it also provides the ability to run unmodified guest VMs as TDs.
This article describes the design goals of Intel TDX, provides an overview of the technology, and then reviews the performance of partitioned TDs based on an implementation with Microsoft Azure*.
Technical Overview
Intel TDX is designed to isolate TD VMs from the virtual machine manager (VMM), hypervisor, and other non-TD software on the host platform. Intel TDX is designed to enhance a platform user’s control of data security and IP protection. Intel TDX can also enhance the cloud service providers (CSP) ability to provide managed cloud services without exposing tenant data to adversaries.
Design and Security Goals
- Provide a hardware-isolated virtual machine based trusted execution environment (TEE) called TD. Hence, no application changes are required to get data protection.
- Provide isolation for TD from system software adversaries like administrative insiders (for example, DC admin, developer, or technician), VMM, SMM, BIOS, etc.
- Provide isolation for TD from hardware adversaries that can extract cloud tenant’s secrets or spoof tenant’s data/memory by hooking into system interfaces (DDR bus).
- Provide confidentiality and integrity protection for TDs.
- Provide ability to run nonconfidential legacy guests/VMs as TDs using TD partitioning, thus supporting ability to lift and shift customer workloads into the TEE.
The Intel TDX solution is built using a combination of Virtual Machine Extensions (VMX) instruction-set architecture (ISA) extensions, Intel® Total Memory Encryption – Multi-Key (Intel® TME-MK)1 technology, and a CPU-attested software module (Intel TDX module). It achieves the above design and security goals by providing:
- Memory and CPU state confidentiality and integrity to help keep the sensitive IP and workload data secure from most software-based attacks and many hardware based attacks.
- Remote attestation capability that enables a relying party (either the owner of the workload or a user of the services provided by the workload) to establish that the workload is running on an Intel TDX-enabled platform located within a TD prior to providing that workload data.
For details on Intel TDX architecture, refer to the Intel TDX white paper.2 We have also published the performance considerations of Intel TDX for enlightened guests.5
Intel TDX TD partitioning architecture3 further provides the ability to support an unmodified VM as a TD, thus providing CSPs with the ability to lift and shift workloads into hardware isolated VMs or trust domains (TDs) without having to make any changes to the guest operating system (OS). TD partitioning extends the base Intel TDX architecture by allowing TDs to contain multiple (up to 4) virtual machines (VMs). This allows an Intel TDX enabled L1 VM (primary VM) to run as L1 VMM and host nested unmodified VM (L2 VMs) on top of it, as shown in Figure 1.
Figure 1. Intel TDX base versus TD partitioning architecture
Note L1 VMM can support only one unmodified VM per its single instance (other L2 VMs must be enlightened). Intel TDX is enabled in Windows* with Microsoft Hyper-V* and is deployed in Microsoft Azure* confidential computing as part of the new Intel TDX-based DCesv5 and ECesv5-series confidential VMs.4
The rest of this article provides the performance analysis for Intel TDX based on an Azure implementation, specifically for partitioned TDs, or legacy Windows and Linux* guests running on Hyper-V.
Security-Related Performance Considerations
To help protect the private memory and CPU state of the TD from the host VMM, BIOS, and any platform software, Intel TDX relies on the following solutions to aid in ensuring data confidentiality and integrity of partitioned TDs.5
Intel TDX Transitions
While VMM continues to be the resource manager for the TDs, the Intel TDX module is used to help enforce the security policies for the TDs. When a TD vCPU chooses to exit to host VMM, the Intel TDX module saves the CPU register state and scrubs these registers before transferring execution control to the VMM. When the TD is subsequently resumed, the CPU state of the TD vCPU is restored.
Additionally with TD partitioning, inter-VM transitions always happen within the same TD vCPU.
- A vCPU running in L2 VM normally exits to the L1 VMM but can also choose to exit to the host VMM directly.
- The L1 VMM can initiate a reentry back to L2 VM or exit to host VMM.
- Likewise, the host VMM may perform reentry to L2 VM or to L1 VMM.
These state transitions are crucial in ensuring confidentiality and integrity of the TD vCPU state and may affect TD performance.
Memory Encryption
Intel TDX uses the Intel TME-MK engine to enable memory encryption and decryption with additional enhancements for integrity protection. This can translate to additional memory access latency and corresponding effect on memory bandwidth for read/write operations. Also, cross-CPU socket memory access going through the Intel® Ultra Path Interconnect (Intel® UPI) are encrypted and can incur additional latencies.
For host VMM and any non-TD software, the Intel TME-MK architecture used by Intel TDX supports a "TME-Bypass" mode to bypass memory encryption to avoid the memory encryption/decryption overhead.
I/O Virtualization
TD memory is considered private and not accessible externally by the VMM or devices. However, during I/O operations like network or file I/O over VMBus or similar inter-partition communication channel, the TD operating system exchanges data with VMM or host OS over “shared pages” that were explicitly made visible to the VMM. Guest operating systems for TD typically maintain a pool of shared pages and allocate bounce buffers from that pool for I/O operations. When the data is ready to be exchanged with the VMM, the data is copied in and out of this shared memory region in order to keep the TD memory secure. These copy operations may affect the TD I/O performance.
To assess TD performance, we consider a mix of compute, memory, and I/O intensive workloads. In each scenario, the performance of TD is compared against a nonconfidential legacy VM for Windows and Linux guests with similar workloads.
Configuration
Platform Configuration
The system used for performance assessments is a 4th generation Intel Xeon Scalable processor with Intel TDX features in 2 sockets configuration, 52 physical cores per socket with hyperthreading and turbo enabled. The system has 1 TB of memory with 16 x 64 GB 4800 MHz DDR5 DIMMs. The host operating system is Microsoft Hyper-V preview build 25954. The workloads are run inside Hyper-V generation 2 Windows and Linux guests, each configured with 16 vCPUs and 64 GB memory. The Windows guest runs Windows Server* preview build 25954. The Linux guest runs Ubuntu* 22.04 with custom Linux 6.2 kernel (pending upstream) with Intel TDX support.
Guest Classifications and Terminology
The following are the different guest configurations used in this study:
- Legacy VM: Nonconfidential guest VM running on a platform where Intel TDX and Intel TME-MK features are disabled in the BIOS.
- Partitioned TD: Hardware-isolated confidential VM referred to as trust domains (TDs) running in partitioned mode. Intel TDX feature is enabled in BIOS and host OS/VMM.
Performance Results
In this section, we discuss two types of workloads:
- CPU/memory intensive workload: SPECRATE* 2017 integer and floating point benchmark suite tested on Windows and Linux guests. Units of operations considered is rate of operations.
- I/O intensive workloads: Windows guest running Internet Information Services (IIS) for Windows Server stressed with Web Capacity Analysis Tool (WCAT) and Linux guest running NGINX* WordPress* web server stressed with Seige. These workloads stress the CPU with moderate to high disk and network activity depending on load level.
We consider a ratio of workload throughputs between TD and nonconfidential VMs to quantify the effect of Intel TDX on performance. For each of the benchmark results described below, the tests are performed three times to measure any run-to-run variations in results.
Note These are early results from prerelease software builds and further improvements in performance are expected with the final release with continued software optimizations.
CPU/Memory Intensive Workloads
SPEC CPU* 2017 benchmark is an industry-standardized, CPU intensive suites for measuring and comparing compute intensive performance, stressing a system’s processor, memory subsystem, and compiler. The SPEC binaries used in the study are compiled using Intel® oneAPI DPC++/C++ Compiler 2022.1 (for Windows) and GCC 12.1 (for Linux) with Intel-optimized compiler options.
Protecting the workload with confidential computing on Windows partitioned TD versus legacy VM shows a 2% performance difference with SPECRATE 2017 integer est. and 3% difference with SPECRATE 2017 floating point est. geometric mean scores. The performance difference observed here can be attributed to overhead from memory encryption and Intel TDX transitions.
Figure 2. Windows guest SPEC CPU 2017 est. performance on Windows partitioned TD versus legacy VM
On Linux partitioned TDs, we observe a 3% performance difference with SPECRATE 2017 integer est. and 3% with SPECRATE 2017 floating point est. in geometric mean scores compared to Linux legacy VM.
Figure 3. Linux guest SPEC CPU 2017 est. performance on Linux partitioned TD versus legacy VM
I/O Intensive Workloads
Windows Guest Running IIS Web Server
Internet Information Services (IIS) is a general-purpose web server from Microsoft and a popular choice for hosting websites and web applications on Windows. In this study, the guest VM under test is configured with an IIS web server role and it is stressed using WCAT (Web Capacity Analysis Tool), a lightweight HTTP load generation tool, with two additional nodes acting as WCAT clients. The WCAT clients are configured to use a “FullMix” profile, which simulates a mix of static and dynamic content requests comprising images, CSS and JavaScript files, and ASP.NET pages. The result measures the throughput in transactions/sec reported as relative comparison of partitioned TD versus legacy VM.
With a higher I/O request rate, there is a corresponding increasing effect on Intel TDX performance. In this study, we test two load scenarios: 100% load and 60% load, calibrated based on legacy VM vCPU utilization. For the 100% load scenario, the web requests from the clients are driven to simulate a high-volume traffic and understand a typical worst-case performance of a web server under heavy load. In this scenario, we observe around 15% performance difference in throughput with the Windows partitioned TDs compared to legacy VM. This difference is primarily from additional memory copies involved with bounce buffer usage and Intel TDX transitions. Under the lesser CPU load, we observe 6% performance difference with partitioned TDs due to availability of more CPU headroom to handle the additional memory copies.
Figure 4. IIS Web Server performance on Windows partitioned TD versus legacy VM
Linux Guest Running NGINX WordPress Web Server
NGINX is a popular open source, event-driven web server and reverse proxy. It is also a popular choice for load balancing of HTTP and TCP/UDP traffic. In this study, we use a LEMP (Linux-NGINX-MySQL-PHP) stack with NGINX configured as a web server running inside the guest VM. The WordPress back end is serviced by up to 20 PHP-PFM processes with a MySQL database. The web server is tested using Seige, an http/https testing and benchmarking tool, running from two separate physical client machines in the same subnet. The cipher suite tested in this study is TLS_AES_256_GCM_SHA384 representing TLS 1.3. The results are reported as throughput measuring web requests completed per second.
Similar to the IIS Web Server study, we test the NGINX WordPress server at 100% load to understand the worst-case TD performance. For the moderate to high load scenario, we use 70% load to present a typical use case. Both load points are calibrated based on legacy VM vCPU utilization. At 100% load, we observe 9% difference in throughput with the Linux partitioned TDs versus legacy VM. At 70% load, additional CPU headroom is available to handle bounce buffers that are used to keep TD memory secure. So, we observe only 6% throughput difference with the Linux partitioned TDs, which can be primarily attributed to the Intel TDX transitions and memory encryption.
Figure 5. NGINX WordPress performance on Linux partitioned TD versus legacy VM
Conclusion
As importance of runtime data protection becomes more critical with trends like datafication, more IT departments and cloud service providers are moving toward the paradigm of confidential computing as the default method for deploying customer workloads and protecting enterprise data. Intel TDX technology introduced in 4th generation Intel Xeon Scalable processors provides a way for customers to easily adopt confidential compute and take advantage of the data protection features that it provides. Intel TDX TD partitioning architecture further provides a way to lift and shift legacy customer workloads into trust domains and is deployed by key customers like Azure. While this technology provides great flexibility and ease of enabling (no changes to guest VMs), this paper addresses the performance considerations of TD partitioning. Based on our performance analysis for Intel TDX deployment in Azure, we see that 2%-3% performance effect of Intel TDX on CPU and memory intensive workloads (SPEC CPU) and varying but higher range for I/O intensive workloads (WordPress and IIS web server). We expect software optimizations to further mitigate the performance impact of Intel TDX in future deployments along with Intel® Trust Domain Extensions Connect (Intel® TDX Connect)6 to maximize I/O throughput in the future.
References
3 Intel TDX TD Partitioning Architecture Specification
4 Dcesv5 and Ecesv5 Series VMs in Azure
5 Intel TDX Performance Considerations for Enlightened Guests
6 Intel TDX Connect Architecture Specification
Notices and Disclaimers
Code names are used by Intel to identify products, technologies, or services that are in development and not publicly available. These are not "commercial" names and not intended to function as trademarks.
Performance varies by use, configuration, and other factors. Learn more on the Performance Index site. Real world performance impacts may vary, and any stated figures herein are estimates.
For workloads and configurations, see the following resource. Results were not audited and may vary. Intel TDX is supported on select 4th Generation Xeon Scalable Processors.
System Configuration
System under test:
Host: 52-core 4th Generation Intel Xeon Scalable Processor with Intel® TDX feature in 2 socket configuration with 1TB memory using 16 x 64 GB 4800 MHz DDR5 DIMMs, Intel SSDSC2KB019T8 OS drive, high-performance Intel SSDPECKE064T8 NVMe data drive, Mellanox Connect-X 5 (40Gbps) NIC, Hyperthreading ON, Turbo ON, Directory Mode ON. Host OS: Microsoft Hyper-V preview build 25954, power policy set to high-performance mode.
The BIOS changes for the different guest configurations are as follows:
- Legacy VM: TME OFF, TME-MK OFF, TDX OFF, SGX OFF.
- Partitioned TD: TME ON, TME-MK ON, TME Bypass ON, TDX ON, SGX ON.
Guest: 16 vCPU, 64 GB memory Hyper-V generation 2 VM. The guest operating systems used for legacy VM and partitioned TD are as follows:
- Windows: Windows Server Hyper-V preview build 25954.
- Linux: Ubuntu 22.04 with custom Linux 6.2 kernel (pending upstream). Patchset: https://github.com/dcui/linux/tree/decui/Ubuntu-azure-6.2-6.2.0-1007.7_22.04.1/tdx/HCL-v1-2023-0710
Workload client system (used for IIS Web Server and NGINX WordPress workloads):
2nd generation Intel Xeon Scalable Processor (formerly code named Cascade Lake) using Intel Xeon Gold 6252 Processor in a two-socket configuration, 128 GB platform memory, Intel SSDSC2BA800G3 OS drive, Intel XL710-Q2 (40 Gbps) NIC. BIOS configurations: Hyperthreading ON, Turbo ON.
Workload Configuration
- SPEC CPU: SPEC CPU 2017 compiled with Intel® oneAPI DPC++/C++ compiler 2022.1 (for Windows) and gcc compiler 12.1 (for Linux) with Intel optimized compiler options. The tests are conducted using SPEC-Rate suite. Tests were conducted in October 2023. The results were not audited.
- IIS Web Server: Microsoft IIS Web Server role installed from Windows build (version 10.0.25954.1000). The clients used for testing Web Server performance is WCAT (v6.5). Tests were conducted in October 2023.
IIS IIS Web Server Overview
Download WCAT 6.3 (x64)
- NGINX-WordPress: The software stack used here represents a typical LEMP stack on the server side using asynch_mode_nginx (v0.5.1), PHP-FPM (v7.4), MySQL (v 8.0.34) and WordPress (v5.9.3). The clients used to test LEMP performance is siege (v3.1.4). Tests were conducted in October 2023.
NGINX
PHP-PFM
MySQL
WordPress
Seige