Introduction
This whitepaper introduces a solution to protect the runtime integrity of a VM based on Intel® Trust Domain Extensions (Intel® TDX) using Integrity Measurement Architecture (IMA) technology and the related attestation process. It includes a discussion of the underlying technology leveraged, the enablement of runtime measurement inside a Trusted Domain guest VM (TDVM), and several use cases in the field.
Intel® TDX Overview
Intel TDX refers to an Intel technology that extends virtual machine extensions (VMX) and Intel® Total Memory Encryption – Multi-Key (Intel® TME-MK), with a new kind of virtual machine guest called a trust domain (TD). A TD runs in a CPU mode designed to protect the confidentiality of its memory contents and CPU state from any other software, including the hosting virtual machine monitor (VMM) [1].
Figure 1 illustrates the high-level software stack for running a TD on a platform, which includes three major components: an Intel TDX loader and an Intel TDX module, with an Intel TDX enlightened VMM. The Intel TDX module provides a secure environment and structure to protect some resources used by the Intel TDX guest VM. For detailed information on Intel TDX and each component, please refer to [1]. The memory and state of the VM is protected from the underlying software components and other VMs.
Figure 1. Intel® TDX
The introduction of Intel TDX provides users with a security solution to address data confidentiality and integrity challenges in a modern data center from the platform perspective. However, as the industry has quickly seized on the value of confidential computing technologies, users have started to pay more attention to the integrity of their workload at runtime. To extend the trust supply chain from hardware to software during runtime, we chose the Linux* kernel Integrity Measurement Architecture (IMA) to extend the chain of trust for measurement and further prove the integrity of the programs and workload in this confidential runtime environment.
Integrity Measurement Architecture (IMA)
IMA is the Linux kernel functionality used to achieve system integrity during runtime. It calculates the hashes of files and programs before they are loaded, supports reporting on them, and validates that they adhere to a predefined list [2]. Since IMA adheres to the Trusted Computing Group (TCG) open standards, it can work with the hardware Trusted Platform Module (TPM) to maintain the aggregate integrity value over the runtime measurement list.
IMA was added to the Linux kernel as part of the Linux kernel integrity subsystem developed in version 2.6.30. It introduced several hooks within the Linux kernel to intercept behaviors, including file read, file mmap, etc., and supports creating and collecting hashes of opened files before their contents are accessed for reading or execution.
IMA includes two subsystems: Measurement and Appraisal. Measurement takes responsibility for collecting, storing, and attesting measurements. Appraisal, introduced in kernel version 2.6.36, does the appraisal and audit work by comparing a collected hash to a stored hash and denies access in the event of a mismatch.
Figure 2. IMA workflow
IMA Usage in Runtime Integrity Protection
As a critical security component in the kernel for ensuring the integrity of the running system, IMA attracts much attention from the community. Multiple research papers are related to leveraging IMA for greater fine-grained runtime integrity protection or addressing issues with container-level runtime integrity protection [2].
Multiple projects leverage IMA to protect runtime integrity in their scenarios. Keylime*, a Cloud Native Computing Foundation (CNCF) sandbox project, provides a highly scalable measured boot attestation and runtime integrity measurement solution. Keylime uses a TPM and the Linux IMA to provide a remote attestation platform, secure payload delivery, and a revocation framework to give users full control over attestation policy and security attack remediation. OSTree, an upgrade system for Linux-based operating systems that performs atomic upgrades of complete filesystem trees, also leverages IMA to enhance security while monitoring and updating the kernel. Operating systems such as Google* Container-Optimized OS [3], Oracle Podman*, OpenAtom OpenEuler* [4], and OpenSuse* [5] all provide IMA support to enhance runtime integrity protection.
IMA Usage in Confidential Computing
Confidential computing necessitates stringent security measures and increased monitoring to safeguard sensitive data during processing. The attestation process in confidential computing proves the properties of the system, mostly its Trusted-Computing Base (TCB), to a third party by providing proof that the execution environment, usually a hardware Trusted Execution Environment (TEE), can be trusted before starting to execute code or before delivering any secret information. While attestation can verify the platform's trustworthiness, it primarily concentrates on measurements taken during boot or launch time. There is a notable gap in runtime integrity measurement, and the IMA framework in the Linux kernel appears to be the solution to bridge this gap.
Research has been conducted on integrating IMA and Arm TrustZone* to support runtime integrity measurement for applications located in edge appliances, client appliances, and other devices [6]. In data centers, it is also necessary to establish runtime security protection for hosts, clusters, and applications. Therefore, we introduced IMA in Intel TDX to validate that the programs and applications running on the server are measurable and verifiable during runtime. This will help build the trust supply chain on a higher level, such as for confidential clusters and SaaS applications.
Runtime Measurement Enablement in Intel TDX
IMA Enablement in Intel TDX
Traditional TDVM does not yet support TPM, while the existing implementation of IMA depends on it. This paper discusses our innovative approach to enable IMA in TDVM without TPM. To enable runtime measurement in TDVM, we leverage the existing attestation procedure in Intel TDX. Each TD is provided with four runtime measurement registers (RTMR). The implementation of RTMR is the same as PCR, as defined in the TCG standard. During boot time, measurements can be extended in the RTMR of the TDVM, which will be included as part of the attestation quote. We reuse the RTMR register to store the aggregate integrity value of the runtime measurement list from IMA.
To enable IMA with RTMR, changes to the kernel are necessary. The current IMA implementation is tightly bound to the TPM if the aggregate value is to be anchored in a trusted hardware module. To make IMA work with Intel TDX, we introduced a separate RTMR driver in the kernel to hold the SHA384 bank (currently, SHA384 is the only hash algorithm supported by Intel TDX RTMR) for hash calculations. In addition, we added a new ima_rtmr_extend function to IMA to extend the measurement to RTMR [2] instead of using the Platform Configuration Register (PCR) [10] in TPM. The event logs for the measurement list will be stored in the Confidential Computing Event Log (CCEL) table.
Figure 3. Critical changes for IMA enablement in TDX
For the measurement list saved in the kernel securityfs, SHA1 is replaced by SHA384 as the default hash algorithm to calculate the template hash, which is stored in /sys/kernel/security/integrity/ima/ascii_runtime_measurements.
IMA itself will operate in the same way as it does natively after being enabled with RTMR. For information on how to use IMA, please refer to the IMA documents [9].
Our plan for enabling IMA in Intel TDX includes support for the Virtual Trusted Platform Module (vTPM). Once the vTPM feature is ready, IMA can be supported natively in TDVM with no changes to the kernel.
Policy configurations
IMA policy is the key component to determine the measurement scope when using IMA for runtime measurement. The Linux kernel already provides several built-in policies for users to choose from. The five built-in policies are [7]:
- tcb – measures all programs exec’d, files mmap’d for exec, and all files opened with the read mode bit set by either the effective uid (euid=0) or uid=0.
- appraise_tcb – appraises the integrity of files owned by root.
- secure_boot – appraises the integrity files (for example, kexec kernel image, kernel modules, firmware, policy, etc.) based on file signatures.
- fail_securely – forces file signature verification failure on privileged mounted filesystems with the SB_I_UNVERIFIABLE_SIGNATURE flag.
- critical_data – measures kernel integrity critical data.
The built-in policies seem coarse-grained for specific usage in runtime measurement. The Linux kernel also provides several keywords to configure the rules. Details can be found at Kernel ima_policy ABI [8]. After defining the custom policy, the user must load the policy into /sys/kernel/security/integrity/ima/policy, either in an initramfs or early in the boot process through an init script in the sysinit run level [9]. Once the policy has been applied, IMA will perform integrity monitoring based on the custom policy.
Runtime Integrity Attestation
Keylime* with IMA
As introduced in the section on IMA usage in confidential computing, Keylime is an existing cloud native project which is an end-to-end solution for bootstrapping hardware-rooted cryptographic trust for remote machines, the provisioning of encrypted payloads, and run-time system integrity monitoring [10]. To monitor the run-time integrity of the system, Keylime leverages IMA to track all the events happening within the cloud/edge environment and compare the measurements against its allowlist. Once there is a potential security breach from a certain node, Keylime will alert the users and revoke the node from the cluster. This mechanism can also be leveraged in the confidential computing environment.
Keylime leverages IMA to generate runtime measurements for confidential clusters and requires attestation for runtime integrity validation of the system. However, Keylime does not include a service to verify that a VM runs in a TEE. Therefore, other solutions or services are needed for TEE attestation.
Attestation with Runtime Measurements Based on Intel® Trust Authority
The process of attestation is crucial in confidential computing, as it establishes the platform’s reliability. Most hardware TEEs now provide mechanisms for attestation against launch-time/boot-time measurements, but they rarely provide attestation against runtime measurements. Utilizing IMA, Intel TDX could achieve attestation against runtime measurements and verify the runtime states of the system. To achieve this, customers currently require an attestation service such as Intel® Trust Authority.
Intel Trust Authority is a SaaS-based attestation service developed by Intel to provide remote verification of the trustworthiness of a compute asset based on policy. Starting with an Intel trusted execution environment, it may expand to broader device verification, including IPUs, GPUs, platform roots of trust, and more. It is architected as a cloud-native microservice platform running on a managed Kubernetes service, with appropriate abstractions on different cloud infrastructure platforms, on-prem, and edge locations [11]. After finishing the verification, the user will receive the Intel Trust Authority’s endorsement regarding the remote platform's validity and corresponding trust level.
To collect the evidence for attestation (TDVM quote), hardware vendor-provided SDKs must be integrated into the customer workload. To address the extra work or complexity such integration requires of customers, Intel Trust Authority provides trustauthority-client, an evidence-fetching tool that interfaces with the TEE software stack and hardware to retrieve the evidence on behalf of the customer. Currently, trustauthority-client works as a full-fledged tool that mainly helps customers to retrieve quotes from an Intel® Software Guard Extensions (Intel® SGX) or Intel TDX platform. Additional functions/flow allow network connections to a designated Intel Trust Authority endpoint for attestation requests, configuration of Intel Trust Authority API keys for different use case scenarios, and secret provisioning through trustauthority-client’s robust crypto implementations. In the near future, the trustauthority-client will cover additional TEE variants, along with Intel Trust Authority SaaS service expansion.
Figure 4. Intel Trust Authority workflow
Intel Trust Authority’s trustauthority-client provides a CLI tool which can be easily integrated with bootstrap scripts to perform the attestation before accessing valuable digital assets remotely. As the security guarantee of the client is highly desired, we need to build trustauthority-client into the chain of trust that can also be verified and validated by Intel Trust Authority. This can be done in different ways on different platforms, and leveraging IMA runtime measurements is the most straightforward implementation, as it has already been supported in the Linux stack for Intel TDX.
By setting up corresponding IMA policies to cover the designated path and file owner of the Intel Trust Authority’s trustauthority-client, IMA can automatically perform integrity monitoring and validation without any other intervention/alteration. The runtime measurements collected by IMA, which contain hashes of various runtime events and a hash of the client, will be recorded into the Intel TDX RTMR [2] as part of the Intel TDX attestation quote. The system event logs collected by trustauthority-client will be sent together with a quote to Intel Trust Authority for verification.
Customizable verification policies and reference values can be set in Intel Trust Authority to perform the validation against runtime measurements. In this way, the integrity of trustauthority-client and the integrity of the runtime system can be verified and checked during attestation to confirm that the trustauthority-client and the runtime system are in a trust state.
Figure 5. Sample custom verification policy for Intel Trust Authority
Making trustauthority-client part of the trusted supply chain resolves customer concerns regarding malicious evidence fetching or forging, and ensures genuine evidence is provided to the attestation service. By incorporating runtime measurements into the attestation process, the security and integrity of the platform, programs, and applications are further extended in the confidential computing world.
Summary
The existing TEE (e.g., Intel TDX) implementation provides VM confidentiality at runtime and boot time integrity with boot time measurement. Runtime integrity is also important. IMA is a Linux kernel implementation that provides runtime measurement for integrity verification. However, the current implementation of IMA depends on TPM. In this paper, we have introduced an innovative solution to leverage IMA and the runtime registers (RTMR) provided by Intel TDX to collect runtime measurements in an Intel TDX environment. An abstract layer has been introduced to enable the IMA subsystem to work with the Intel TDX RTMR instead of TPM. With this solution, a confidential VM can provide confidentiality and both boot time and runtime integrity during its life cycle.
Intel Trust Authority provides a SaaS-based attestation service. It can perform attestation verification against a quote (a string containing the representation of the attestation which includes the measurements in Intel TDX RTMR) and check if a VM runs in a trusted TD. With IMA and the abstract layer provided in this paper, Intel Trust Authority can help verify the runtime integrity in a confidential VM supported by Intel TDX during the attestation process, providing enterprises with new options to prove the runtime integrity of the programs and applications of the whole system.
Disclaimer
The approach and implementation of RTMR over IMA can be found in the reference code from Linux Reference Stack, please check Whitepaper: Linux* Stacks for Intel® Trust Domain Extension 1.0 for details. The reference code may not able be upstreamed.
About the Authors
Haidong Xia is a principal engineer at Intel, and lead architect for Intel Trust Authority. He also manages an architecture team. Haidong has been working in security for 20+ years, with experience in cloud and full stack of a platform.
Ken Lu is a Cloud Native Software Architect at Intel, focused on confidential computing technology enabling and adoption on Kubernetes framework for confidential cluster, container etc. He initiated several open source projects, including CCNP(Confidential Cloud Native Primitives) and CNAP (Cloud Native AI Pipeline).
Ruoyu Ying is a Cloud Software Engineer at Intel and an enthusiast for cloud native, security and confidential computing. She has rich experience in developing security features in several cloud native projects to drive the adoption of TEE technologies such as Intel SGX and Intel TDX.
Xiaocheng Dong is a Cloud Software Engineer at Intel and specializes in developing software stacks of confidential computing technologies like Intel SGX and Intel TDX. He is skilled in confidential computing and zero-trust solutions in the cloud-native environment.
Yanhui Zhao is a Cloud Software Architect at Intel, working on enabling differential security solutions based on Intel hardware. He has worked on TEE technologies across different platforms and architectures and driven several related product and service launches over the past few years.
References
[1] Intel, "Intel® TDX White Papers," February 2023. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-trust-domain-extensions.html.
[2] Q. S. Y. X. a. Z. W. P. U. B. C. Wu Luo, "Container-IMA: A privacy-preserving Integrity Measurement Architecture for Containers," in RAID 2019, 2019.
[3] "Google Container-Optimized OS," [Online]. Available: https://cloud.google.com/container-optimized-os/docs/concepts/security.
[4] "OpenEuler Trusted Computing," [Online]. Available: https://docs.openeuler.org/en/docs/21.09/docs/Administration/trusted-computing.html.
[5] "OpenSuse," [Online]. Available: https://en.opensuse.org/SDB:Ima_evm.
[6] Y. D. P. D. Y. G. &. C. W. Liantao Song, "TZ-IMA: Supporting Integrity Measurement for Applications with ARM TrustZone," in ICICS 2022: Information and Communications Security, 2022.
[7] "Kernel parameters," [Online]. Available: https://www.kernel.org/doc/html/v6.2/admin-guide/kernel-parameters.html.
[8] "Kernel IMA policy ABI," [Online]. Available: https://www.kernel.org/doc/Documentation/ABI/testing/ima_policy.
[9] "Integrity Measurement Architecture," [Online]. Available: https://wiki.gentoo.org/wiki/Integrity_Measurement_Architecture#How_do_I_load_a_custom_IMA_policy.3F.
[10] "Keylime Github," [Online]. Available: https://github.com/keylime/keylime.
[11] "Intel® Trust Authority," Intel, [Online]. Available: https://www.intel.com/content/www/us/en/security/trust-authority.html.
[12] H. Sidhpurwala, "How to use the Linux kernel's Integrity Measurement Architecture," 22 October 2020. [Online]. Available: https://www.redhat.com/en/blog/how-use-linux-kernels-integrity-measurement-architecture.