Tutorial: Configuring EdgeX Hardware Security Hooks to use a TPM

Overview

The presence of physical security threats is one way in which edge-based applications differ from their cloud based counterparts: confidential data is at risk due to device cloning and similar attacks. In this tutorial you will learn how to enable the hardware security hooks in the EdgeX Foundry IoT middleware and use these hooks to supply a TPM-protected encryption key for protecting EdgeX's secret store. Doing so will eliminate the plaintext storage of a critical on-disk encryption key.

Prerequisites:

This tutorial assumes that the user account has been added to the "docker" group so that docker commands can be issued without the use of the "sudo" command.

It is additionally assumed for this tutorial that the TPM2 software stack is not installed. Specifically, no TSS software stack (libtss2-esys.so) and no TPM resource broker (tpm2-abrmd service) should be installed. Having these components installed may lead to the sample linking to an untested version of the TSS libraries or the TPM resource broker holding exclusive control of the TPM character device.

Steps

Step 1: Install EdgeX Foundry middleware

Follow the EdgeX Foundry Getting Started User's Guide for Docker to set up your EdgeX Foundry installation. (Note that this tutorial requires a feature present in the EdgeX Hanoi or later release.)

Navigate to the "releases/nightly-build/compose-files" folder of the developer-scripts repository.

Start the framework:

$ make run

Most EdgeX containers remain running after they are started, but some containers are one-shot—they run and then terminate. Ensure that the framework is running properly by checking for an absence of containers that have failed with error codes:

$ docker ps -a
CONTAINER ID   IMAGE                             STATUS 
3567acfce923   nexus3...proxy-setup-go:master    Exited (0) Less than a second ago 

Also check for liveness of the EdgeX core containers. For example:

$ docker logs edgex-core-data
...
level=INFO ts=2020-08-06T01:05:47.877440433Z app=edgex-core-data source=event.go:287 msg="Putting event on message queue"
level=INFO ts=2020-08-06T01:05:47.877674793Z app=edgex-core-data source=event.go:305 msg="Event Published on message queue. Topic: events, Correlation-id: 8d674d88-68d9-4816-a464-befe4d26ead7 " 

In the event of problems, reach out to the EdgeX community through the EdgeX Slack #help channel.

Step 2: Verify presence of unencrypted Vault master key

With the EdgeX framework running, dump the data file that contains the Vault master key and send it through the Python JSON beautifier for readability. If the Python JSON beautifier is not installed, just look for the "keys" and "keys_base64" objects in the non-beautified output. (Note: the command below is "docker exec" with a space between "docker" and "exec".)

$ docker exec -it edgex-vault-worker cat /vault/config/assets/resp-init.json | python3 -mjson.tool
{
    "keys": [
        "32dd97a45d3be094c12a0574407da5c82cd4a960f05f43110f5b78b4bd6b23bf54",
        "10028d7ac76ede405fec01ccc5e57315c4d2aad170716fa76448322f59098a476e",
        "3dafd69ad18bd1f2f8f4ffb4c44c91e5eb953265fd1b34c0adf3c8b3ab4e336439",
        "b996ea597803b3f782d0276bbf807a98cdd92c9361f3e8333c913c4ba057e6bcf2",
        "8f8496edb6f2d8df04841cbf7e597f3eb357aefb54e9d78c44bda8c021797dd149"
    ],
    "keys_base64": [
        "Mt2XpF074JTBKgV0QH2lyCzUqWDwX0MRD1t4tL1rI79U",
        "EAKNesdu3kBf7AHMxeVzFcTSqtFwcW+nZEgyL1kJikdu",
        "Pa/WmtGL0fL49P+0xEyR5euVMmX9GzTArfPIs6tOM2Q5",
        "uZbqWXgDs/eC0Cdrv4B6mM3ZLJNh8+gzPJE8S6BX5rzy",
        "j4SW7bby2N8EhBy/fll/PrNXrvtU6deMRL2owCF5fdFJ"
    ]
}

Observe that these keys are the raw unencrypted keys needed to unseal the Vault-based secret store, which is a known security weakness (CWE-313) found in many systems. This fact is also noted in the log file generated by EdgeX's edgex-security-secretstore-setup utility:

$ docker logs edgex-vault-worker | grep master
level=INFO ts=2020-08-06T00:46:37.411447344Z app=edgex-security-secretstore-setup source=vmkencryption.go:85 msg="Not encrypting Vault master key"

The objective of this tutorial is to configure edgex-security-secretstore-setup to encrypt the Vault master key to address this security weakness.

Step 3: Compile the TPM secret protection code sample

Locate the Intel Developer Zone "Code Sample: Secret key provisioning and retrieval in C using Intel® Platform Trust Technology".

Download the sample, extract it to a local folder, and follow the instructions in README.md and INSTALL.md to compile and build the sample. Please follow the instructions closely—several non-standard command-line options are required to build the sample. When the sample is built correctly, there will be a "local" folder within the sample that contains local copies of the tpm2-software libraries and utilities, and the "tools" folder will have a "tpmseed" executable.

Walk through the code sample and experiment with it to build an understanding of what the code is doing and how it works. Fully comprehending the sample may take several hours. There will be a point in the sample where the bootstrapping script asks for a PCR list for protection of a generated key. To make it easier to perform negative testing (simulated attack), it may be useful to specify "sha256:16" as the PCR list. PCR[16] is a debug PCR and not suitable for production usage, but it is useful for the tutorial as PCR[16] can be reset without rebooting the system.

This tutorial assumes that the development machine and the target machine are the same. If secret provisioning and retrieval works in the sample, then it should also work when integrated into the EdgeX Framework as part of this tutorial.

Running the code sample will result in the generation of several configuration files that are needed for runtime secret protection. These files, located in the "db" folder of the "scripts" directory, are:

• authorized.policy: This file specifies the authorization policy on the TPM-protected seed value. It is required when running EdgeX for the first time, when the EdgeX secret store is initialized.
• pcr.list: This file is needed by the utility to tell the TPM which PCRs authorize the release of the TPM-protected seed value.
• (hex-digits).signature: This file is needed by the TPM to authorize the release of the TPM-protected seed value.
• signing_key_public.pem: This file is needed by the TPM to authorize the release of the TPM-protected seed value.

Neither the signing key (signing_key_private.pem) nor the policy database (pcr.policy, pcr.values, pcr.values.bin) will be deployed into the EdgeX environment. The security of the entire solution depends on the contents of signing_key_private.pem remaining confidential: it should never be deployed to an edge device.

If at any point during execution of the code sample the error "ERROR: Esys_Load(0x902) - tpm:warn(2.0): out of memory for object contexts" is returned, this means that the TPM's limited memory has been exhausted. A reboot is necessary to clear this condition. If this condition persists after a reboot, consider changing the code and examples to use the built-in kernel resource manager (/dev/tpmrm0) instead of the raw TPM device (/dev/tpm0)—the resource manager will clean up TPM resources at process termination in case the programs neglect to do it themselves. Unlike the TPM resource broker daemon (tpm2-abrmd), the kernel resource manage does not hold an exclusive lock on the TPM device.

Step 4: Create a Docker container to inject TPM utility

Copy and paste from this tutorial to create the following files in the root directory of the TPM sample code:

docker-entrypoint.sh

#!/bin/sh -e

# Install files from the container into a Docker volume mounted at $PREFIX

cp -rpd "${SRC}/bin" "${PREFIX}/bin"
cp -rpd "${SRC}/etc" "${PREFIX}/etc"

exit 0

ikm_hook

#!/bin/sh

tpmseed_dir="/opt/tpmseed"
datadir="${tpmseed_dir}/data"
datafile="tpmseed"
device="/dev/tpm0"
etc="${tpmseed_dir}/etc"
len="32"

mkdir -p "${datadir}"
exec "${tpmseed_dir}/bin/tpmseed" "--sealing-policy=${etc}/authorized.policy" "--private=${datadir}/${datafile}.priv" "--public=${datadir}/${datafile}.pub" "--signature-key=${etc}/signing_key_public.pem" "--signature=${etc}/{hash}.signature" "--pcr-list=${etc}/pcr.list" -n "${len}" -T "${device}"

Dockerfile

FROM busybox:latest

ENV SRC=/src
ENV PREFIX=/opt/tpmseed

VOLUME [ "/opt/tpmseed" ]

RUN mkdir /src /src/bin /src/etc

COPY ikm_hook /src/bin/
COPY tools/tpmseed /src/bin/
RUN chmod 555 /src/bin/ikm_hook /src/bin/tpmseed

COPY scripts/db/authorized.policy /src/etc
COPY scripts/db/pcr.list /src/etc
COPY scripts/db/signing_key_public.pem /src/etc
COPY scripts/db/*.signature /src/etc

COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod 555 /usr/local/bin/docker-entrypoint.sh

ENTRYPOINT [ "docker-entrypoint.sh" ]

Next, build a container (tpm-hooks-installer:latest) containing the TPM utility and the necessary configuration files.  (Note that there is a space before the period (".") that may not show.)

$ docker build -t tpm-hooks-installer:latest .

Step 5: Modify docker-compose file to inject TPM utility

Navigate to the "releases/nightly-build/compose-files" folder of the developer-scripts repository.

It is necessary to selectively merge some snippets of YML to docker-compose-nexus-add-security.yml.

To the existing "volumes" section of the YML, add a "tpmseed" volume:

volumes:
  tpmseed:

Add a new "tpmseed" service to the "services" section of the YML:

services:
  tpmseed:
    image: tpm-hooks-installer:latest
    container_name: tpmseed
    read_only: true
    volumes:
      - tpmseed:/opt/tpmseed:z

Merge the following keys into the existing "vault-worker" service. (Suggestion: search for "-worker:")

  vault-worker:
    devices:
      - "/dev/tpm0:/dev/tpm0"
    environment:
      IKM_HOOK: /opt/tpmseed/bin/ikm_hook
    volumes:
      - tpmseed:/opt/tpmseed:z
    depends_on:
      - tpmseed

(Adding the device should ensure that the TPM device is added to the device cgroup for the container automatically.)

Next change to the parent directory and run:

$ make build

to regenerate the docker-compose files. (Re-run "make build" every time a file in the "source" directory is changed!")

Step 6: Reset EdgeX installation

Navigate to the Navigate to the "releases/nightly-build/compose-files" folder of the developer-scripts (https://github.com/edgexfoundry/developer-scripts) repository and run the command:

$ make clean

The clean command will tear down the running EdgeX framework and remove the persistent docker volumes holding the EdgeX persistent state. This is necessary in order to force re-initialization of the EdgeX secret store.
Confirm they are gone with the following command:

$ docker volume list
DRIVER              VOLUME NAME

If there are still EdgeX volumes listed here, kill any remaining docker processes and remove the docker volumes with "docker volume prune -f".

Step 7: Restart EdgeX installation with hardware security hooks enabled

Navigate to the Navigate to the "releases/nightly-build/compose-files" folder of the developer-scripts (https://github.com/edgexfoundry/developer-scripts) repository and run:

$ make run

Inspect to the logs to confirm that hardware encryption has been enabled:

$ docker logs edgex-vault-worker | grep master
level=INFO ts=2020-08-06T04:04:58.327884568Z app=edgex-security-secretstore-setup source=vmkencryption.go:96 msg="Enabled encryption of Vault master key"

Inspect the data file previously inspected in step 2 and confirm that all the keys are now encrypted. Note that "keys" and "keys_base64" arrays are missing and replaced with "encrypted_keys" and "nonces":  (Note: the command below is "docker exec" with a space between "docker" and "exec".)

$ docker exec -ti edgex-vault-worker cat /vault/config/assets/resp-init.json | python3 -mjson.tool
{
    "encrypted_keys": [
        "3137e26dec05af30e42a2d4b61349dca7f2e4d441aaf10ebd59a6a1a38f36f01babe3055f2d2775e1402cc970bfb34fb72",
        "3a886b5028e3256f46e4a95898d6af9cad923e0c3d4f24db47858e6df7a49d1e2fb13cd3e3713cc66f0ea2963e29e3efd6",
        "e880b1f14f7692ed771d3d820f227386efcf39b3431feb40464d67b9acf1e2eddb68725df31e343cd7db299465984d3b10",
        "e94f76857ae688a40313a82756342b5354f56a6ed9e10ba5cdcf12cf31b290e9dcb1c0cb08c8977c944081ee75ebdcb686",
        "bb8d3222dd1b81dca5c8dba686e9115dfde561409f118ae5d2a0bc904444a30635af8abdb8fdfaa580ccae110b3091c472"
    ],
    "nonces": [
        "e4f8579c214f7190b120046d",
        "ed9bb86dfd62c75d5ebf396b",
        "f51542e01da8ad10a3190b20",
        "3adff146c3221b96250e6007",
        "dca51bef25cf77fa97282fc4"
    ]
}

Also note the presence of an additional file: (Note: the command below is "docker exec" with a space between "docker" and "exec".)

$ docker exec -ti edgex-vault-worker xxd /vault/config/assets/kdf-salt.dat
00000000: ab08 86da c25b 5070 72ef bead 20cf de40  .....[Ppr... ..@
00000010: 5f49 2662 ec89 bf9b 0db9 b774 69e7 438a  _I&b.......ti.C.

This is a random salt that is mixed in with the input key material. It can compensate for a weak input key. It also provides a mechanism to reset the encryption keys without having to re-provision a TPM-based seed. Every re-installation of EdgeX, even on the same hardware, will generate unique encryption keys to protect the Vault master key.

Step 8: Negative testing

Suppose for purposes of illustration that the PCR[16], the debug PCR, was used for sealing. It is possible to simulate a hypothetical attacker attempting to circumvent the boot process, which will change PCR measurements, and render the seed irretrievable.

First, shut down the framework. Navigate to the "releases/nightly-build/compose-files" folder of the developer-scripts (https://github.com/edgexfoundry/developer-scripts) repository and run

$ make down

Second, navigate to the root shell created during the code sample walkthrough that has the TPM2-tools on the PATH. (Recall that these utilities were installed into the "local/bin" folder of the code sample.)

Change PCR[16] by extending a new value into it. Any 32-byte long hex value will do. (Do not be alarmed that the new PCR value does not resemble either the old value or the extend value. This is intentional.)

# tpm2_pcrread sha256:16
sha256:
  16: 0x0000000000000000000000000000000000000000000000000000000000000000

# tpm2_pcrextend 16:sha256=4837043708f92e928beded1cc9f40bfca7a24cd93aab67b1947236fbbaabed32

# tpm2_pcrread sha256:16
sha256:
  16: 0x7AEAD8DEB4C18BB0C5F8B0EC8D7EC179E046BA9A0B86DB7B8145474F7CAB1ADC

Third, switch back to the EdgeX "compose-files" folder, re-launch the framework, and observe the seed is irretrievable:

$ make run

$ docker logs -f edgex-vault-worker 
level=ERROR ts=2020-08-06T05:01:43.319450214Z app=edgex-security-secretstore-setup source=vmkencryption.go:90 msg="Error reading input key material from IKM_HOOK - encryption not enabled: exit status 1"
level=ERROR ts=2020-08-06T05:01:43.319524885Z app=edgex-security-secretstore-setup source=init.go:93 msg="failed to enable vault master key encryption: exit status 1"

The framework will eventually fail because Vault could not be unsealed.

Lastly, reset the PCR to its initial state and prove all returns to normal.

In the EdgeX "compose-files" folder:

$ make down

In the code sample:

# tpm2_pcrreset 16

# tpm2_pcrread sha256:16
sha256:
  16: 0x0000000000000000000000000000000000000000000000000000000000000000

"tpm2_pcrreset" will only work for PCR[16]. If any other PCR was used, a reboot would have been necessary to reset the PCR back to its initial state.

In the EdgeX "compose-files" folder:

$ make run

$ docker logs edgex-vault-worker | grep master
level=INFO ts=2020-08-06T04:04:58.327884568Z app=edgex-security-secretstore-setup source=vmkencryption.go:96 msg="Enabled encryption of Vault master key"

The EdgeX framework will now start normally.

Discussion

EdgeX Foundry uses a software-based secret store to manage system and application secrets. These secrets include database passwords, private keys for TLS, and general-purpose secret storage for EdgeX applications. EdgeX Foundry leverages an open source product called Vault that is commonly used in cloud-based applications as its secret storage backend. The recommended production hardening guidelines for Vault require Vault to run a physically separate dedicated machine where the database encryption key is split and delivered to N human operators who must then collaborate to "unseal" the database at runtime. This is clearly a problem for a headless edge device deployed into the field with limited Internet connectivity. Thus, EdgeX Foundry provides a built-in utility to store the database encryption key to local disk (a Docker volume) and provide it to Vault automatically upon cold start of the framework. Consequently, regardless of the steps taken to protect these secrets at the application level, they ultimately depend on a secret that is stored on media in plaintext.

The hardware security hooks in the EdgeX Framework allow the database encryption key(s) to themselves be encrypted instead of being stored in plaintext. When the hardware security hooks are enabled, the framework invokes an external utility to retrieve a cryptographic "seed." This seed is used as input key material to a key derivation function, where a unique cryptographic key is generated for each of the N shares of the database encryption key. These encryption keys, while generated deterministically, appear random and unrelated to the original input seed. The input seed is generated by a TPM-based random number generator. This random number is then either wrapped by a TPM-based key and stored on disk or stored directly in the TPM's NVRAM. In either case, a policy is associated with this seed value that requires an attestation of the system state, as represented by the TPM platform configuration registers (PCR). When the device is in an active state, judicious PCR selection can protect the seed from many forms of tampering with the system boot sequence. When the device is powered-off, the data needed to recover the encryption key is safely locked away inside of the TPM, reducing exposure of sensitive data at rest.

Conclusion

Intel Platform Trust Technology is a capability on select Intel platforms that provides a firmware integrated TPM used to protect secret data and keys. This tutorial provided a concrete example of replacing a plaintext secret key stored on disk with a key wrapped by a TPM-protected secret and where access to the key was authorized by local attestation of the system state.

Please check Intel Developer Zone for additional examples of how Intel hardware security features can be used to enhance the security of your applications.

Feedback

Please contact iot-security-samples@intel.com if you have feedback, questions, or wish to report issues with this tutorial.

Notices

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

Author Biography

Bryon is a security architect in Intel's Internet of Things Group and a contributor to the EdgeX Foundry since 2018 and member of EdgeX Foundry's security working group. Bryon is working to raise the IoT security bar by connecting higher level software stacks such as EdgeX to hardware-based capabilities present in Intel platforms.