Intel Acceleration Stack Quick Start Guide for Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA

ID 683633
Date 12/04/2020
Public
Download
Document Table of Contents

B. Handling Graceful Thermal Shutdown

Note:
  • Qualified OEM server systems provide adequate cooling for standard workloads and the use of pacd may be optional.
  • For details about using pacd, including considerations that may lead to an unexpected system reboot, refer to the pacd documentation on the OPAE web page.

The Intel® FPGA PAC Daemon (pacd) is a program that can be used to help protect the server from crashing due to hardware reaching an upper or lower non-recoverable sensor threshold. pacd is capable of monitoring any of the 23 sensors reported by Board Management Controller. pacd can be run standalone, as a daemon, or as a systemd service. When the OPAE tools extra package is installed, pacd gets placed in the OPAE binaries directory (default: /usr/bin ) along with configuration and service files – pacd.conf and pacd.service, respectively.

On startup, pacd sets its thresholds to a constant -5 degrees or 5 percent lower power under the BMC threshold values. This is to pass on the Graceful Thermal Shutdown responsibility to pacd.

pacd periodically reads the sensor values and if the values exceed the threshold, it sends a SIGHUP signal to all running processes and makes the board inaccessible from the host. The daemon waits for a configurable time specified by -c in pacd.conf, as described below, to cool down the board. After this configurable wait time elapses, the pacd service programs the specified AFU. Ensure that the AFU host application monitors for a SIGHUP signal and exits.

Note: Partial reconfiguration cannot be initiated from a Virtual Machine. Hence, pacd cannot run on a Virtual Machine.
pacd can be set up as a systemd service as follows (using a shell with elevated privileges (sudo)):
  1. Edit the pacd.conf file to update the “DefaultGBSOptions” entry with a list of AFUs appropriate for your FIM. Use the full absolute path to each AFU file and precede each file name with ‘-n’.
    sudo vim /usr/bin/pacd.conf
    
    Edit entry:
    DefaultGBSOptions=-n /home/<username>/intelrtestack/\
    a10_gx_pac_ias_1_2_1/hw/samples/nlb_mode_3/bin/nlb_mode_3_unsigned.gbs
    Note:
    Optional settings include:
    • PCIe address (For example: -S, -B, -D, -F), pacd monitors all Intel® FPGA PACs matching the PCIe address components specified. For example, if you specify -B 5 only, all Intel® FPGA PACs on PCIe bus 5 becomes monitored.
    • Sensor Threshold—The thresholds are global, so specifying -T 11:95.0:93.0 monitors sensor 11 on all selected Intel® FPGA PACs. When the value exceeds 95.0, it causes the default bitstream specified with -n in pacd.conf to be programmed (PR). The sensor is considered triggereduntil its value drops below 93.0.
    • You can specify a cool down period by:
      1. Changing the CooldownInterval=<time period>
      2. Setting -c <time period> for ThresholdOptions in pacd.conf or

      If both are set, then the -c <time period> for ThresholdOptions takes precedence.

    • The Sensor Number can be found by running this command:
      sudo fpgainfo bmc

    Examine the remaining option variables and adjust as appropriate for your system.

  2. Copy pacd.conf to the default systemd service configuration directory (typically /etc/sysconfig ).
    RHEL:
    sudo cp /usr/bin/pacd.conf /etc/sysconfig/
    Ubuntu:
    sudo cp /usr/bin/pacd.conf /etc/default
  3. Edit the pacd.service file to update “EnvironmentFile” entry to reflect where the pacd.conf file was copied. Prepend the path name with a single dash ‘-‘, and specify the path as absolute.
    sudo vim /usr/bin/pacd.service

    Edit entry:

    RHEL:
    EnvironmentFile=-/etc/sysconfig/pacd.conf
    Ubuntu:
    EnvironmentFile=-/etc/default/pacd.conf
  4. Copy pacd.service to /etc/systemd/system/pacd.service. This makes pacd visible to systemd.
    RHEL:
    sudo cp /usr/bin/pacd.service /etc/systemd/system/
    Ubuntu:
    sudo cp /usr/bin/pacd.service /lib/systemd/system/
  5. Start pacd as a systemd service.
    Note: Please use sudo if command cannot be run in regular user mode.
    systemctl daemon-reload
    systemctl start pacd.service
  6. Optional: To enable pacd to re-start on boot, execute
    systemctl enable pacd.service

    To check whether pacd has started and to check state or actions, please examine the log file (specified in pacd.conf on the “LogFile” line).

    For a full list of systemctl commands, run the following command:
    systemctl -h
  7. To verify that the service is running, run the following command:
    systemctl status pacd.service
    
  8. To stop the service:
    systemctl stop pacd.service