Developer Guide

  • 2021.3
  • 11/18/2021
  • Public
Contents

Real-Time Communication Demo

Demos show technologies and how to write code using those technologies. Demos are not performance benchmarks. The output examples shown here are for illustration only. Your output may vary.
The
tcc_rt_communication_demo
is a set of example programs and scripts that demonstrates the benefit of combining the Cache Allocation Library with Time-Sensitive Networking (TSN) provided by Intel. The
tcc_rt_communication_demo
operates in two modes: non-optimized (
nonopt
) and optimized (
opt
).
The
tcc_rt_communication_demo
in
nonopt
mode;
  • Does not use
    Intel® TCC Mode
    in the BIOS, so the platform is not optimized for real-time tasks.
  • Uses raw sockets (
    AF_PACKET
    ) to send/receive data over the network. Network traffic competes with TCC network packets.
  • Does not use
    Virtual Channels
    , so data sent over PCI (video, audio, USB data, etc.) competes with TCC data.
  • Does not use L2/L3 cache usage optimizations, so data processed on the same core evicts TCC data from the cache and increases the time to access the data.
The
tcc_rt_communication_demo
in
opt
mode:
  • Uses
    Intel® TCC Mode
    set to
    ON
    in the BIOS so the platform is optimized for real-time task execution.
  • Uses the eXpress Data Path (
    AF_XDP
    ) packets optimized for packet processing time. Packets will travel to the memory faster.
  • Uses
    Virtual Channels
    to pass network data to the memory and the processor so that competing PCI traffic does not affect TCC data processing.
  • Uses the
    Software SRAM
    feature to keep data in the cache with a predictable access time.
TSN is a set of standards for real-time Ethernet communication, created by the IEEE 802.1 working group. This demo also uses the OPC Unified Architecture (OPC UA*) for data transfer between systems. The Pub/Sub type of communication implemented by the open62541 library is used by this demo.
In this guide, the IEEE terms “grandmaster” and “slave” are referred to as “primary” and “secondary,” respectively.
The demo requires two target systems, referred to as Board A and Board B to simulate two scenarios.
In the first, or
basic
scenario, Board A runs a data processing workload and sends a network packet to Board B every 500 µsec.
In the second, or
single-input single-output (SISO)
scenario, Board A runs a data processing workload and sends a network packet to Board B. Board B receives the network packet, runs a data processing workload, and returns a network packet to Board A.
To simulate a data processing task, the boards run a cyclic workload. The workload generates a pseudo-random linked list and walks through the list.
The demo supports optimized (
opt
) and non-optimized (
noopt
) modes of operation. In the
opt
mode, the pseudo-random linked list is placed into cache using the Cache Allocation Library to make data access time more predictable. You can run the demo in each mode and compare the results.
The network packet is an L2 Ethernet-based packet. The packet contains sequence number and source timestamp information. To generate network interference, the demo uses the Linux* program, iperf3.
The next sections describe:
  1. Setting up the hardware.
  2. Running the
    basic
    and the
    single input single output
    scenarios.
  3. Plotting and analyzing sample data results.

Step 1: Set Up the Hardware

  1. Get the following recommended hardware configuration:
Units Needed
Equipment Specifications
2
The target systems, see Get Started Guide for UEFI BIOS or Get Started Guide for Slim Bootloader . Each target system contains an integrated TSN-compatible Intel® Ethernet Controller that you will use for this demo.
1
Minimum CAT-5E Ethernet cable
  1. Assign the systems as follows:
Board A
Transmitter. The platform on which the Precision Time Protocol (PTP) clock of the TSN-compatible Intel® Ethernet Controller acts as the secondary clock.
Board B
Receiver. The platform on which the Precision Time Protocol (PTP) clock of the TSN-compatible Intel® Ethernet Controller acts as the primary clock.
  1. Set up the hardware:
    1. Make sure you have completed the steps in the Get Started Guide for UEFI BIOS or Get Started Guide for Slim Bootloader with each board.
    2. Connect one end of a CAT-5e or better (for example, ‘CAT-6’) Ethernet cable to the RJ45 connector of
      Board A
      .
    3. Connect the other end of the cable to the RJ45 connector of
      Board B
      .
    4. Power on the two boards.
    5. To run the sample in the optimized mode, the boards must be configured to have at least 128 KB of L2 cache for allocation. Update the L2 buffer size setting in the configuration file
      ${HOME}/.tcc.config
      to 131072 on both boards. Example of the updated configuration file:
    { "L2": { "buffer_size_bytes": 131072 }, "L3": { "buffer_size_bytes": 262144 } }
    For details about the system configuration for using the cache allocation library, see Before Using the Cache Allocation Library.

Step 2: Run the Demo

In this step, you will run the demo in two scenarios: Basic and Single-Input Single-Output (SISO). You will collect and analyze the workload execution latencies and inter-packet latencies of both scenarios.
The following examples use a specific network interface name. This interface name can vary from board to board so confirm that you are using the right one. Interface names are connected with the TSN-compatible Intel® Ethernet Controller for internal communication between Board A and Board B. Do not pass interface names used for external communication with these boards. Doing so leads to loss of connectivity with the boards when the sample sets up a predefined network address on the interface. If the interface name used for external communication was provided and communication with the board was lost, refer to Troubleshooting.
Run the Basic Example
In this example, a single talker (Board A) sends messages to a single listener (Board B), using the OPC UA* APIs.
To run the basic example:
  1. [Board B]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the listener.
      tcc_rt_communication_demo --profile basic-b-opt --interface <interface name>
      Where
      Option
      Description
      --profile basic-b-opt
      Specify that the Basic profile is running on Board B, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board B.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l, a Linux* user-space daemon that supports time synchronization based on IEEE 802.1AS Generalized Precision Time Protocol (gPTP).
    • Starts phc2sys, a Linux* user-space program that is used to synchronize the system clock of the processor to the PTP clock of the Ethernet controller.
    • Runs iperf3 to generate network interference (optionally, see Command-Line Options for details).
  2. [Board A]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the talker.
      tcc_rt_communication_demo --profile basic-a-opt --interface <interface name>
      Where
      Option
      Description
      --profile basic-a-opt
      Specify that the Basic profile is running on Board A, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board A.
    Board A starts sending packets.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l.
    • Starts phc2sys.
    • Sets Linux* qdisc to enable Time-Aware Traffic Scheduling and LaunchTime.
    • Runs the iperf3 tool to generate network interference (optionally, see Command-Line Options for details).
    1. Wait for the transmitter application to stop. The application prints workload latency statistics and exits.
  3. [Board B]
    Switch to Board B.
    1. Wait until the demo’s script finishes.
    2. Go to the following directory and check the output data file (
      log_e2e.log
      ):
      cd ./tcc_rt_communication_demo_output/ ls -la
Run the SISO Example
In this example, Board A (the monitor) transmits a packet to Board B (the compute) that returns a response to Board A using the OPC UA* APIs.
For accurate results, you must be prepared to launch the sample on each board no more than 2 or 3 seconds apart.
Before starting communication, the compute and the monitor select the base time at which they will start sending messages. The base time is logged in the console and must be the same on both boards for SISO communication. To have the same base time, the compute and the monitor must be launched no more than 2 or 3 seconds apart.
[INFO] Selected base time: 1622106338 (12:51:38), time now: 1622106334 (12:51:34)
The same base time must be selected for the compute and the monitor.
To run the SISO example:
  1. [Board B]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the compute.
      tcc_rt_communication_demo --profile siso-single-b-opt --interface <interface name>
      Where
      Option
      Description
      --profile siso-single-b-opt
      Specify that the SISO profile is running on Board B, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board B.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l.
    • Starts phc2sys.
    • Sets the Linux* qdisc daemon to enable Time-Aware Traffic Scheduling and LaunchTime.
    • Runs iperf3 to generate network interference (optionally, see Command-Line Options for details).
  2. [Board A]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the monitor and actuator.
      tcc_rt_communication_demo --profile siso-single-a-opt --interface <interface name>
      Where
      Option
      Description
      --profile siso-single-a-opt
      Specify that the SISO profile is running on Board A, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board A.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l.
    • Starts phc2sys.
    • Sets the Linux* qdisc daemon to enable Time-Aware Traffic Scheduling and LaunchTime.
    • Runs iperf3 to generate network interference (optionally, see Command-Line Options for details).
  3. [Board A]
    1. Wait until the demo’s script finishes.
    2. Go to the following directory and check the output data file (
      log-roundtrip.log
      ):
      cd ./tcc_rt_communication_demo_output/ ls -la
Run the SISO Example From One Side (Board A)
The SISO example requires that the sample is launched on Board A and Board B at about the sample time. To faciliate this, the remote launch mode is introduced. In this mode, you can launch the SISO example on one board (Board A). The SISO example on Board B is launched automatically, selecting the same start time (base time to start the network communication).
To switch to the remote launch mode, use the optional argument
--remote-address <remote-target-IP>
.
If the sample is switched to remote launch mode, the following arguments can be used:
  • --remote-interface <remote-interface>
    specifies the remote network interface name.
  • --remote-config-path <remote-config-path>
    specifies the path to the config files on the remote side.
  • --remote-exec-path <remote-exec-path>
    definines the path to the sample executables on the remote board.
The sample on the remote side will be launched in a pre-defined working directory as
/tmp
, and the the output directory is pre-defined as
/tmp/tcc_rt_communication_demo_output
. Intel recommendeds to specify the remote paths with full paths. If using relative paths, specify them with respect to the working directory,
/tmp
.
Basic steps required for launching SISO example in remote launch mode:
  1. Set up the passwordless SSH connection from Board A to the Board B on an external (non-TSN) network interface:
    ssh-copy-id 192.168.1.100
  2. Configure the boards to communicate in the required (optimized or non-optimized) SISO mode (Board A). Command line example:
    tcc_rt_communication_demo --profile siso-single-a-noopt --interface enp1s0 --remote-address 192.168.1.100 --mode setup --remote-interface enp1s0 --remote-config-path /home/user/sample/cfg --remote-exec-path /home/user/sample/install
  3. Launched the sample from one side (Board A). Command line example:
    tcc_rt_communication_demo --profile siso-single-a-noopt --interface enp1s0 --remote-address 192.168.1.100 --mode run --remote-interface enp1s0 --remote-config-path /home/user/sample/cfg --remote-exec-path /home/user/sample/install

Check the Quality of Time Synchronization Between Boards

This demo uses two additional programs
ptp4l
and
phc2sys
, which help with time synchronization between boards over the network and on boards between the PTP hardware clock and the system clock. Time synchronization starts automatically during demo execution. It is possible to restart time synchronization without other demo execution steps. For details, see the execution modes (clock) in Command-Line Options.
Each time-synchronization application (ptp4l and phc2sys) has its own log file: `
  • /var/log/ptp4l.log
    for
    ptp4l
  • /var/log/phc2sys.log
    for
    phc2sys
These files can be opened by file-reading tools such as
cat
,
tail
, or
less
. For example, to view the
ptp4l
log file, you can use the following command:
cat /var/log/ptp4l.log
.
You can also list log file data directly during runtime to check the quality of time synchronization. Use the
tail
tool with
--follow
to view new log entries during runtime. For example, use the following command:
tail --follow /var/log/ptp4l.log
.
To check the time offset between primary and secondary clocks, you can use the
ptp4l
tool.
To check the offset between the PTP hardware clock and the system clock, you can use the
phc2sys
tool.
To check synchronization quality, open the corresponding log file using one of methods described above to see the following data:
  • Offset (in nanoseconds)
  • Frequency offset (in parts per billion, ppb)
  • Path delay (in nanoseconds)
The most important value is the offset in nanoseconds. For
ptp4l
, the tool shows the time offset between the primary and secondary clocks in nanoseconds. For
phc2sys
, the tool shows the time offset between the PTP hardware clock and the system clock. Generally, values less than 100 nanoseconds are an indicator of good time synchronization, but this value may vary and depends on hardware components. In this demo, a value less than 1 microsecond is acceptable.

Plot the Data From Log Files

On your host system, use
tcc_rt_communication_demo_plotter.py
to plot the data from
log_e2e.log
and
log-roundtrip.log
.
Prerequisite: Python* release 3.7 or newer.
To plot the data:
  1. On the host system, source the environment file to set up environment variables:
    source ~/intel/tcc_tools/latest/env/vars.sh
  2. Go to the script directory:
    cd ${TCC_ROOT}/samples/plot_scripts
  3. Install required python modules:
    sudo pip3 install -r prerequisites.txt
    or
    sudo python3 -m pip install -r prerequisites.txt
  4. Copy
    tcc_rt_communication_demo_output/log_e2e.log
    and
    tcc_rt_communication_demo_output/log-roundtrip.log
    from the target system to the script directory in the host system.
  5. Run the script:
    python3 tcc_rt_communication_demo_plotter.py --input log_e2e.log --output e2e.png python3 tcc_rt_communication_demo_plotter.py --input log-roundtrip.log --output roundtrip.png
  6. You can specify multiple log files (for example, for optimized and non-optimized profiles) to show all statistics on one diagram:
    python3 tcc_rt_communication_demo_plotter.py --input log_e2e-opt.log log_e2e-noopt.log --output e2e.png
The script prints latency statistics to the console and generates images depending on arguments.

Analyze the Demo Results

The results shown here may not be identically reproduced because inter-packet latency is very sensitive and may vary based on the duration of the test and the health and state of the platform. The format of the graph may differ based on your version of
matplotlib
and system specifications.
The transmitter measures workload execution time and dumps the data in the log file
tcc.stat
. To measure workload execution time, the transmitter uses the Measurement Library.
Demo tools measure transmission latency and dump the data in the log files,
log_e2e.log
and
log-roundtrip.log
. The first log is collected by the receiver for the
basic
mode, and the second log is collected for the
single-input single-output (SISO)
mode.
The graph below for
log_e2e-noopt.log
and
log_e2e-opt.log
shows an example of the transmission latency for the
basic
profile in the optimized and non-optimized modes. To measure transmission latency, the receiver (on Board B) tracks the time when a packet is received and measures the difference between the timestamp in the packet set by the transmitter (on Board A) and the current timestamp from the system. The left panel of the next image shows a histogram of transmission latency, where the x-axis shows transmission count and the y-axis shows time in microseconds. The right side shows the transmission latency of each packet.
The graph below for
log-roundtrip-noopt.log
and
log-roundtrip-noopt.log
shows an example of the roundtrip transmission latency for the
SISO
profile in the optimized and non-optimized modes. In this case, the transmitter sends a packet to the receiver and receives the corresponding packet from the receiver. The transmitter calculates the full roundtrip time based on the original timestamp set on Board A before sending the packet, and the timestamp measured on receiving the return packet.

Command-Line Options

Usage: tcc_rt_communication_demo -p <profile name> -i <interface name> [-m <mode name>] [-n] [-t <target>] [-c <directory with profiles configuration>] [-o <output file>] [-e <path to binary executables>] [-v]
Option
Description
-p, --profile <profile name>
Required. Define the profile name to be executed. The following profiles are supported by the demo application:
  1. Basic, with and without optimizations:
    • basic-a-opt
    • basic-b-opt
    • basic-a-noopt
    • basic-b-noopt
  2. SISO, with and without optimizations:
    • siso-single-a-opt
    • siso-single-b-opt
    • siso-single-a-noopt
    • siso-single-b-noopt
-i, --interface <interface name>
Required. Define the network interface name to be used for TSN-based data transmission. For example: enp1s0.
-m, --mode <mode name>
Optional. Define the operation mode to customize demo behavior. The following modes are supported by the demo application:
  • all (default): run all modes/stages: TSN configuration, clock synchronization, and the demo application
  • setup: Similar to the
    all
    mode, except that it does not start the demo application: (re-)configure the network interfaces, traffic control, and (re-)start ptp4l and phc2sys clock syncronization
  • clock: (Re-)start ptp4l and phc2sys clock synchronization
  • tsn: Configure the network interface and the traffic control for TSN
  • run: Only run the demo application; TSN and clock should already be configured
-n, --no-best-effort
Optional. Do not use best-effort traffic (iperf3) together with other components of the demo application.
-t, --target <target>
Optional. Define the type of target board to select configuration files for execution. Available types are:
  • AUTO (default): autodetect the target
  • EHL: use configuration for Intel Atom® x6000E Series processors
  • TGL-U: use configuration for 11th Gen Intel® Core™ processors
  • TGL-H: use configuration for Intel® Xeon® W-11000E Series processors
-c, --config-path <directory with profiles configuration>
Optional. Define the directory with the profiles’ configuration files. ‘./cfg/’ is used by default. The directory should contain two sub-directories:
  • basic
    for the Basic profiles’ configuration files
  • siso-single
    for the SISO profiles’ configuration files
-o, --output-file <output file>
Optional. Define the filename for storing the output statistics.
  • ./tcc_rt_communication_demo_output/log_e2e.log
    is used by default for the Basic profiles
  • ./tcc_rt_communication_demo_output/log-roundtrip.log
    is used by default for the SISO profiles
-e, --exec-path <directory with binary executables>
Optional. Define the directory with binary executables and shared libraries such as opcua_server and libtcc_rtcdatamgr.so. The demo uses binary executables from the default general system directory and thus the binary executables are available by searching through the PATH environment. This directory is used before the directories from the PATH environment variable.
-v, --verbose
Optional. Enable the verbose mode.
-h, --help
Optional. Print command-line help.
-a, --remote-address <remote target IP>
Required for the remote launch mode. If provided, switches the sample to the remote launch mode, making it possible to launch the sample from board A. Defines the IP address for the 2nd remote target. This parameter can only be combined with
basic-a-opt/noopt
or
siso-single-a-opt/noopt
profiles.
-g, --remote-config-path <remote config path>
Required for the remote launch mode. Define the directory on the remote side that contains the profiles configuration files. The directory should contain 2 subdirectories: ‘basic/’ for basic-<…> profiles configuration files, and ‘siso-single/’ for siso-single-<…> profiles configuration files.
-u, --remote-exec-path <remote directory with binary executables>
Optional. Define the directory on the remote side with binary executables such as opcua_server and libtcc_rtcdatamgr.so. Use this option if the sample executables are located in directories that do not appear in the PATH environment variable. This option prepends the provided directory to the PATH variable on the remote target.
-r, --remote-interface <remote interface name>
Optional. Define the name of the network interface on the remote side to be used for TSN-based data transmission in the remote launch mode. If not specified, the local interface name will be used in the remote launch mode for the remote interface.
-b, --base-time <the base time in UTC to start network communications>
Optional. Define the base time for TSN communications. Other times defined in the configuration files are calculated with respect to this base time. If the base time is not provided, it is selected automatically. Also, this argument is used internally in the remote launch mode to select the same launch time for both boards. The base time must be selected in such a way that it is between 2 and 20 seconds from the launch time of the opcua_server.
Example
tcc_rt_communication_demo -p basic-a-opt -i enp1s0 -m run -n

Troubleshooting

Option
Description
tcc_rt_communication_ demo
reports:
Unknown interface name was specified.
Specify valid interface name (check with ifconfig tool,
for example).
Make sure you provide the correct interface name for the TSN-compatible Intel® Ethernet Controller:
Run:
lspci -v
Locate the Peripheral Component Interconnect (PCI) bus information (example: 01.00.0) of the TSN-compatible Intel® Ethernet Controller:
Run:
ethtool -i <enp1s0>
and find the bus information (example: 0000:01:00.0) that matches the previously found address.
tcc_rt_communication_ demo
reports:
Directory <path> with
configuration profiles does not exist.
Make sure you provide the correct path to the directory containing configuration files for the demo. If the directory does not exist, create it and add subdirectories containing configuration files for the basic and SISO modes.
tcc_rt_communication_ demo
reports:
Path to output file (<path name>)
does not exist.
Make sure you provide the correct path to the directory for the output file. If the directory does not exist, create it.
tcc_rt_communication_ demo
reports:
Profile <profile name> not supported.
Use [-h] to see help information.
Make sure to run the
tcc_rt_communication_demo
script with a supported profile. Supported profiles are: basic-a-opt, basic-a-noopt, basic-b-opt, basic-b-noopt, siso-single-a-opt, siso-single-a-noopt, siso-single-b-opt, and siso-single-b-noopt.
tcc_rt_communication_ demo
reports:
Mode <mode name> not supported.
Use [-h] to see help information.
Make sure to run the
tcc_rt_communication_demo
script with a supported mode. Supported modes are: clock, tsn, run, setup, and all.
tcc_rt_communication_ demo
reports:
Binary "<binary name>" was not found in
directories provided by PATH environment variable.
Try to launch script with [-e] option and provide path
to binary executables manually.
Make sure the following binary executables are installed in one of the directories indicated by the PATH environment variable:
  • opcua-server
If the binaries are not installed in one of these directories, specify the correct directory using the [-e] CLI option.
tcc_rt_communication_ demo
reports:
Path to directory with binary
executables (<path name>) does not exist.
Make sure you provide the correct path to the directory with the following binary executables:
  • opcua-server
If the directory does not exist, create it and place the binaries into it.
Loss of connectivity with the board after using a network interface name that is not related to the internal communication between Board A and Board B.
Restart the board using one of the following methods:
  1. Through the GUI shell interface if the board is connected to a keyboard, video, and mouse (KVM) device.
  2. By pressing the reset button.
Loss of connectivity between Board A and Board B during demo execution.
Restart the boards using one of the following methods:
  1. Through the GUI shell interface if the board is connected to a keyboard, video, and mouse (KVM) device.
  2. By pressing the reset button.
The transmission latency is too long for the SISO example, more than 100 ms.
Make sure the compute and the monitor select the same base time (can be checked in the console logs); if the base time is different, try launching the compute and the monitor at the same time (with the time difference of less than 2 seconds).
The transmission latency is very large or negative.
The time synchronization is lost. Check the following log files:
/var/log/ptp4l.log
and
/var/log/phc2sys.log
on both boards. Run the
tcc_rt_communication_demo
script with added [-m clock] option. This command will restart the time synchronization.
tcc_rt_communication_demo
in the optimized mode fails to start. The console logs show:
Error: cannot allocate buffer for the pointer chasing workload
.
Possible solutions:
  • Make sure you have completed the steps in the Get Started Guide for UEFI BIOS or Get Started Guide for Slim Bootloader with each board.
  • Make sure the
    Intel® TCC Mode
    and
    Software SRAM
    options are enabled in BIOS.
  • Make sure you have enabled the runtime support for software SRAM on each board by running the
    tcc_setup_ssram.sh enable
    script.
  • Make sure the L2 buffer size setting in the configuration file
    ${HOME}/.tcc.config
    is at least 131072 on both boards or set to zeroes.
  • Export the
    TCC_LOG_LEVEL=DEBUG
    environment variable and run the sample again. This variable enables the console logs for the cache allocation library to show errors about the software SRAM allocation.
  • Try enabling the SW SRAM running
    tcc_setup_ssram.sh enable
    after disabling the SW SRAM by running
    tcc_setup_ssram.sh disable

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.