Developer Guide

  • 2021.2
  • 06/11/2021
  • Public
Contents

Real-Time Communication Demo

Demos show technologies and how to write code using those technologies. They are not performance benchmarks. The output examples shown here are for illustration only. Your output may vary.
The
tcc_rt_communication_demo
is a set of example programs and scripts that demonstrates the benefit of combining the Cache Allocation Library with Time-Sensitive Networking (TSN) provided by Intel.
TSN is a set of standards for real-time Ethernet communication, created by the IEEE 802.1 working group. This demo also utilizes OPC Unified Architecture (OPC UA*) for data transfer between systems. The Pub/Sub type of communication implemented by the open62541 library is used by this demo.
In this guide, the IEEE terms “grandmaster” and “slave” are referred to as “primary” and “secondary,” respectively.
The demo requires two target systems, referred to as Board A and Board B.
The demo simulates two scenarios. In the
basic
scenario, Board A runs a data processing workload and then sends a network packet to Board B, once every 500 µsec.
In the second scenario,
single-input single-output (SISO)
, Board A runs a data processing workload and then sends a network packet to Board B. Board B receives the network packet, runs a data processing workload, and sends a network packet back to Board A.
To simulate a data processing task, the boards run a cyclic workload. The workload generates a pseudo-random linked list and walks through the list.
The demo supports optimized (
opt
) and non-optimized (
noopt
) modes of operation. In the optimized mode, the pseudo-random linked list is placed into cache using the Cache Allocation Library to make data access time more predictable. You can run the demo in each mode and compare the results.
The network packet is an L2 Ethernet-based packet. The packet contains sequence number and source timestamp information.
To generate network interference, the demo uses the Linux* program, iperf3.
The next sections describe:
  1. Setting up the hardware
  2. Running the
    basic
    and the
    single input single output
    scenarios
  3. Plotting and analyzing sample data results

Step 1: Set Up the Hardware

Acquire the following recommended hardware configuration:
Units Needed
Equipment Specifications
2
The target systems, see Get Started Guide. Each target system contains an integrated TSN compatible Intel® Ethernet Controller, which you will use for this demo.
1
Minimum CAT-5E Ethernet cable
Assign the systems as follows:
Board A
Transmitter. The platform on which the Precision Time Protocol (PTP) clock of the TSN-compatible Intel® Ethernet Controller acts as the primary clock.
Board B
Receiver. The platform on which the Precision Time Protocol (PTP) clock of the TSN-compatible Intel® Ethernet Controller acts as the secondary clock.
To set up the hardware:
  1. Make sure you have completed the steps in the Get Started Guide with each board.
  2. Connect one end of a minimum CAT-5E Ethernet cable to the RJ45 connector of
    Board A
    .
  3. Connect the other end of the cable to the RJ45 connector of
    Board B
    .
  4. Power on the two boards.
  5. To run the sample in the optimized mode, the boards must be configured to have at least 128 KB of L2 cache for allocation. Update the L2 buffer size setting in the configuration file
    ${HOME}/.tcc.config
    to 131072 on both boards. Here is an example of the updated configuration file:
    { "L2": { "buffer_size_bytes": 131072 }, "L3": { "buffer_size_bytes": 262144 } }
    For details about the system configuration for using the cache allocation library, see Before Using the Cache Allocation Library.

Step 2: Run the Demo

In this step, you will run the demo in two scenarios: Basic and Single-Input Single-Output (SISO). You will collect and analyze the workload execution latencies and inter-packet latencies of both scenarios.
The following examples use a specific network interface name. This interface name can vary from board to board so confirm that you are using the right one. Interface names are connected with the TSN-compatible Intel® Ethernet Controller for internal communication between Board A and Board B. Do not pass interface names used for external communication with these boards. Doing so leads to loss of connectivity with the boards when the sample sets up a predefined network address on the interface. If the interface name used for external communication was provided and communication with the board was lost, refer to Troubleshooting.
For systems running the PR1 release of the Yocto Project*-based BSP for Intel Atom® x6000E Series processors, you must rebuild tcc_rt_communication_demo:
  1. Go to the sample directory:
    cd /usr/share/tcc_tools/samples/tcc_rt_communication_demo
  2. Change the library name in the Makefile to libittnotify64.a:
    sed 's/ittnotify/ittnotify64/g' -i Makefile
  3. Build the sample:
    make install
Run the basic example:
In this example, a single talker (Board A) sends messages to a single listener (Board B), using the open62541 APIs.
For systems running the PR1 release of the Yocto Project*-based BSP for Intel Atom® x6000E Series processors, you must rebuild tcc_rt_communication_demo and execute it from the built binaries:
  1. Build the sample using the instructions above.
  2. When you are ready to run the sample, go to the build directory:
    cd /usr/share/tcc_tools/samples/tcc_rt_communication_demo/install
  3. Run the sample from the built binaries by using the
    -e ./
    argument. For example:
    ./tcc_rt_communication_demo.py --profile basic-a-opt --interface enp0s30f4 -v -m run -e ./
When you run the example below, make sure to go to the build directory and run the
./tcc_rt_communication_demo.py
(instead of
tcc_rt_communication_demo
) command with the
-e ./
argument.
To run the basic example:
  1. [Board B]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the listener.
      tcc_rt_communication_demo --profile basic-b-opt --interface <interface name>
      Where
      Option
      Description
      --profile basic-b-opt
      Specify that the Basic profile is running on Board B, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board B.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l, a Linux* user-space daemon that supports time synchronization based on IEEE 802.1AS Generalized Precision Time Protocol (gPTP).
    • Starts phc2sys, a Linux* user-space program that is used to synchronize the system clock of the processor to the PTP clock of the Ethernet controller.
    • Runs iperf3 to generate network interference (optionally, see Command-Line Options for details).
  2. [Board A]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the talker.
      tcc_rt_communication_demo --profile basic-a-opt --interface <interface name>
      Where
      Option
      Description
      --profile basic-a-opt
      Specify that the Basic profile is running on Board A, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board A.
    Board A starts sending packets.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l.
    • Starts phc2sys.
    • Sets Linux* qdisc to enable Time-Aware Traffic Scheduling and LaunchTime.
    • Runs the iperf3 tool to generate network interference (optionally, see Command-Line Options for details).
    1. Wait for the transmitter application to stop. The application prints workload latency statistics and exits.
  3. [Board B]
    Switch to Board B.
    1. Wait until the demo’s script finishes.
    2. Go to the following directory and check the output data file (
      log_e2e.log
      ):
      cd ./tcc_rt_communication_demo_output/ ls -la
Run the SISO example:
In this example, Board A (the monitor) transmits a packet to Board B (the compute) that returns a response to Board A using the open62541 APIs.
For accurate results, you must be prepared to launch the sample on each board at about the same time, no more than 2 or 3 seconds apart.
Before starting communication, the compute and the monitor select the base time at which they will start sending messages. The base time is logged in the console and must be the same on both boards for SISO communication. To make the base time the same, the compute and the monitor must be launched with only a small time difference, no more than 2 or 3 seconds.
[INFO] Selected base time: 1622106338 (12:51:38), time now: 1622106334 (12:51:34)
The same base time must be selected for the compute and the monitor.
For systems running the PR1 release of the Yocto Project*-based BSP for Intel Atom® x6000E Series processors, you must rebuild tcc_rt_communication_demo and execute it from the built binaries:
  1. Build the sample using the instructions above.
  2. When you are ready to run the sample, go to the build directory:
    cd /usr/share/tcc_tools/samples/tcc_rt_communication_demo/install
  3. Run the sample from the built binaries by using the
    -e ./
    argument. For example:
    ./tcc_rt_communication_demo.py --profile basic-a-opt --interface enp0s30f4 -v -m run -e ./
When you run the example below, make sure to go to the build directory and run the
./tcc_rt_communication_demo.py
(instead of
tcc_rt_communication_demo
) command with the
-e ./
argument.
To run the SISO example:
  1. [Board B]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the compute.
      tcc_rt_communication_demo --profile siso-single-b-opt --interface <interface name>
      Where
      Option
      Description
      --profile siso-single-b-opt
      Specify that the SISO profile is running on Board B, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board B.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l.
    • Starts phc2sys.
    • Sets the Linux* qdisc daemon to enable Time-Aware Traffic Scheduling and LaunchTime.
    • Runs iperf3 to generate network interference (optionally, see Command-Line Options for details).
  2. [Board A]
    1. Start a new terminal.
    2. Run
      ifconfig
      to get the interface name of the Ethernet controller, for example,
      enp1s0
      . Note the name for future reference.
      ifconfig
    3. Launch the monitor and actuator.
      tcc_rt_communication_demo --profile siso-single-a-opt --interface <interface name>
      Where
      Option
      Description
      --profile siso-single-a-opt
      Specify that the SISO profile is running on Board A, with optimizations.
      --interface <interface name>
      Specify the interface name of the Ethernet controller on Board A.
    The script does the following:
    • Sets up the network interface.
    • Starts ptp4l.
    • Starts phc2sys.
    • Sets the Linux* qdisc daemon to enable Time-Aware Traffic Scheduling and LaunchTime.
    • Runs iperf3 to generate network interference (optionally, see Command-Line Options for details).
  3. [Board A]
    1. Wait until the demo’s script finishes.
    2. Go to the following directory and check the output data file (
      log-roundtrip.log
      ):
      cd ./tcc_rt_communication_demo_output/ ls -la

Check the Quality of Time Synchronization Between Boards

This demo uses two additional programs
ptp4l
and
phc2sys
, which help with time synchronization between boards over the network and on boards between the PTP hardware clock and the system clock. Time synchronization starts automatically during demo execution. It is possible to restart time synchronization without other demo execution steps. For details, see the execution modes (clock) in Command-Line Options.
Each time synchronization application (ptp4l and phc2sys) has its own log file:
/var/log/ptp4l.log
for
ptp4l
and
/var/log/phc2sys.log
for
phc2sys
. These files can be opened by file-reading tools such as
cat
,
tail
, or
less
. For example, to view the
ptp4l
log file, you can use the following command:
cat /var/log/ptp4l.log
.
You can also list log file data directly during runtime to check the quality of time synchronization. Use the
tail
tool with
--follow
to view new log entries during runtime. For example, use the following command:
tail --follow /var/log/ptp4l.log
.
To check the time offset between primary and secondary clocks, you can use the
ptp4l
tool.
To check the offset between the PTP hardware clock and the system clock, you can use the
phc2sys
tool.
To check synchronization quality, open the corresponding log file using one of methods described above to see the following data:
  • Offset (in nanoseconds)
  • Frequency offset (in parts per billion, ppb)
  • Path delay (in nanoseconds)
The most important value is the offset in nanoseconds. For
ptp4l
, the tool shows the time offset between the primary and secondary clocks in nanoseconds. For
phc2sys
, the tool shows the time offset between the PTP hardware clock and the system clock. Generally, values less than 100 nanoseconds are an indicator of good time synchronization, but this value may vary and depends on hardware components. In this demo, a value less than 1 microsecond is acceptable.

Plot the Data From Log Files

On your host system, use
tcc_rt_communication_demo_plotter.py
to plot the data from
log_e2e.log
and
log-roundtrip.log
.
Prerequisites:
  • Python* release 3.7 and newer.
To plot the data:
  1. On the host system, source the environment file to set up environment variables:
    source ~/intel/oneapi/setvars.sh
  2. Go to the script directory:
    cd ${TCC_ROOT}/samples/plot_scripts
  3. Install required python modules:
    sudo pip3 install -r prerequisites.txt
    or
    sudo python3 -m pip install -r prerequisites.txt
  4. Copy
    tcc_rt_communication_demo_output/log_e2e.log
    and
    tcc_rt_communication_demo_output/log-roundtrip.log
    from the target system to the script directory in the host system.
  5. Run the script:
    python3 tcc_rt_communication_demo_plotter.py --input log_e2e.log --output e2e.png python3 tcc_rt_communication_demo_plotter.py --input log-roundtrip.log --output roundtrip.png
  6. You can specify multiple log files (for example, for optimized and non-optimized profiles) to show all statistics on one diagram:
    python3 tcc_rt_communication_demo_plotter.py --input log_e2e-opt.log log_e2e-noopt.log --output e2e.png
The script prints latency statistics to the console and generates images depending on arguments.

Analyze Demo Results

The results shown here may not be identically reproduced because inter-packet latency is very sensitive and may vary based on the duration of the test and the health and state of the platform. The format of the graph may differ based on your version of
matplotlib
and system specifications.
The transmitter measures workload execution time and dumps the data in the log file
tcc.stat
. To measure workload execution time, the transmitter uses the Measurement Library.
Demo tools measure transmission latency and dump the data in the log files,
log_e2e.log
and
log-roundtrip.log
. The first log is collected by the receiver for the
basic
mode, and the second log is collected for the
single-input single-output (SISO)
mode.
The graph below for
log_e2e-noopt.log
and
log_e2e-opt.log
shows an example of the transmission latency for the
basic
profile in the optimized and non-optimized modes. To measure transmission latency, the receiver (on Board B) tracks the time when a packet is received and measures the difference between the timestamp in the packet set by the transmitter (on Board A) and the current timestamp from the system. The left panel of the next image shows a histogram of transmission latency, where the x-axis shows transmission count and the y-axis shows time in microseconds. The right side shows the transmission latency of each packet.
The graph below for
log-roundtrip-noopt.log
and
log-roundtrip-noopt.log
shows an example of the roundtrip transmission latency for the
SISO
profile in the optimized and non-optimized modes. In this case, the transmitter sends a packet to the receiver and receives the corresponding packet from the receiver. The transmitter calculates the full roundtrip time based on the original timestamp set on Board A before sending the packet and the timestamp measured on receiving the packet back.

Command-Line Options

Usage: tcc_rt_communication_demo -p <profile name> -i <interface name> [-m <mode name>] [-n] [-t <target>] [-c <directory with profiles configuration>] [-o <output file>] [-e <path to binary executables>] [-v]
Option
Description
-p, --profile <profile name>
Required. Define the profile name to be executed. The following profiles are supported by the demo application:
  1. Basic, with and without optimizations:
    • basic-a-opt
    • basic-b-opt
    • basic-a-noopt
    • basic-b-noopt
  2. SISO, with and without optimizations:
    • siso-single-a-opt
    • siso-single-b-opt
    • siso-single-a-noopt
    • siso-single-b-noopt
-i, --interface <interface name>
Required. Define the network interface name to be used for TSN- based data transmission. For example: enp1s0.
-m, --mode <mode name>
Optional. Define the operation mode to customize demo behavior. The following modes are supported by the demo application:
  • all (default): run all modes/stages: TSN configuration, clock synchronization, and run the demo application
  • clock: (Re-)start ptp4l and phc2sys clock synchronization
  • tsn: Configure the network interface and the traffic control for TSN
  • run: Only run the demo application; TSN and clock should already be configured
-n, --no-best-effort
Optional. Do not use best-effort traffic (iperf3) together with other components of the demo application.
-t, --target <target>
Optional. Define the type of target board to select configuration files for execution. Available types are:
  • AUTO (default): autodetect the target
  • EHL: use configuration for Intel Atom® x6000E Series processors
  • TGL-U: use configuration for 11th Gen Intel® Core™ processors
-c, --config-path <directory with profiles configuration>
Optional. Define the directory with the profiles’ configuration files. ‘./cfg/’ is used by default. The directory should contain two sub-directories:
  • basic
    for the Basic profiles’ configuration files
  • siso-single
    for the SISO profiles’ configuration files
-o, --output-file <output file>
Optional. Define the filename for storing the output statistics.
  • ./tcc_rt_communication_demo_output/log_e2e.log
    is used by default for the Basic profiles
  • ./tcc_rt_communication_demo_output/log-roundtrip.log
    is used by default for the SISO profiles
-e, --exec-path <directory with binary executables>
Optional. Define the directory with binary executables and shared libraries such as opcua_server and libtcc_rtcdatamgr.so. The demo uses these binary executables from the general system directory with binaries as per default and thus they are available by searching through the PATH environment. This directory is used before the directories from the PATH environment variable.
-v, --verbose
Optional. Enable the verbose mode.
-h, --help
Optional. Print command-line help.
Example
tcc_rt_communication_demo -p basic-a-opt -i enp1s0 -m run -n

Troubleshooting

Option
Description
tcc_rt_communication_ demo
reports:
Unknown interface name was specified.
Specify valid interface name (check with ifconfig tool,
for example).
Make sure you provide the correct interface name of the TSN-compatible Intel® Ethernet Controller:
Run:
lspci -v
Locate the Peripheral Component Interconnect (PCI) bus information (example: 01.00.0) of the TSN-compatible Intel® Ethernet Controller:
Run:
ethtool -i <enp1s0>
and find the bus information (example: 0000:01:00.0) that matches the previously found address.
tcc_rt_communication_ demo
reports:
Directory <path> with
configuration profiles does not exist.
Make sure you provide the correct path to the directory containing configuration files for the demo. If the directory does not exist, create it and add subdirectories containing configuration files for the basic and SISO modes.
tcc_rt_communication_ demo
reports:
Path to output file (<path name>)
does not exist.
Make sure you provide the correct path to the directory for the output file. If the directory does not exist, create it.
tcc_rt_communication_ demo
reports:
Profile <profile name> not supported.
Use [-h] to see help information.
Make sure to run the
tcc_rt_communication_demo
script with a supported profile. Supported profiles are: basic-a-opt, basic-a-noopt, basic-b-opt, basic-b-noopt, siso-single-a-opt, siso-single-a-noopt, siso-single-b-opt, and siso-single-b-noopt.
tcc_rt_communication_ demo
reports:
Mode <mode name> not supported.
Use [-h] to see help information.
Make sure to run the
tcc_rt_communication_demo
script with a supported mode. Supported modes are: clock, tsn, run, and all.
tcc_rt_communication_demo
reports:
FATAL ERROR: Binary "opcua_server" was not found in directories provided by PATH environment variable. Try to launch script with [-e] option and provide path to binary executables manually.
For systems running the PR1 release of the Yocto Project*-based BSP for Intel Atom® x6000E Series processors, see the documentation about rebuilding the demo and running it with the [-e] option.
tcc_rt_communication_ demo
reports:
Binary "<binary name>" was not found in
directories provided by PATH environment variable.
Try to launch script with [-e] option and provide path
to binary executables manually.
Make sure the following binary executables are installed in one of the directories indicated by the PATH environment variable:
  • opcua-server
If the binaries are not installed in one of these directories, specify the correct directory using the [-e] CLI option.
tcc_rt_communication_ demo
reports:
Path to directory with binary
executables (<path name>) does not exist.
Make sure you provide the correct path to the directory with the following binary executables:
  • opcua-server
If the directory does not exist, create it and place the binaries into it.
Loss of connectivity with the board after using a network interface name that is not related to the internal communication between Board A and Board B.
Restart the board using one of the following methods:
  1. Through the GUI shell interface if the board is connected to a keyboard, video, and mouse (KVM) device.
  2. By pressing the reset button.
Loss of connectivity between Board A and Board B during demo execution.
Restart the boards using one of the following methods:
  1. Through the GUI shell interface if the board is connected to a keyboard, video, and mouse (KVM) device.
  2. By pressing the reset button.
The transmission latency is too long for the SISO example, more than 100 ms.
Make sure the compute and the monitor select the same base time (can be checked in the console logs); if the base time is different, try launching the compute and the monitor at the same time (with the time difference of less than 2 seconds).
The transmission latency is very large or negative.
The time synchronization is lost. Check the following log files:
/var/log/ptp4l.log
and
/var/log/phc2sys.log
on both boards. Run the
tcc_rt_communication_demo
script with added [-m clock] option. This command will restart the time synchronization.
tcc_rt_communication_demo
in the optimized mode fails to start. The console logs show:
Error: cannot allocate buffer for the pointer chasing workload
.
Possible solutions:
  • Make sure you have completed the steps in the Get Started Guide with each board.
  • Make sure the
    Intel® TCC Mode
    and
    Software SRAM
    options are enabled in BIOS.
  • Make sure you have enabled the runtime support for software SRAM on each board by running the
    tcc_setup_ssram.sh enable
    script.
  • Make sure the L2 buffer size setting in the configuration file
    ${HOME}/.tcc.config
    is at least 131072 on both boards.
  • Export the
    TCC_LOG_LEVEL=DEBUG
    environment variable and run the sample again. This variable enables the console logs for the cache allocation library to show errors about the software SRAM allocation.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.