Intel® Edge Software Device Qualification (Intel® ESDQ) for Vision

Version: 6.0   Published: 04/19/2021  

Last Updated: 03/02/2022

Overview

Intel® Edge Software Device Qualification (Intel® ESDQ) for Vision package provides customers with the capability to run an Intel provided Vision test suite at the target system, with the goal of enabling partners to test their platform for Intel® Edge Insights for Vision package.  

The information below is specific to the Intel® ESDQ for Vision package test modules. For documentation on the Intel® ESDQ CLI binary, refer to Intel® Edge Software Device Qualification (Intel® ESDQ) Overview

Select Configure & Download to download Intel® ESDQ for Vision and refer to Get Started for installation steps.  

Configure & Download


Target System Requirements

  • One of the following processors:
    • 6th, 7th, 8th, 10th, 11th, 12th Gen Intel® Core™ processor.
    • 6th, 7th, 8th, 10th, 11th, 12th Gen Intel® Xeon® processor.
    • Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics.
  • At least 16 GB RAM.
  • At least 64 GB hard drive.
  • A stable Internet connection.
  • Ubuntu* 18.04 LTS with Kernel 5.4 or above.
  • Ubuntu* 20.04 LTS with Kernel 5.4 or above.
  • Kernel of 5.4 and above needed to support 10th and 11th Gen Intel® Core™ processors.
  • Kernel of 5.11 needed to support 12th Gen Intel® Core™ processor, Intel Atom® x6000E processor series, Pentium® processors, and Intel® Celeron® N and J processors series.
  • Intel® Vision Accelerator – Intel® Movidius™ Myriad™ X (Optional).

Test Modules

Intel® ESDQ Vision contains the following test modules.

Intel® Edge Insights for Vision Test Module Overview

Intel® Edge Insights for Vision Test Module is a validation framework for Intel® Media SDK, Intel® Distribution of OpenVINO™ toolkit 2021.4.2 in a Container and Intel® Distribution of OpenVINO™ toolkit 2021.4.2 Runtime. 

The Intel products mentioned above are intended for Intel partners (ODMs, OEMs, SIs). Our partners may choose to have customized derivative builds of Intel’s reference designs, and their performance numbers may vary from that of Intel’s designs. This calls for a base framework that allows partners to compare their design’s performance and generate a hardware health status report.  

Edge Insights for Vision Test Module contains scripts to validate the Edge Insights for Vision package. The test suite serves the following purpose: 

  • Health Check: It runs a health check to make sure all the components that are part of the vision package listed below are properly installed and configured correctly. 
    • Reference Implementation - Multi-Camera Detection for Social Distancing
    • Intel® Distribution of OpenVINO™ toolkit 2021.4.2 in a Container 
    • Intel® Distribution of OpenVINO™ toolkit 2021.4.2 Runtime 
    • Intel® Deep Learning Streamer (Intel® DL Streamer) 
    • EdgeX* Foundry 
    • Docker* Community Edition (CE) 
    • Docker Compose 
    • K3s* (Lightweight Kubernetes*)
    • Azure* IoT Edge Prerequisites 
    • Intel® Edge Software Device Qualification
  • Sanity tests: The test module contains sanity tests for Azure/Kubernetes K3S. 
  • Functionally tests at component level: For each of the Intel® Media SDK and OpenVINOTM toolkit components, the test suite runs a series of sample applications such as decode, transcode, encode & object detection on CPU, GPU and Intel Vision accelerators (if present) and validates the output of the sample applications to make sure that all the components work as expected. 
    • MediaSDK_decode_h264 
    • MediaSDK_transcode_h264_mpeg2 
    • MediaSDK_encode_h265 
    • MediaSDK_setcommands_h264 
    • MediaSDK_setcommands_h265 
    • MediaSDK_sample_vpp_sw
    • Openvino_CPU_reshape 
    • Openvino_CPU_styletransfer 
    • Openvino_GPU_reshape 
    • Openvino_GPU_ styletransfer 
    • Openvino_HDDL_reshape 
    • Openvino_HDDL_styletransfer 
    • Openvino_CPU_objectdetection

    • Openvino_CPU_objectdetection_ssd_async  
    • Openvino_GPU_objectdetection  
    • Openvino_GPU_objectdetection_ssd_async 
    • Openvino_HDDL_objectdetection  
    • Openvino_HDDL_objectdetection_ssd_async 
    • Openvino_container_CPU_objectdetection  
    • Openvino_container_CPU_ image_processing  
    • Openvino_container_GPU_objectdetection  
    • Openvino_container_GPU_ image_processing  
    • Openvino_container_HDDL_objectdetection  
    • Openvino_container_HDDL_ image_processing

 


Intel® Vision Accelerator Test Suite (Intel® VATS) Overview

Intel® VATS is an OpenVINO™ toolkit based extensible validation framework, which is scalable and portable. The Intel® VATS automated test framework is for Intel® Vision Accelerators, such as the Intel® Movidius™ Myriad™ X VPU platform. 

Intel® Vision Accelerator Design products are reference designs intended for Intel partners (ODMs, OEMs, SIs). Our partners may choose to build custom derivatives of Intel's reference designs, whose performance numbers may deviate from that of Intel's designs. This calls for a base framework that allows partners to compare their designs' performance and stability numbers to those advertised by Intel. 

Intel® VATS has categorized test cases into the following categories to test different functionalities of Intel® VATS:  

Usability tests: This set of tests will ensure Intel® Movidius™ Myriad™ X VPU firmware is loaded in a reasonable time and is able to reset successfully. After the reset, the test ensures the Intel® Movidius™ Myriad™ X VPU driver is responding to the hardware ping and can receive VPU IO responses from OpenVINO™ toolkit. 

  • firmware_loading_time 
  • reset_1 
  • hardware_ping 
  • io_ping_test 

Performance tests: This set of tests will run OpenVINO™ toolkit benchmark_apps such as alexnet, densenet, googlenet, mobilenet, squeezenet and resnet to acquire latency and throughput performance numbers. 

  • performance_alexnet 
  • performance_densenet-121 
  • performance_googlenet-v1 
  • performance_mobilenet-ssd 
  • performance_mobilenet-v1-1.0-224 
  • performance_mobilenet-v2 
  • performance_squeezenet1.1 
  • performance_resnet-50-pytorch 

Reliability test: This test will capture the thermal reading of the Intel® Movidius™ Myriad™ X VPU device after loading the neural network and running inference via benchmark_app. 

  • thermal_test 

Test cases being run using Intel® ESDQ test 

1. firmware_loading_time 

  • This test will load the model to the Intel® Movidius™ Myriad™ X VPU device through the OpenVINO™ toolkit inference engine library. It will measure the time taken to successfully load the model and print the result. The offload time can be acquired from output/firmware_loading_time.log. 
  • Passing Criteria: The test passes if the models can be offloaded to Intel® Movidius™ Myriad™ X VPU and time taken to offload the model can be acquired successfully. Below is a snippet of the test result.  

Screenshot of firmware loading test

2. reset_1 

  • This test will reset the Intel® Movidius™ Myriad™ X VPU device to its initial state. The status of the Intel® Movidius™ Myriad™ X VPU device after reset can be acquired from output/reset_1.log. 
  • Passing Criteria: The Intel® Movidius™ Myriad™ X VPU is successfully reset to its initial state. Below is a snippet of the test result.

Screenshot of reset test

3. hardware_ping 

  • This test will ping Intel® Movidius™ Myriad™ X VPU driver information before and after firmware is loaded with a specific model. The driver status can be acquired from output/hardware_ping.log. 
  • Passing Criteria: The test passes if the Intel® Movidius™ Myriad™ X VPU driver information is present before and after the firmware is loaded. Below is a snippet of the test result.

Screenshot of hardware ping test

4. io_ping_test 

  • This test will check Intel® Movidius™ Myriad™ X VPU IO responses via OpenVINO™ toolkit inference engine API. The ping result can be acquired from output/io_ping_test.log. 
  • Passing Criteria: The test passes if the Intel® Movidius™ Myriad™ X VPU device can successfully receive VPU IO responses from the OpenVINO™ toolkit inference engine API. Below is a snippet of the test result.

Screenshot of io ping test

5. Performance test 

  • This test will run OpenVINO™ toolkit benchmark_app to acquire latency and throughput performance numbers. 
  • Passing Criteria: The test passes if inference for specific neural networks can be run without error and Latency and Throughput values can be acquired. Intel® VATS will not do benchmarking of inference performance. The performance_. log can be acquired from output/performance_.log. Below is a snippet of the test result.

Screenshot of performance test

6. thermal_test 

  • This test will capture the thermal reading of the Intel® Movidius™ Myriad™ X VPU device after loading the neural network and running inference via benchmark_app. The thermal reading can be acquired from output/thermal_test.log
  • Passing Criteria: The test passes if the thermal reading can be captured. Below is a snippet of the test result.

Screenshot of thermal test


Get Started 

Intel® ESDQ CLI tool is installed as part of the Edge Insights for Vision package that uses OpenVINO™ 2021.4.2. An upgrade from prior Intel® ESDQ versions is also available.  

NOTE: Be aware that screenshots may show a package version number that is different from the current release. See the Release Notes for information on the current release.

Download and Install Intel® ESDQ for Vision

1. Select Configure & Download to download Intel® ESDQ for Vision and then follow the steps below to install it.  

Configure & Download

Configure and download

 

2. Select Version or Tag, Target System and Distribution options based on your system environment. Click the Download button to download Intel® ESDQ for Vision.Accept the license agreement.

Installer showing download progress

 

3. Copy and Save the Product Key

Product Key dialog

 

4. Click the Edit button to reconfigure the options and download another version of ESDQ for Vision.

5. Transfer the downloaded package to the target Ubuntu system. 

6. View the edgesoftware_configuration.xml included in the edge software zip package and look for the Intel® ESDQ ingredient and test modules for Intel® VATS and Vision package. 

<project id="6216928e8bad76002bce2ef4" label="esdq/Vision_Test_Module" path="installation/src" version="2021.4.1" /> 
<project esb_install="true" id="61969a82ff6f230021e43d7c" label="esdq" path="installation/src" version="6.0.0" /> 
<project id="60f084f3acbb0a002bcaee71" label="esdq/VATS_Test_Module" path="installation/src" version="2021.4" />  

7. Provide executable permission to edgesoftware. 

chmod +x ./edgesoftware

8. Run the command below to install Intel® ESDQ CLI and the test modules (Vision and Intel® VATS). 

./edgesoftware install

9. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download (or from the saved Product Key in the step above). 

10. When the installation is complete, you see the message “Installation of package complete” and the installation status for each module. 

Screenshot of installation complete

11. After modules installation is completed successfully, reboot the system:

Reboot

  


Run the Application

For the complete Intel® ESDQ CLI, refer to Intel® Edge Software Device Qualification CLI Overview.  The test modules are already available in the target system. 

1. Change directory to the location of Intel® ESDQ folder using the command:

cd <installed_dir>/edge_software_device_qualification/Edge_Software_Device_Qualification_6.1/esdq

2. Run Intel® ESDQ test and generate report: 

./esdq run –r 

The report should look like the following example:

Test report summary and system info

Test report modules info

Test report test suites

Test report test modules

 


Release Notes 

Intel® Edge Insights for Vision Test Module 

Current Version: 2021.4.2    

  • Intel® Edge Insights for Vision Test Module is upgraded to version 2021.4.2 to support OpenVINO™ toolkit 2021.4.2 and will not provide backward compatibility with older OpenVINO™ toolkit versions.
  • This current release supports both Ubuntu* 18.04 and 20.04.  
  • For 9th - 11th Generation Intel® Core™ processors, Kernel 5.4 is needed. For 12th Generation Intel® Core™ processors, Intel Atom® x6000E Series, Intel® Pentium® processors, and Intel® Celeron® N and J Series processors, Kernel 5.11 is needed to get the GPU working. There are known issues with Media SDK test. See the Known Issues section for more details.

New in this Release    

  • Container OpenVINO™ “super _resolution_demo”  tests are replaced with “image processing demo” tests for the device options of CPU and GPU.
  • Fixed Media sdk tests to support 12th Gen Intel® Core™ processors.  
  • AWS Greengrass is removed from the Vision Package.

Known Issues    

  • To get the GPU working on latest Gen Intel Atom® x6000E processor series and Pentium® processor and Intel® Celeron® processor N and J Series, Kernel 5.11 is needed along with the following:
    • Install intel-compute-runtime  
    • Add ID to grub (if the driver is not yet enabled in i915)
    • On platforms that do not have HEVC VDEnc enabled, H265 encode is not supported. In this case, “mediasdk_sample_encode” test case will fail.

 

Intel® Vision Accelerator Test Suite (Intel® VATS) 

Version: 2021.4.2   

  • Intel® VATS supports OpenVINO™ toolkit 2021.4.2

New in this Release    

  • Validated Intel® VATS on  OpenVINO™ toolkit 2021.4.2 version in ESDQ 6.0.

Known issues    

  • Intel® VATS tests are not executed if HDDL device is not updated with correct driver information.

 


Troubleshooting

Intel® Edge Insights for Vision Test Module 

Network Requirements

If faced with one of the following errors, check your firewall/proxy settings. You may update apt source to your local mirror site or run Intel® ESDQ in VPN network:

  • Failed to install python requirements
  • Failed to clone the GIT REPO” for model downloads
  • Request timed out after 3 minutes Unable to connect to github

Intel® VATS Prerequisites  

Ensure that the OpenVINO™ toolkit package is installed on the machine. The OpenVINO™ toolkit version used as part of this Intel® ESDQ package is 2021.4.752. It can also be downloaded here.  

Dependencies 

OpenVINO™ toolkit: This assumes all required OpenVINO™ toolkit and Intel® VATS and VPU drivers and dependencies are installed by installing the Edge Insights for Vision package from the Intel® Developer Catalog. If any of the modules are missing on the platform, follow the guides below to ensure the required dependencies are installed. 

Run the Intel® ESDQ command as mentioned in the section Intel® ESDQ 6.1 Download and Install to install all the dependencies and initiate the test execution. The approximate time for the Intel® VATS test to complete is 10 minutes. 

Below are the troubleshooting steps if you encounter the following error in Intel® VATS. 

Running inference is failed with the following error generated: 

E: [ncAPI] [ 965618] [MainThread] ncDeviceOpen:677 Failed to find a device, rc: X_LINK_ERROR

 Possible solution 

  1. Reboot the host system and re-run. 
  2. Hard stop the HDDL Plugin backend service (hddldaemon) and reset all Intel® Movidius™ VPUs before running an application. 
kill -9 $(pidof hddldaemon autoboot) pidof hddldaemon autoboot  
# Make sure none of them is alive 
source /opt/intel/openvino/bin/setupvars.sh  
${HDDL_INSTALL_DIR}/bin/bsl_reset 

Reboot the system. 

For more information, please refer to this site.  

      
openvino GPU tests failed 

[ ERROR ] Failed to create plugin /opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/libclDNNPlugin.so for device GPU 

Please, check your environment 

[CLDNN ERROR]. clGetPlatformIDs error -1001 

Possible Solution 

To access GPGPU capabilities, you need to have the correct permissions. The following will list the group assigned ownership of the render nodes, and list the groups you are a member of: 

stat -c "%G" /dev/dri/render*
groups ${USER}

If a group is listed for the render node, which isn’t listed for you, you will need to add yourself to the group using gpasswd. In the following, you will be added to the ‘render’ group and a new shell spawned with that group active: 

sudo gpasswd -a ${USER} render 
newgrp render

 

undefined symbol: hddl_check_device 

RuntimeError: Failed to create plugin /opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/libHDDLPlugin.so for device HDDL 

Please, check your environment 

Cannot load library '/opt/intel/openvino_2021/deployment_tools/inference_engine/lib/intel64/libHDDLPlugin.so': /opt/intel/openvino_2021/deployment_tools/inference_engine/external/hddl/lib/libmvnc-hddl.so.0: undefined symbol: hddl_check_device 

Possible Solution 

The connected model of VPU is related to IEI Mustang-V100-MX8-R10 accelerator; see Intel® Movidius™ VPUs Setup Guide for Use with Intel® Distribution of OpenVINO™ toolkit to follow additional steps. IEI Mustang-V100-MX8-R11 accelerator doesn't require any additional steps.

Intel® Distribution of OpenVINO™ toolkit Prerequisites

GPU Operations

To get GPU working on Intel Atom® x6000E Series processors, Intel® Pentium® processors, and Intel® Celeron® N and J Series processors, the following steps need to be followed:

 

Note: Requires Kernel 5.11 or above.  

Install intel-compute-runtime driver:    

sudo add-apt-repository ppa:intel-opencl/intel-openclsudo apt updatesudo apt install -y libze-loader libze-intel-gpu intel-opencl-icd

Add PCI device ID to grub (if the driver is not yet enabled in i915):    

sudo nano /etc/default/grubfind GRUB_CMDLINE_LINUX_DEFAULTGRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.force_probe=4571"###   

sudo update-grub

  

Media sdk sample encode test failed  

Media sdk sample encode test failed with the message: 

“ERROR], sts=MFX_ERR_UNSUPPORTED(-3),…"  

Follow the steps below as a possible solution: 

  1. sudo apt install --reinstall linux-firmware 
  2. Add i915.enable_guc=2  in /etc/default/grubGRUB_CMDLINE_LINUX_DEFAULT="quiet splash i915.force_probe=* i915.enable_guc=2" 
  3. sudo update-initramfs –u 
  4. sudo update-grub 
  5. Reboot the system.
  6. After reboot, execute the command:  dmesg | grep guc
  7. Verify the logs for guc information. If output doesn't contain guc details, then get the latest guc/huc firmware. Copy the firmware to /lib/firmware/i915/, and reboot the system.

              

Support Forum

If you're unable to resolve your issues, contact the Support Forum

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.