Intel® Edge Software Device Qualification (Intel® ESDQ) for AI Box for Video Analytics

Version: 2.5   Published: 09/29/2021  

Last Updated: 09/09/2022

Overview

Intel® Edge Software Device Qualification (Intel® ESDQ) for Edge AI Box for Video Analytics provides customers with the capability to run an Intel provided test suite at the target system, with the goal of enabling customers to self-qualify on their platform.   

The information below is specific to the Intel® ESDQ for Edge AI Box for Video Analytics package. For documentation on the Intel® ESDQ CLI binary, refer to Intel® Edge Software Device Qualification (Intel® ESDQ) Overview.

Select Configure & Download to download Intel® ESDQ for Edge AI Box for Video Analytics and refer to Get Started for installation steps.  

Configure & Download


Target System Requirements 

  • 11th or 12th Generation Intel® Core™ processors and Celeron® processors
  • Operating System: 
    • Ubuntu* 20.04 with Kernel Version 5.8 or newer  
    • Ubuntu* 20.04 with Kernel Version 5.17.15 or newer for 12th Generation Intel® Core™ processors  

NOTE: Ubuntu 20.04.4 comes with Kernel version 5.13.0-51 as default. In order to upgrade your Kernel version to 5.15 or higher, follow the instructions on the Ubuntu website.

  • At least 128 GB disk space 
  • At least 8 GB (4 GB x2) memory with 8 GB swap space 
  • Direct Internet Access
  • A stable internet connection

 


How It Works

The test module for Edge AI Box for Video Analytics interacts with the Intel® ESDQ CLI through a common test module interface (TMI) layer, which is part of the Intel® ESDQ binary.

Selected components from customized download options will be validated. Execute the automated test suite. 

Test results will be stored in the output folder. Intel® ESDQ generates a complete test report in HTML format, along with detailed logs packaged as one zip file, which you can manually choose to email to the ESH support team.

AI Box Test Module 

The AI Box test module is a validation framework for Edge AI Box for Video Analytics. It validates the installation status as part of health check. The test module includes test cases for Intel® Distribution of OpenVINO™ toolkit 2022.1 Runtime along with OpenVINO™ Benchmark performance test. It also includes test cases for other modules of the AI Box package, such as AWS* Greengrass, Azure* PnP bridge and Tensorflow* as shown below: 

  • Installation of AWS CLI
  • AWS Greengrass installation
  • Installation of Azure CLI
  • Installation of Azure explorer
  • Azure PnP bridge repo sample path
  • Installed Tensorflow version
  • Installed OpenVINO-Tensorflow version
  • classification_sample_async with CPU
  • hello_reshape_ssd with CPU
  • hello_query_device
  • hello_classification_sample with CPU
  • hello_nv12_input_classification with CPU
  • run_tc_benchmark_app with FP32_googlenet-v1_CPU
  • Image Classification Async Python* sample with CPU
  • hello_query_device Python sample
  • hello_classification Python sample with CPU
  • hello_reshape_ssd Python sample with CPU
  • classification_sample_async with GPU
  • hello_reshape_ssd with GPU
  • hello_classification_sample with GPU
  • hello_nv12_input_classification with GPU
  • run_tc_benchmark_app with FP32_googlenet-v1_GPU
  • Image Classification Async Python sample with GPU
  • hello_classification Python sample with GPU
  • hello_reshape_ssd Python sample with GPU

 

Health Check

The test suite runs a health check to make sure all the components that are part of the AI Box package listed below are properly installed and configured correctly. 

  • Intel® Distribution of OpenVINO™ toolkit 2022.1 Runtime 
  • Intel® Edge Software Device Qualification
  • Tensorflow suggested version 
  • Azure PnP bridge suggested version 
  • AWS Greengrass suggested version

AI Box Metrics Test

The AI Box Metrics (aibox_metrics) test consists of pipeline validation and inference benchmark functionality. The pipeline test processes a video file and creates a new video file annotated with analytic results. The functionality contains six stages: video decode, object detection, object tracking, object classification, metadata watermarking and video encode.

To verify the functionality, the test downloads the required models and model-proc files from GitHub repositories for Open Model Zoo for OpenVINO™ toolkit and Intel® Deep Learning Streamer (Intel® DL Streamer). The test reports of 4 pipeline streams captured and generated the content under aibox_metrics_pipelineTestName with throughput information for Vehicle_Attribute_Pipeline_CPU and Vehicle_Attribute_Pipeline_GPU along with test log file information in the ESDQ report HTML page.

The test artifact folder has a single frame extracted watermark video images as cpuPipeline_vehicle_attribute_00.1.jpg for CPU pipeline test and gpuPipeline_vehicle_attribute_00.1.jpg for GPU pipeline test. 

Intel® Distribution of OpenVINO™ Toolkit Test

AI Box 2.5 test module supports Intel® Distribution of OpenVINO™ toolkit 2022.1.For each of the Intel® Distribution of OpenVINO™ toolkit components, the test suite runs a series of sample applications such as style transfer, reshape and object detection on CPU and GPU. The test suite validates the output of the sample applications to make sure that all the components work as expected.

Benchmark Performance Test

The benchmark tests demonstrate high performance gains on several public neural networks on multiple Intel® CPUs and GPUs covering a broad performance range. Use this data to help decide which hardware is best for applications and solutions, or to plan AI workload on the Intel computing already included in the solutions. 

OpenVINO™ benchmark application tests are executed with inference models of resnet-50-tf, ssdlite_mobilenet_v2, yolo-v3-tiny-tf, yolo-v4-tf and googlenet-v1 on target device as CPU and GPU. The metrics of Latency and Throughput for these models are provided in ESDQ report HTML page.

You can see the benchmark application test results under the output folder. The complete execution log of benchmark app tests can be found in the AIBox_BmTestOutput.out file.  The actual inference throughput results can be read in the log file. The CSV file (benchmark.csv) output information will appear as shown below. 

A console window showing system output of Benchmark results in CSV file showing latency and throughput.
Figure 1. Benchmark Results in CSV File

 

NOTE: In Intel® Distribution of OpenVINO™ toolkit 2022.1 Yolo-v3 models are no longer supported. They are replaced with Benchmark application tests that are provided in Intel® Distribution of OpenVINO™ toolkit. 

For AI Box 2.5, Intel® Distribution of OpenVINO™ toolkit 2022.1 Benchmark application tests are executed in asynchronous mode on CPU and GPU devices. The tests captured Latency and Throughput results, which is displayed in the reports as FP32_googlenet-v1_CPU and FP32_googlenet-v1_GPU under aibox_metrics_benchmarkTestName. 

Intel® Media SDK Test (not supported in AI Box 2.5 release) 

Intel® Media SDK enables hardware acceleration for fast video transcoding, image processing and media workflows. For each of the Intel® Media SDK components, the test suite runs a series of sample applications such as decode, transcode, encode with respect to CPU and GPU. The test suite validates the output of the sample applications to make sure that all the components work as expected.   

NOTE: The Intel® Media SDK is deprecated in Intel® Distribution of OpenVINO™ toolkit 2022.1. Hence, the corresponding tests are removed in AI Box 2.5 and later versions.

Smart Video and AI Workload Reference Implementation Test (not supported in AI Box 2.5 release)

The SVET test included in the AIBOX Test Module validates the Smart Video AI Workload Reference Implementation that is part of the package. The test sources the Intel® Distribution of OpenVINO™ toolkit and Smart Video and AI Workload environment setup and executes the video_e2e_sample with the par_file entry. To run the video_e2e_sample, the script prompts for the super user password. This test performs the face detection scenario with the 16 channels using the 1080p.h264 video. The output is then collected in the log file.

 

Get Started 

Intel® ESDQ CLI tool is installed as part of the Edge AI Box for Video Analytics package. 

NOTE: Be aware that screenshots may show a package version number that is different from the current release. See the Release Notes for information on the current release.

 

Prerequisites

Follow the instructions in the Edge AI Box for Video Analytics article Get Started section. 

Download and Install Intel® ESDQ for Edge AI Box for Video Analytics

  1. Select Configure & Download to download Intel® ESDQ Edge AI Box for Video Analytics and then follow the steps below to install it. Click on Customize Download to select the configuration.
    Configure & Download
    Browser window showing the configure and download screen. It says "Select options below to download," with options being version or tag, target system, distribution, and download.
    Figure 2. Configure & Download Page
  2. Choose the required configuration based on your test system environment. We recommend you select Select Version or Tag as 2.5, Target System as Ubuntu 20.04 LTS, Environment as Development, and Distribution as Customize Download on the displayed options screen.
  3. Click the Customize Download button to download the Intel® Edge Software Device Qualification (Intel® ESDQ) for Edge AI Box for Video Analytics.
  4. Click the Edit button to reconfigure the options and download another version of Intel® ESDQ for AI Box for Video Analytics in subsequent releases.
    A partial view of a web app window showing the Edit Download options.
    Figure 3. Edit Download

 

  1. Click the Next button until the Download button appears (screen 6 of 6). 
    Browser window showing screen 1 of the Customize Download set of screens showing OpenVINO.
    Figure 4. Customize Download - Screen 1

 

  1. You will notice one of the components in the list is Intel® Edge Software Device Qualification. Click the Download button.
    Browser window showing Screen 6 of the Customize Download set of screens showing the Download option.
    Figure 5. Customize Download - Screen 6
  1. Select Accept License Agreement on the pop-up window, then follow the instructions below.
  2. Copy and Save the Product Key
    Browser window of Installer showing 3 steps to get started: Prepare the target system; copy the zip file to your target system; and extract and install the software.
    Figure 6. Product Key

 

  1. Transfer the downloaded package to the target Ubuntu* system.
  2. View the edgesoftware_configuration.xml included in the edge software zip package and look for the Intel® ESDQ ingredient and test modules for Edge AI Box for Video Analytics. 
    <project path="installation/src" id="62f48b834d3c980023d23111" version="2.5" label="esdq/Aibox_Test_Module"/>
    <project path="installation/src" id="62f3c2590f5af50020aa44d2" version="9.0" label="esdq" esb_install="true"/> 
    
  3. Provide executable permission to edgesoftware
    chmod +x ./edgesoftware
  4. Run the command below to install Intel® ESDQ and Edge AI Box for Video Analytics. 
    ./edgesoftware install
  5. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download (or from the saved Product Key in the step above). 
  6. When the installation is complete, you see the message Installation of package complete and the installation status for each module. 
    A console window showing system output during the install process. At the end of the process, the system displays the message “Installation of package complete” and the installation status for each module.
    Figure 7. Installation Status
 
  1. Find the module ID for the Intel® ESDQ by running the command: 
    ./edgesoftware list --default

    Example output:  
     

    A system console window showing the output of the Edgesoftware list command. The installed modules are listed with their ID, name, and version information.
    Figure 8. Edgesoftware List

 

  1. After module installation is completed successfully, reboot the system:
    reboot

 


Run the Application

For the complete Intel® ESDQ CLI, refer to Intel® ESDQ CLI Overview.  The test modules are already available in the target system.

  1. Change directory with the command: 
    cd $<ESH_Installed_directory>/edge_ai_box_for_video_analytics/Edge_AI_Box_for_Video_Analytics_2.5/esdq
  2. Run Intel® ESDQ test and generate report:
    ./esdq run –r 
       

The test report should look like the following examples:

NOTE: The results shown below are for illustration only.

Test report showing summary info.

 

 

Test report showing System Info

 

Test report showing Module Info with Pass or Success status.

 

Test Report showing Test Suites with Pass or Success status.

 

Test Report showing Latency and Throughput info.

 


Release Notes 

Current Version: 2.5

New in This Release   

  • Updated the test module to support Intel® Distribution of OpenVINO™ toolkit 2022.1 Samples. 
  • Removed SVET, Intel® Media SDK and Yolo tests.
  • Included the functionality test of Pipeline and Inference Benchmark requirements. 

 

Version: 2.0

New in This Release   

  • Integrated Intel® Distribution of OpenVINO™ toolkit 2021.4.2. 
  • Included the functionality test of Azure IoT PnP Bridge, Intel® Distribution of OpenVINO™ toolkit TensorFlow Bridge and Amazon Web Services Greengrass.

Known Issues

  • Smart Video and AI Workload Reference Implementation test requires the user to feed in the root password. However, when Intel® ESDQ is run on a graphical interface, the system switches to console mode during SVET execution and the control to Intel® ESDQ terminal that displays the password prompt is lost. This results in a hang situation. To work around this issue, login to the system using SSH terminal and execute the Intel® ESDQ tests.

Version: 1.2

New in This Release  

  • Latency and Throughput information included in HTML report. 
  • Included the functionality test of Smart Video and AI Workload Reference Implementation.

Version 1.0

New in This Release  

  • Initial features for recommended configuration. 

Known Issues 

  • If yolo-v3-tf.xml is not downloaded, then Latency and Throughput metrics will be null in the Intel® ESDQ HTML report page.

Troubleshooting


Check your firewall/proxy settings if any of the following errors occur:

  • Failed to install Python requirements
  • Failed to clone the GitHub repository for OpenVINO™ model downloads
  • Request timed out after 3 minutes Unable to connect to github

 

FFmpeg is Not Available

  • If FFmpeg is not available on the Ubuntu system, run the below commands:
    sudo apt update
    sudo apt install ffmpeg 

     

Support Forum

If you're unable to resolve your issues, contact the Support Forum

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.