Intel® Edge Software Device Qualification (Intel® ESDQ) for Intel® Video AI Box

ID 680138
Updated 10/10/2023
Version 3.0
Public

author-image

By

Overview

Intel® Edge Software Device Qualification (Intel® ESDQ) for Intel® Video AI Box allows customers to run an Intel-provided test suite at the target system, aiming to enable partners to qualify on their platform. 

The following information is specific to the Intel® ESDQ for Intel® Video AI Box package. For documentation on the Intel® ESDQ CLI binary, refer to Intel® Edge Software Device Qualification (Intel® ESDQ) Overview.

Select Configure & Download to download Intel® ESDQ for Intel® Video AI Box from the Intel® Video AI Box page and refer to Get Started for the installation steps.  

Configure & Download

Target System Requirements 

  • 11th, 12th, or 13th generation Embedded Intel® Core™ processors
  • 12th generation Intel® Core™ Desktop processors with Intel® Arc™ A380 Graphics
  • Intel Atom® Processor X Series, (formerly Alder Lake-N)
  • Intel® Processor N-series (formerly Alder Lake-N) 
  • Operating System: 
    • Ubuntu* 22.04.2 or higher
  • At least 80GB of disk space 
  • At least 8GB of memory 
  • Direct Internet access 

Ensure you have sudo access to the system and a stable Internet connection.

How It Works

The AI Box Test Module in the Intel® Video AI Box interacts with the Intel® ESDQ CLI through a common test module interface (TMI) layer, which is part of the Intel® ESDQ binary. 

The selected components from the download options will be validated via the automated test suite. 

Test results will be stored in the output folder. Intel® ESDQ generates a complete test report in HTML format and detailed logs packaged as one zip file, which you can email to the Edge Software Hub (ESH) support team. 

AI Box Test Module 

The AI Box test module is the validation framework for Intel® Video AI Box. This module validates the installation of software packages and measures the performance of the platform using the following benchmarks:

OpenVINO Benchmark 

The following neural network models are benchmarked using the OpenVINO™ benchmark tool. Both latency and throughput are measured. Benchmark results are included in the ESDQ report.  

  • resent-50-tf  
  • ssdlite-mobilenet-v2  
  • yolo-v3-tiny  
  • yolo-v3  
  • yolo-v4  
  • efficientnet-b0  
  • yolo-v5 n/s/m 
  • yolo-v8 n/s/m 

Video Analytic Benchmark

The video analytic benchmark is based on a six-stage pipeline that implements a vehicle detection and attribute classification use case. The pipeline consists of decode, object detection, object tracking, object classification, watermark, and encode stages. The models used are downloaded from OpenVINO™ Model Zoo. You can run up to 25 streams of videos simultaneously. 

Figure 1. Video Analytic Benchmark Pipeline

Figure 2. Vehicle Detection and Attribute Classification in Video Analytic Benchmark

Video Decode and Composite Benchmark 

Using the platform's GPU, the video decode and composite benchmark measures the average frame rate to decode multiple 1080p video streams simultaneously and tile them into a multi-view layout. The benchmark covers both H.264 and H.265 encoded video streams. The composited stream is re-encoded with the corresponding codec and saved as output for visual inspection. For integrated GPU, 16 video streams are decoded; 36 video streams are used for discrete GPU.

Figure 3. Video Decode and Composite Benchmark 

Memory Benchmark

The memory benchmark measures the sustained memory bandwidth based on STREAM.

GPU AI Frequency Measurement

The GPU inference frequency benchmark was designed to stress the GPU for an extended period. The benchmark records the GPU frequency while it runs an inference workload using the OpenVINO™ Benchmark Tool. 

Figure 4. GPU Inference Frequency Chart

Get Started 

The AI Box Test Module and the Intel® ESDQ CLI tool are installed as part of the Intel® Video AI Box development package. As a result, many of the steps are common with the instructions given in the Get Started section of Intel® Video AI Box.

NOTE: The screenshots may show a package version number different from the current release. See the Release Notes for information on the current release.

Download and Install Intel® ESDQ for Intel® Video AI Box

  1. Select Configure & Download to the download Intel® Video AI Box package. 
    Configure & Download
  2. Choose the required configuration based on your test system environment.  The following figure shows the recommended selection.

Figure 5. Configure and Download Page

 

  1. Press Download. In the next screen, accept the license agreement and copy the Product Key.

Figure 6. Product Key in Download Page

 

  1. Transfer the downloaded package to the target Ubuntu* system and unzip:       
unzip intel_video_ai_box.zip
  1. Go to the intel_video_ai_box/ directory:
cd intel_video_ai_box
  1. Change permission of the executable edgesoftware file:
chmod 755 edgesoftware
  1.  Install the Intel® Video AI Box package:
./edgesoftware install
  1. When prompted, enter the Product Key. You can enter the Product Key mentioned in the email from Intel confirming your download (or the Product Key you copied in Step 3). 

Note for People’s Republic of China (PRC) Network:

  • If you are connecting from the PRC network, the following prompt will appear during the bmra base installation: 

Figure 7. Prompt to Enable PRC Network Mirror 

 

  • Type Yes, and the installation script will replace certain download sources. This will avoid download failure in the PRC network.   
  • Then, while installing the AI Box Test Module, the following prompt will appear:  

 Figure 8. Prompt to Access Proxy Server 

 

  • Type Yes to replace certain GitHub* links.
  1. when prompted for the BECOME password, enter your Linux* account password during installation.

Figure 9. Prompt for BECOME Password  

 

  1. When prompted to reboot the machine, press Enter. Ensure to save your work before rebooting.

Figure 10. Prompt to Reboot

 

  1. After rebooting, resume the installation:
cd intel_video_aibox
./edgesoftware install
  1. After the ESDQ is installed, you will be prompted for the password. You can enter the password to proceed.

 Figure 11. Prompt for Password

 

  1.  When the installation is complete, you will see the message “Installation of package complete” and the installation status for each module.

Figure 12. Installation Complete Message 

 

  1.  Reboot the system:
sudo reboot

Run the Application

The installed Intel® Video AI Box package was configured for the Platform Qualification environment, and the benchmarks were run using this environment.

For the complete Intel® ESDQ CLI, refer to Intel® ESDQ CLI Overview. To find the available Intel® Video AI Box tests, run the following command:

cd intel_video_aibox/Intel_Video_AI_Box_3.0
esdq --verbose module run aibox --arg "-h"


You have the option to run an individual test or all tests together. The results from each test will be collated in the HTML report.

Run Full ESDQ

Run the following commands to execute all the Intel® Video AI Box tests and generate the full report.

cd intel_video_aibox/Intel_Video_AI_BOX_3.0
esdq --verbose module run aibox --arg "-r all"


Run OpenVINO Benchmark

The OpenVINO benchmark measures the performance of commonly used neural network models on the platform. The following models are supported:

  • resent-50
  • ssdlite-mobilenet-v2
  • yolo-v3-tiny
  • yolo-v3
  • yolo-v4
  • efficientnet-b0
  • yolo-v5 n/s/m
  • yolo-v8 n/s/m

The following OVRunner runner commands benchmark all the models using dGPU for 180 seconds.

cd intel_video_aibox/Intel_Video_AI_BOX_3.0
esdq --verbose module run aibox --arg "-r OVRunner -d dGPU -t 180"

OVRunner parameters:

  • -d {CPU, GPU, iGPU, dGPU} specifies the device type. 
  • -t <seconds> specifies the benchmark duration in seconds.
  • -m {resnet-50-tf, ssdlite_mobilenet_v2, yolo-v3-tiny-tf, yolo-v4-tf, efficientnet-b0, yolo-v3-tf, yolo-v5n, yolo-v5s, yolo-v5m, yolo-v8n, yolo-v8s, yolo-v8m} specifies model. Do not include this parameter to run all models.

The following is the example report: 

Figure 13. Sample OpenVino Benchmark Results

Run Video Analytic Benchmark 

The video analytic benchmark is based on a six-stage pipeline that implements a vehicle detection and attribute classification use case using Intel® DLStreamer.

Run the following DLRunner command:

cd intel_video_aibox/Intel_Video_AI_BOX_3.0
esdq --verbose module run aibox --arg "-r DLRunner -d GPU [-s 4/8/16/25]"

DLRunner Parameters:

  • -d {CPU, GPU, iGPU, dGPU} specifies device type.

  • -s {4,8,16,25} specifies number streams. Do not specify this parameter to run all.

The following is the example report: 

Figure 14. Sample Video Analytic Benchmark Results

Run Memory Benchmark

The memory benchmark measures the sustained memory bandwidth based on STREAM. To measure memory bandwidth for media processing, invoke the MemBenchmark runner:

cd intel_video_aibox/Intel_Video_AI_BOX_3.0
esdq --verbose module run aibox --arg "-r MemBenchmark"


After the MemBenchmark runner completes, the location of the test report is displayed. The following is the example report:

Figure 15. Sample Memory Benchmark Results

Run GPU AI Frequency Measurement

The GPU inference frequency benchmark was designed to stress the GPU for an extended period. The benchmark records the GPU frequency while it runs an inference workload using the OpenVINO™ Benchmark Tool.

The following FreqRunner runner command measures GPU frequency for 10 seconds.


cd intel_video_aibox/Intel_Video_AI_BOX_3.0
esdq --verbose module run aibox --arg "-r FreqRunner -d GPU -t 10"

FreqRunner parameters:

  • -d {CPU, GPU, iGPU, dGPU} specifies the device type.
  • -t <seconds> specifies the benchmark duration in seconds.

The following is an example plot:  

Figure 16. Sample Plot for GPU AI Frequency Benchmark Results


Run Video Decode and Composite Benchmark

The video decode and composite benchmark measure the average frame rate to decode multiple 1080p video streams simultaneously and tile them into a multi-view layout using the platform’s integrated GPU.

The following VMRunner runner command invokes this benchmark for H.264 decode.

cd intel_video_aibox/Intel_Video_AI_BOX_3.0
esdq --verbose module run aibox --arg "-r VWRunner -d GPU -c h264" 

VMRunner Parameters:

  • -d {GPU, iGPU, dGPU} specifies device type.
  • -c {h264,h265} specifies codec type. Do not specify this parameter to run the benchmark for h264 and h265.

The following is the example report: 

Figure 17. Sample Video Decode Benchmark Results

Sample Full ESDQ Report

Figure 17. Sample ESDQ Report

Release Notes 

Current Version: 3.0

New in This Release   

  • Updated ESDQ infrastructure to version v11.0.0
  • Built Benchmarks and executed in Video Analytics Base Library containers.
  • Added support for discrete GPU.
  • Added memory benchmark.

Known Issues

[AIBox-208]: Values in the test execution summary are not correct for the AI Box project.

[AIBOX-254]: VDBoxes are not 100% utilized in video decode and composite benchmarks.

Version: 2.6

New in This Release   

  • Updated the test module to support Intel® Distribution of OpenVINO™ toolkit 2022.2 Samples. 
  • Added tests for Intel® Distribution of OpenVINO™ toolkit in a container. 
  • Added video decode 16 channel 1080p 4x4 video wall tests.
  • Included GPU Inference Frequency Plot test.

Version: 2.5

New in This Release   

  • Updated the test module to support Intel® Distribution of OpenVINO™ toolkit 2022.1 Samples. 
  • Removed SVET, Intel® Media SDK and Yolo tests.
  • Included the functionality test of Pipeline and Inference Benchmark requirements. 

Version: 2.0

New in This Release   

  • Integrated Intel® Distribution of OpenVINO™ toolkit 2021.4.2. 
  • Included the functionality test of Azure IoT PnP Bridge, Intel® Distribution of OpenVINO™ toolkit TensorFlow Bridge, and Amazon Web Services Greengrass.

Known Issues

  • Smart Video and AI Workload Reference Implementation test requires the user to feed in the root password. However, when Intel® ESDQ is run on a graphical interface, the system switches to console mode during SVET execution, and the control to Intel® ESDQ terminal that displays the password prompt is lost. This results in a hang situation. Log in to the system using the SSH terminal and execute the Intel® ESDQ tests to work around this issue.

Version: 1.2

New in This Release  

  • Latency and Throughput information included in the HTML report. 
  • Included the functionality test of Smart Video and AI Workload Reference Implementation.

Version 1.0

New in This Release  

  • Initial features for recommended configuration. 

Known Issues 

  • If yolo-v3-tf.xml is not downloaded, then Latency and Throughput metrics will be null in the Intel® ESDQ HTML report page.

Troubleshooting

Please refer to the Troubleshooting section of the Intel® Video AI Box package.

Support Forum

If you're unable to resolve your issues, contact the Support Forum. 

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.