The OpenVINO™ Toolkit and the Intel® Media SDK

Introduction

The OpenVINO™ toolkit is composed of a variety of Intel tools that work together to provide a complete computer vision pipeline solution that is optimized on Intel® architecture. This article will focus on the Intel® Media SDK component of the toolkit. The Intel Media SDK is a high level API for specific video processing operations: decode, process, and encode. It supports H.265, H.264, MPEG-2, and more codecs. Video processing can be used to resize, scale, de-interlace, color conversion, de-noise, sharpen, and more. The Intel Media SDK works in the background to leverage hardware acceleration on Intel® architecture with optimized software fallback for each individual hardware platform. Hence, developers do not need to change the code from platform to platform, and can focus more on the application itself rather than hardware optimization.

To learn more about OpenVINO toolkit:

OpenVINO™ Toolkit

To learn more about the Intel Media SDK:

Intel® Media SDK

Intel® Media SDK Set-Up

OpenVINO toolkit 2018 R1.2 can be downloaded for free here: OpenVINO™ Toolkit Choose & Download.

Instructions on how to install on Ubuntu* 16.04.3 are described here: Installing the OpenVINO™ Toolkit for Linux*.

OpenVINO toolkit will install the Intel Media SDK as well in /opt/intel/mediasdk/.

To verify the install:

Check the environment variables are correct. MFX_HOME, LIBVA_DRIVERS_PATH, and LIBVA_DRIVER_NAME should be set as seen below.

$ set | grep MFX_HOME
MFX_HOME=/opt/intel/mediasdk
$ set | grep LIBVA
LIBVA_DRIVERS_PATH=/opt/intel/mediasdk/lib64
LIBVA_DRIVER_NAME=iHD

Code snippet 1. Check the environment variables

Otherwise set the environment variables:

export MFX_HOME=/opt/intel/mediasdk
export LIBVA_DRIVERS_PATH=/opt/intel/mediasdk/lib64
export LIBVA_DRIVER_NAME=iHD

Code snippet 2. Set the environment variables if needed

Install  vainfo:

sudo apt-get install vainfo

Code snippet 3. install vainfo

Execute the command  vainfo in a command prompt to ensure it can find the iHD_drv_video.so and there are no other errors.

$ vainfo
error: can't connect to X server!
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.1 (libva 2.0.1.pre1)
vainfo: Driver version: Intel iHD driver - 16.9.git_f6283e_4-11-2018
vainfo: Supported profile and entrypoints
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointStats
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointFEI
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointFEI
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: VAEntrypointFEI
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointFEI

Code snippet 4. vainfo command output

If there are errors on vainfo, you will need to download the below script and run it as a normal user. For example, you might see "libva error: /opt/intel/mediasdk/lib64/iHD_drv_video.so has no function __vaDriverInit_0_32" in this case, the below script will help fix the media driver install.

wget http://software.intel.com/sites/default/files/managed/bb/ff/build_msdk_ubuntu.tar.gz
tar –xvf build_msdk_ubuntu.tar.gz
cd build_msdk_ubuntu
./build_msdk_ubuntu.sh

Code snippet 5. Install Media SDK dependencies

The script will prompt for three things: enter the sudo password when asked, hit enter when asked to input the working directory so it uses the defaults, and input "Yes" to check out the tested commit.

It is recommended to install a YUV capable video player to view the videos, e.g. vooya* and install with sudo dpkg –i <deb file>.

Build the Samples

Media SDK comes with a number of sample programs for decode, process, encode, and others. To run them, the samples need to be built first:

cd /opt/intel/mediasdk/samples/samples
sudo perl build.pl --cmake=intel64.make.debug --mfx-home=/opt/intel/mediasdk --clean
sudo make -j8 -C __cmake/intel64.make.debug

Code snippet 6. Build the Media SDK samples

The compiled samples can be found here:

cd/opt/intel/mediasdk/samples/samples/__cmake/intel64.make.debug/__bin/debug

Code snippet 7. location of the compiled samples

To use the samples, move them out of the /opt/ directory, somewhere on the Desktop for example. That way it doesn't require sudo to run, and sudo user does not have the proper environment variables set up.

Video Input

In this paper we will focus on H.264 format as most webcams output video in H.264 format. However the video for this article will start in mp4 format as that is more commonly used and available on the internet. Hence we need to transcode the video from mp4 to H.264. Below will outline two ways to accomplish this, one accelerated version and one not.

Cars driving on the road mp4 video.

Intel® Quick Sync Video Transcode Acceleration

An Intel® Processor with integrated graphics will allow you to take advantage of Intel® Quick Sync Video. Intel Quick Sync Video provides a dedicated media processing capabilities inside of Intel® Processor Graphics technology. To check the processor capabilities visit: Product Specifications

Copy the SDK header files to include/mfx:

mkdir /opt/intel/mediasdk/include/mfx
sudo cp /opt/intel/mediasdk/include/*.h /opt/intel/mediasdk/include/mfx

Code snippet 8. Copy the Media SDK header files

Install the dependencies:

sudo apt-get update -qq && sudo apt-get -y install \
  autoconf \
  automake \
  build-essential \
  cmake \
  git-core \
  libass-dev \
  libfreetype6-dev \
  libsdl2-dev \
  libtool \
  libva-dev \
  libvdpau-dev \
  libvorbis-dev \
  libxcb1-dev \
  libxcb-shm0-dev \
  libxcb-xfixes0-dev \
  pkg-config \
  texinfo \
  wget \
  zlib1g-dev

Code snippet 9. Install the dependencies

Create the "libmfx.pc" file, inside a folder on the Desktop is fine.

prefix=/opt/intel/mediasdk
exec_prefix=/opt/intel/mediasdk/
libdir=${prefix}/lib/lin_x64/
includedir=${prefix}/include/
Name: libmfx
Description: mfx
Version: 1.0.0
Cflags: -I${includedir} 
Libs: -L${libdir} -lmfx -ldl -lstdc++ -lrt -lva -lva-drm
Libs.private: -lstdc++ -ldl

Code snippet 10. libmfx.pc file

Updated the PKG_CONFIG_PATH environment variable with the path to the libmfx.pc file:

export PKG_CONFIG_PATH= "location of libmfx.pc file"

Code snippet 11. export PKG_CONFIG_PATH environment variable

Get the FFMPEG source code and build it:

git clone https://github.com/Intel-FFmpeg-Plugin/Intel_FFmpeg_plugins.git
cd Intel_FFmpeg_plugins/
./configure --enable-libmfx
make -j4
sudo make install

Code snippet 12. build ffmpeg

Check that the codecs are installed properly:

ffmpeg -codecs | grep 'qsv'

Code snippet 13. check Intel® Quick Sync Video codecs installed

Figure 1. shows Intel® Quick Sync Video codecs installed

Convert the video using the Intel® Quick Sync Video accelerator. You can ignore the info message about "A decode call did not consume any data: expect more data at input", that just indicates it has reached the end of the file.

ffmpeg -hwaccel qsv -c:v h264_qsv -i car-detection.mp4 -b:v 2M -preset veryfast -c:v h264_qsv out_car_detection.h264

Code snippet 14. convert mp4 video to h264 format with Intel® Quick Sync Video acceleration

Transcode Without Intel® Quick Sync Video

To transcode the video without Intel® Quick Sync Video, install "ffmpeg" and the "libx264-dev" codec.

sudo apt-get install ffmpeg
sudo apt-get install libx264-dev
ffmpeg -i car-detection.mp4 -an -vcodec libx264 -crf 23 out_car-detection.h264

Code snippet 15. convert mp4 video to h264 format without Intel Quick Sync Video acceleration

Functions

We will focus on the three main operations the SDK optimizes and comes with sample code for: Decode, VPP (Video Processing Pipeline), and Encode.

Figure 2. Intel® Media SDK pipeline

Decode

The decoding sample supports the following input video formats: H.264 (AVC, MVC – Multi-View Coding), MPEG-2 video, VC-1, JPEG/Motion JPEG, HEVC, VP8). It will render a YUV 4:2:0 video as output. Decode can also resize the video at the same time, just change the width and height parameters. Do not leave the width and height parameters out of the command, even if not resizing.

Example decode command:

./sample_decode h264 -nv12 -i out_car-detection.h264 -w 768 -h 448 -o car_detection_decode_768x448.nv12

VPP (Video Processing Pipeline)

To do some video processing after decoding, the VPP sample supports to following features:

VPP functions:

• N:1 Frame Composition
• Resizing
• Color Conversion
• Deinterlacing
• Denoising 
• Frame Rate Conversion
• Brightness/Contrast/Saturation
• Sharpening

Example resizing the video:

./sample_vpp -sw 768-sh 448 -scc nv12 -dw 400 -dh 200 -istab 1 -dcc nv12 -i car_detection_decode_768x448.nv12 -o car_detection_VPP_400x200.nv12

Code snippet 17. vpp command

The istab option will scale the image to fit the new size, otherwise it will crop it to the new size by default.

Encode

Encode sample will encode the video back to its original format or to another format that is supported by decode as input.

./sample_encode h264 -i car_detection_VPP_400x200.nv12 –nv12 -o car_detection_Encode_400x200.h264 -w 400 -h 200

Code snippet 18. encode command

Summary

This article has been an introduction to the Intel Media SDK and how to leverage it to accelerate the video workloads for decoding, processing, and encoding a video. It can be great addition to an existing computer vision pipeline to process videos. While inference is normally the largest percentage of the processing time, that time is reduced using the Intel® Deep Learning Deployment Toolkit model optimizer. Hence the time to process the video becomes a second optimization that is worth exploring to fully optimize the computer vision pipeline.

OpenVINO™ Toolkit

About the author

Whitney Foster is a software engineer at Intel in the Core and Visual Computing Group working on scale enabling projects for Internet of Things, Intel® vPro™ technology, and OpenVINO toolkit.