Intel® Distribution of OpenVINO™ Toolkit Long-Term Support (LTS)

Introduction

The Intel® Distribution of OpenVINO™ toolkit is a comprehensive toolkit for quickly developing applications and solutions that solve a variety of tasks including emulation of human vision, automatic speech recognition, natural language processing, recommendation systems, and many others. Based on latest generations of artificial neural networks, including Convolutional Neural Networks (CNNs), recurrent and attention-based networks, the toolkit extends computer vision and non-vision workloads across Intel® hardware, maximizing performance. It accelerates applications with high-performance, AI and deep learning inference deployed from edge to cloud.

The Intel® Distribution of OpenVINO™ toolkit:

• Enables deep learning inference from the edge to cloud.
• Supports heterogeneous execution across Intel accelerators, using a common API for the Intel® CPU, Intel® GPU, Intel® Gaussian & Neural Accelerator, Intel® Neural Compute Stick 2, Intel® Vision Accelerator Design with Intel® Movidius™ VPUs.
• Speeds time-to-market through an easy-to-use library of CV functions and pre-optimized kernels.
• Includes optimized calls for CV standards, including OpenCV* and OpenCL™.

New and Changed in 2021.4.2 LTS

Major Features and Improvements

This 2021.4.2 LTS release provides functional bug fixes, and minor capability changes for the previous 2021.4.1 Long-Term Support (LTS) release, enabling developers to deploy applications powered by Intel® Distribution of OpenVINO™ toolkit with confidence. To learn more about long-term support and maintenance, go to the Long-Term Support Policy.

Note: A new LTS version is released every year  and is supported for 2 years (1 year of bug fixes, and 2 years for security patches). The LTS version is intended for developers taking OpenVINO™ toolkit to production. For developers that prefer the very latest features and leading performance, standard releases are recommended. Standard releases will continue to be made available three to four times a year.  

• Learn more about what components are included in the LTS release in the Included in this release section

• Support for the 12th generation Intel® Core™ processor family that is built on the Intel 7 process with new performance hybrid architecture that delivers enhancements in multithreaded performance to handle compute-intensive workloads. Learn More

• Specific fixes, capability improvements, and support updates to known issues with:

  • Model Optimizer, specifically issues causing accuracy regression
  • Inference Engine (plug-ins for Inference Engine Python* API, C API, GPU, Intel® Movidius™ Myriad™ VPU, HDDL, and Intel® Gaussian & Neural Accelerator)
    • Added support for the 12th generation Intel® Core™ processor family that enables Intel® Gaussian & Neural Accelerator (Intel® GNA) 3.0 and Intel GNA generation with native 2D convolutions
    • Functional performance improvements to testing and accuracy, fixes to bugs that caused performance degradation for several models, fixed heap-use-after-free, and memory leaks
  • Minor capability changes and bug fixes to the Open Model Zoo, specifically issues that affected the Accuracy Checker in the Deep Learning Workbench
• Additional Jupyter* Notebook tutorials:
  • Use your laptop and webcam to run demonstrations for object detection and human pose estimation
  • Apply 8-bit quantization with Neural Network Compression Framework (NNCF) to optimize your Keras and TensorFlow or PyTorch models
  • Learn how to show live inference, and optimize and quantize a segmentation model
  • See all tutorials

Support Change and Deprecation Notices

• TunableQuantization Algorithm in Post-training Optimization Tool deprecation notice:

  Deprecation Begins	November 16, 2021
  Removal Date	Q1’22

  • 2021.4 LTS is the final release where TunableQuantization algorithm is supported in Post-training Optimization Tool

Model Optimizer

• Fixed issues:
  • Fixed issue when "data" attributes were empty in case of model conversion with extension layers
  • Fixed confusing reminder to check for the latest software while using the latest officially released OpenVINO package
  • Fixed FusedBatchNorm to MVN transformation which was causing accuracy regression

Inference Engine

• C API:
  • Introduced C API versions of model import and export functionality.
• GPU plugin:

  • Improved performance counters for custom and user's GPU layers

  • Support of models in Windows locales with various decimal delimiters
• MYRIAD plugin:

• Firmware
  • Implemented threshold for SoftPlus operation in alignment with specs
    For numerical stability, the operation reverts to the linear function when x > threshold where threshold depends on T and is chosen in such a way that the difference between the linear function and exact calculation is no more than 1e-6t. If T is fp16, the threshold should be 12
  • Improved tests for Proposal operation to calculate results accurately
  • Checks a return code for all SHAVE DMA indirection logic calls in MvTensor component have been added
  • Added support for only 2nd axis for MVN operation

• XLink
  • Added XLINK_THREADS_PRIORITY, XLINK_MAX_STREAM_RES compile flags to configure XLink for custom cases
  • Added protection to avoid issues related to concurrent thread access to shared event queues. It caused some expected performance degradations for several models up to 12%.
  • XLink APIs XLinkReadDataWithTimeout, XlinkWriteDataWithTimeout moved from deprecated state to supported
  • Added new XLink API: XlinkReleaseSpecificData.
  • XLinkOpenStream now allocates streamId only on the host side. The device, in turn, opens streams with forcedstreamId passed by the host.
  • MVNC API
    • Fixed heap-use-after-free and memory leaks vulnerability when a device unexpected disconnection happen 
  • Plugin
    • Fixed PERF_COUNTER metric calculation. When serializing execGraph, milliseconds were actually written to the execTimeMcs field. Now it is calculated in microseconds.
    • Removed deprecated MYRIAD_IR_WITH_SCALES_DIRECTORY config
    • Implemented MYRIAD_SCALES_PATTERN config that allows user to pass regular expression contains a rule to set custom scales for weighable operations (Convolution, Deconvolution, Fully connected). It may be helpful if general scale calculation logic does not work for some models.

• HDDL plugin:

• Firmware
  • Implemented threshold for SoftPlus operation in alignment with specs
    For numerical stability, the operation reverts to the linear function when x > threshold where threshold depends on T and is chosen in such a way that the difference between the linear function and exact calculation is no more than 1e-6t. If T is fp16, the threshold should be 12
  • Improved tests for Proposal operation to calculate results accurately
  • Checks a return code for all SHAVE DMA indirection logic calls in MvTensor component have been added
  • Added support for only 2nd axis for MVN operation

• XLink
  • Added XLINK_THREADS_PRIORITY, XLINK_MAX_STREAM_RES compile flags to configure XLink for custom cases
  • Added protection to avoid issues related to concurrent thread access to shared event queues. It caused some expected performance degradations for several models up to 12%.
  • XLink APIs XLinkReadDataWithTimeout, XlinkWriteDataWithTimeout moved from deprecated state to supported
  • Added new XLink API: XlinkReleaseSpecificData.
  • XLinkOpenStream now allocates streamId only on the host side. The device, in turn, opens streams with forcedstreamId passed by the host.
  • Plugin
    • Removed deprecated MYRIAD_IR_WITH_SCALES_DIRECTORY config
    • Implemented MYRIAD_SCALES_PATTERN config that allows user to pass regular expression contains a rule to set custom scales for weighable operations (Convolution, Deconvolution, Fully connected). It may be helpful if general scale calculation logic does not work for some models.

• GNA plugin:
  • Added support for GNA3.0, new GNA generation with native 2D convolution support. which comes with 12th Gen Intel Core Processors (formerly Alder Lake) 
  • Improved accuracy for models after the Post-Training Optimization Tool.

  • Fixed a bug with state handling in networks with some specific configurations.

Post-Training Optimization Tool (POT)

• Deprecated algorithms:
  • TunableQuantization Algorithm

Open Model Zoo

• Issue 2764 with downloading some public models from google drive was fixed.
• Several issues affecting DL Workbench fixed in Accuracy Checker.
• Minor fixes in model's documentation. 

Deep Learning Streamer

• Changes
  • Support for 12th Gen Intel Core Processors (Alder Lake S platform)
• Fixed issues
  • [ADL] GPU Watermark doesn't work on ADL-S platform
  • [TGL] Missing frames in inference results when using batching with zero-copy
  • [iGPU] Multi-stream pipeline hangs in batch-size and vaapi-sharing case
  • [iGPU] Script to install DPCPP dependencies fails on ubuntu20_data_dev:2021.4.1 docker image

New Distributions

•   Ubuntu-based Docker images are distributed with pre-installed Python without included readline module. If you need this module, use one of the following ways:

• Reinstall Python standard library apt package:

  • For Ubuntu 18: apt update && apt install --reinstall libpython3.6-stdlib
  • For Ubuntu 20: apt update && apt install --reinstall libpython3.8-stdlib
•  Install PyPI gnureadline package: pip3 install gnureadline

New and Changed in 2021.4.1 LTS

Major Features and Improvements

This 2021.4.1 LTS release provides functional bug fixes, and minor capability changes for the previous 2021.4 Long-Term Support (LTS) release, enabling developers to deploy applications powered by Intel® Distribution of OpenVINO™ toolkit with confidence. To learn more about long-term support and maintenance, go to the Long-Term Support Policy.

Note: A new LTS version is released every year  and is supported for 2 years (1 year of bug fixes, and 2 years for security patches). The LTS version is intended for developers taking OpenVINO™ toolkit to production. For developers that prefer the very latest features and leading performance, standard releases are recommended. Standard releases will continue to be made available three to four times a year.   

• Learn more about what components are included in the LTS release in the Included in this release section. Specific fixes to the known issues:

  • Model Optimizer, Inference Engine (Inference Engine Python API, CPU, GPU, MYRIAD, HDDL, and GNA plugin), Deep Learning Streamer, and Post-Training Optimization Tool (POT)

  • Minor capability changes and bug fixes to the Open Model Zoo

• New Jupyter Notebook tutorials that simplify getting started with OpenVINO™ toolkit:

  • Optical Character Recognition (OCR) tutorial - detects text and then performs scene text recognition
  • HuggingFace BERT Quantization tutorial – demonstrates post-training INT8 quantization pipeline for NLP models
  • Hello Detection – demonstrates basic object detection with OpenVINO using a text detection model
  • Hello Segmentation – demonstrates basic semantic segmentation with a road segmentation model
  • New PaddlePaddle-to-OpenVINO tutorials: Image Classification and PaddleGAN Super Resolution
• Download the 2021.4.1 LTS release of the Intel® Distribution of OpenVINO™ toolkit to upgrade to the latest LTS release.

Model Optimizer

• Fixed issues:
  • Degradation of the Retinanet model in INT8 precision has been fixed
  • Split and Concat patterns were added after input to text-detection-00004 model
  • Fixed error in Summarize_graph.py with the message "too many values to unpack (expected 2)"

Inference Engine

• Inference Engine Python API:
  • Fixed issues:
    • Fixed issue with long GIL, detected when loading network
• CPU plugin:
  • Fixed issues:
    • Fixed Performance drops due to SetBlob calls in OV 2021.4
    • Fixed issue when Models Aclnet, aclnet-int8, mobilenetv2-int8-onnx-0001  fail with error 'Supported primitive descriptors list is empty for node:....'
    • Memory leak in oneDNN GEMM (and potentially other operations) have been fixed
    • Access Violation on Windows 10 with 2021R1 has been fixed
    • Fixed issue where ROIPooling operation with bilinear mode calculate border elements incorrectly

• GPU plugin:
  • GPU models loading optimization by the preference of big cores utilization 
  • Fixed issues:
    • Fixed TF resnet 50 v2 accuracy on FP16
• MYRIAD plugin:

  • Errata fix for hardware operations has been reverted. That fix had contained implementation of the data relocation which had caused unnecessary relocations and issues in multi-threaded scenarios. Also, it led to a drop in the accuracy of the yolo-family models and some other models (image-super-resolution-1033, image-super-resolution-1032, unet-camvid-onnx-0001).

  • GPT2 model functionality has been recovered. Now, this model works on MyriadX/HDDL through ONNX RT (UEP).
  • 2 stage product Recall model has been fitted for MyriadX (ma2480) memory capacity (512 MB DDR). A pass that merges some Gather operations has been recovered after opset7 enabling.
• HDDL plugin:
  • Errata fix for hardware operations has been reverted. That fix had contained implementation of the data relocation which had caused unnecessary relocations and issues in multi-threaded scenarios. Also, it led to a drop in the accuracy of the yolo-family models and some other models (image-super-resolution-1033, image-super-resolution-1032, unet-camvid-onnx-0001).
  • GPT2 model functionality has been recovered. Now, this model works on HDDL through ONNX RT (UEP).
  • 2 stage product Recall model has been fitted for each HDDL core (ma2480) memory capacity (512 MB DDR). A pass that merges some Gather operations has been recovered after opset7 enabling.
• GNA plugin:
  • GNA library updated to v3 introducing new Quality of Service (QoS) mechanism which does not require manual status checking and mode setting.
  • Improved the graph compiler to relax layer limitations caused by GNA HW limitations (such as maximum buffer size and batch size).

  • Resolved compilation and accuracy fixes for networks after Post-Training Optimization Tool.

  • Fixed problems with graph compilation errors and hangs for specific models.

  • Improved memory usage for concatenations that are not 64-bytes aligned.
  • Fixed an issue with backward compatibility when importing models exported with previous versions of OpenVINO. 

  • Enabled support for manual scale factors setting for imported models.

Post-Training Optimization Tool (POT)

• Fixed issues:

  • Quantization of outputs for GNA plug-in
  • Insert FQ before the log layer during quantization for GNA plug-in
  • Fix misalignment with FP model in case of outputs quantization for GNA plug-in

Open Model Zoo

• Changes

  • CPU performance improved for FP32 QuartzNet model

  • segmentation_demo extend with an option to show segmentation mask only

  • Accuracy Checker extended with Analysis feature, used in DL WorkBench

  • C++ object_detection_demo extended with support Yolo V4 and Yolo V4 Tiny models
  • Accuracy of Intel noise-suppression-poconetlike-0001 improved (model weights updated)
• Fixed issues:

  • Objects detection demo with error segmentation fault when HDDL is selected based on OpenVINO™ toolkit version 2021.4 LTS 

  • Classification demo: sometimes suffers from zero division exception

  • Inaccurate result for security_barrier_camera_demo with license-plate-recognition-barrier-0007

  • Fixed single_human_pose_estimation_demo/python rendering detection issue

  • OpenVINO™ GitHub issue #2641 - Define armv7 instruction set for OMZ demos built natively on ARM

Deep Learning Streamer

• Changes
  • Added documentation on how to create pre/post-processing configuration files for custom models
  • Out-of-box support for more Open Model Zoo models (person-vehicle-bike-detection-2004 and others with similar output layer), public models EfficientNet and EfficientDet
• Fixed issues:
  • Audio Detection DL Streamers Sample fails on 2021.4
  • Preprocess should check roi valid when using YUV420 format
  • gvaclassify does not work with license-plate-recognition-barrier-0001 model
  • Caught SIGSEGV if use gvatrack before gvainference
  • Skipped classification results and false detection observed in gst_launch/vehicle_pedestrian_tracking sample

New Distributions

• You can download Intel® Distribution of OpenVINO™ toolkit runtime and development archival packages for Red Hat 8 on https://storage.openvinotoolkit.org/repositories/openvino/packages/2021.4.1/ site.
• Containers:
  • Now Windows 20H2 runtime Docker image is available on the DockerHub container registry with CPU plugin support.
    • Includes Inference Engine, OpenCV, Python bindings
    • Supports CPU
  • Now Red Hat UBI 8 development Docker image is available on Red Hat Quay.io and Red Hat Ecosystem Catalog container registries with CPU, GPU plugins support.
    • Includes Inference Engine and OpenCV, samples, demos, Python tools for development: Model Optimizer, Post-training Optimization tool, Accuracy checker, Open Model Zoo tools.
    • Supports CPU and GPU devices.

New and Changed in Release 2021.4 LTS

Major Features and Improvements

• This new 2021.4 Long-Term Support (LTS) Release provides bug fixes, longer-term maintenance and support with a focus on stability and compatibility enabling developers to deploy applications powered by Intel® Distribution of OpenVINO™ toolkit with confidence. A new LTS version is released every year and supported for two years. For those developers that prefer the very latest features and leading performance, standard releases will continue to be made available 3-4 times a year. Read more about the long-term support and maintenance, go to the Long Term Support Policy.
• New Jupyter Notebooks, demos and support for additional public models to make development easier:
  • Ready-to-run Jupyter Notebooks with tutorials for converting TensorFlow and PyTorch models, image classification, segmentation, depth estimation, post-training quantization and more.
  • Audio Noise Suppression & Time Series Forecasting demos
  • Public Models: RCAN and IseeBetter (image super-resolution), Attention OCR (image text prediction), Tacotron 2 (text-to-speech) and ModNet (portrait/image matting)

• Time-to-first-inference latency performance enhancements: Initialization has been optimized on CPU and integrated GPU (iGPU), significantly improving performance at inferencing startup. Setting up inferencing always involves additional initialization time as the network is loaded and configured on the device, especially on GPUs due to their architecture. This setup time has been reduced significantly for many networks by doing more initialization work in parallel among other optimizations.

• Preview of OpenVINO ™ integration with TensorFlow: Although not a part of the 2021.4 LTS release, a new open source component called the OpenVINO™ integration with TensorFlow is available as a public preview. This component is designed for TensorFlow developers newly exploring OpenVINO™ toolkit to try it with minimal code changes, maximizing TensorFlow API compatibility. For highest performance, lowest memory footprint and complete hardware control, adopting native OpenVINO APIs continues to be the recommended approach.

Support Change and Deprecation Notices

• The following deprecated Inference Engine APIs will be removed in 2022.1:

  • ExecutableNetwork::QueryState - use InferRequest::QueryState instead
  • IVariableState / IMemoryState interface - use VariableState wrapper instead
  • VariableState::GetLastState - use VariableState::GetState instead
  • Helpers functions working with UNICODE symbols: fileNameToString, stringToFileName
  • InferenceEngine::Parameter creation from ngraph::Variant, casting to ngraph::Variant shared pointer, InferenceEngine::Parameter::asVariant method.
  • ngraph::Node::get_output_tensor_name(), ngraph::Node::get_input_tensor_name(), ngraph::description::Tensor::get_name(), ngraph::description::Tensor::set_name() - Use ngraph::description::Tensor::get_names() instead.
  • ngraph::runtime::Tensor::get_name(), ngraph::runtime::Tensor::get_scale(), ngraph::runtime::Tensor::set_scale(), ngraph::runtime::Tensor::wait_for_read_ready(), ngraph::runtime::Tensor::wait_for_write_ready() - these methods will be removed without new analogues.
  • The following deprecated Inference Engine Python APIs will be removed:
    • IENetwork:
      • constructor for reading networks from a file/buffer.
      • property inputs - use input_info instead
    • InferRequest:
      • property inputs - use input_blobs instead
      • property outputs - use output_blobs instead
    • ExecutableNetwork:
      • property inputs - use input_info instead
• FPGA deprecation notice:

• Intel® is transitioning to the next-generation programmable deep learning solution, which will be called Intel® FPGA AI Suite and will support OpenVINO™ toolkit when productized.
• As part of this transition, 2020.3.2 LTS was the final release to include support for Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA. Now support offered in 2020.3.2 LTS for FPGA is coming to an end.

• Any customer inquiries regarding Intel® FPGA AI Suite should be directed to your Intel Programmable Solutions Group account manager or subscribe to get notified with the latest updates.

Model Optimizer

• Common changes:
  • Aligned requirements files with other OpenVINO tools to eliminate any conflicts
  • Fixed tensor names propagation to output nodes in Tensorflow*, Kaldi*, Caffe*, MxNet*, which means that CNNNetwork::getOVNameForTensor() API now works properly for all frameworks
  • Added support of FP16 models with shape subgraphs in FP32 precision. This change enables reshape-ability for model with shape sizes greater than 65000 (maximum of fp16)
  • Added support for operation Gelu-7 with additional attribute “approximation_mode”
  • Added --transform key to allow additional transformation execution inside Model Optimizer. Currently we have only LowLatency2 transformation.
• ONNX*:
  • Added support for the following operations:

    • Size – 1, 11

    • QuantizeLinear-13

    • DequantizeLinear-13

• TensorFlow*:
  • Added support for the following operations:
    • Roll

    • GatherV2 operation with nonzero ‘batch_dims’ attribute

    • Einsum-7 with equations without ellipsis and repeated labels (for diagonal extraction)

    • FFT
    • iFTT
  • Extended support for TensorFlow* 1 while to support a body with the current iteration variable used by the general computation in addition to slicing and concatenation
• MXNet*:
  • Added support for the following operations:
    • Roll
    • FFT
    • iFFT

    • Einsum-7 with equations without ellipsis and repeated labels (for diagonal extraction)

• Kaldi*:

  • Added support for TimeHeightConvolutionComponent

Inference Engine

• Model cache feature:
  • For devices supporting Import / Export API (GNA, MYRIAD), we have implemented automatic model cache feature. When enabled, loading of model does the following:
    • On very first application run, honest network compilation is performed, then compiled network is exported on hard disk.
    • On the second and other runs, loading does not perform actual network compilation and takes compiled model from models cache.
    • GPU: Enabling model caching feature enables caching of GPU/CLDNN kernels and this also significantly speeds up load network time for GPU.
  • Loading network from model file API is introduced to simplify coding on application side. This allows for faster performance when model cache feature is enabled, compared to explicitly reading model from IR and loading it from cache.
• Heads Up: Transition to Intel® oneAPI oneTBB is on the way.
  • Currently, the default threading model of the OpenVINO is Intel® Threading Building Blocks (Intel® TBB).  Recently Intel® TBB has been migrated to Intel® oneAPI oneTBB (https://github.com/oneapi-src/oneTBB/). While OpenVINO is still compiled with a pre-oneTBB version (also moved to https://github.com/oneapi-src/oneTBB/releases), a transition to oneTBB will be coming over the next year.
  • If your private and/or third-party components are using Intel® TBB directly, it is strongly advised to plan migration and test for compatibility with Intel® oneAPI oneTBB.
    • As discussed in the official oneTBB migration guide, this may include code changes, performance implications and so on: Migrating from oneTBB
• Common changes:
  • Added new metrics to QueryAPI:
    • DEVICE_TYPE 
    • DEVICE_GOPS
  • Added I4/U4 Precisions

• Deprecated API:

  • IInferRequest interface is deprecated, use InferRequest wrapper:
    • Constructor for InferRequest from IInferRequest:: Ptr is deprecated
    • Cast operator for InferRequest to IInferRequest shared pointer is deprecated
  • ICNNNetwork interface is deprecated by means of deprecation of all its methods, use CNNNetwork wrapper
  • CNNNetwork methods working with ICNNNetwork are deprecated:
    • Cast to ICNNNetwork shared pointer
    • Cast to reference to ICNNNetwork interface
    • Constructor from ICNNNetwork shared pointer
  • IExecutableNetwork is deprecated, use ExecutableNetwork wrappers:
    • Constructor of ExecutableNetwork from IExecutableNetwork shared pointer is deprecated
  • The following ExecutableNetwork methods are deprecated:
    • ExecutableNetwork::reset
    • Cast operator to IExecutableNetwork shared pointer
    • ExecutableNetwork::CreateInferRequestPtr - use ExecutableNetwork::CreateInferRequest instead
  • Version::ApiVersion structure is deprecated, Inference Engine does not have API version anymore
  • LowLatency - use lowLatency2 instead
  • CONFIG_KEY(DUMP_EXEC_GRAPH_AS_DOT) - use InferenceEngine::ExecutableNetwork::GetExecGraphInfo::serialize() instead
  • Core::ImportNetwork with no device - pass device name explicitly.
  • details::InferenceEngineException - use InferenceEngine::Exception and its derivatives instead.
  • InferenceEngine::make_so_pointer which is used to create Extensions library is replaced by std::make_shared<Extension>(..)
  • InferenceEngine::IExtension::Release is deprecated with no replacement
  • Use IE_DEFINE_EXTENSION_CREATE_FUNCTION helper macro instead of explicit declaration of CreateExtension function, which create extension.
  • GPU plugin configuration options:
    • KEY_CLDNN_NV12_TWO_INPUTS GPU plugin option. Use KEY_GPU_NV12_TWO_INPUTS instead
    • KEY_CLDNN_PLUGIN_PRIORITY GPU plugin option. Use KEY_GPU_PLUGIN_PRIORITY instead
    • KEY_CLDNN_PLUGIN_THROTTLE GPU plugin option. Use KEY_GPU_PLUGIN_THROTTLE instead
    • KEY_CLDNN_MEM_POOL GPU plugin option
    • KEY_CLDNN_GRAPH_DUMPS_DIR GPU plugin option
    • KEY_CLDNN_SOURCES_DUMPS_DIR GPU plugin option
    • KEY_DUMP_KERNELS GPU plugin option
    • KEY_TUNING_MODE GPU plugin option
    • KEY_TUNING_FILE GPU plugin option
• Inference Engine Python API:
  • IECore
    • def load_network(network: str, str device_name, config=None, int num_requests=1) - reads and loads network from model file to device with one line of code. When model cache feature is enabled, it is performed faster than read and load due to save time on unnecessary reading network
  • Added VariableState API:
    • InferRequest:
      • def query_state ( ) - gets state control interface for given infer request. Returns a vector of VariableState objects.
    • VariableState:
      • def reset() - reset internal variable state for relevant infer request to a value specified as default for according ReadValue node. 
      • property state - get/set the value of the variable state.
      • property name - a string representing a state name.
• CPU plugin:
  • The plugin was migrated on ngraph::function as an input graph representation instead of legacy one. This allowed getting rid of legacy API usage inside the plugin and opened up new opportunities for further improvements of first inference latency and memory consumption characteristics
  • Introduced automatic fallback on ngraph reference implementations (evaluate() method) for operation which are not directly supported by the plugin. The feature might serve for simplifying OV extensibility mechanism for new operations
  • Added support for new operations:
    • DFT-7
    • iDFT-7
    • Roll-7
    • Gather-7
    • GELU-7
  • Implemented several improvements for int8 inference pipeline including:
    • Int8 ConvolutionBackpropData (Deconvolution) layer support
    • Performance optimizations for mixed precision (FP32 + INT8) models by decreasing the number of layout permutations
  • Significantly improved performance of BF16 inference pipeline for LSTM based models
  • Added performance optimizations for operations: ShuffleChannels, DepthToSpace/SpaceToDepth, ExtractImagePatches, Mish, Concat, Transpose
  • Provided performance improvements for first inference latency on client platforms
  • Decreased memory consumption in latency scenario on client platforms
  • Please see a note on the Intel® oneAPI oneTBB in the Inference Engine section.
• GPU plugin:
• • Most of the GPU plugin options were moved from cldnn/cldnn_config.hpp header to gpu/gpu_config.hpp
  • Added new GPU specific metrics:
    • DEVICE_TOTAL_MEM_SIZE
    • UARCH_VERSION
    • EXECUTION_UNITS_COUNT
  • Added new GPU plugin config options:
    • KEY_GPU_PLUGIN_PRIORITY
    • KEY_GPU_PLUGIN_THROTTLE
    • KEY_GPU_NV12_TWO_INPUTS
    • KEY_GPU_MAX_NUM_THREADS
    • KEY_GPU_ENABLE_LOOP_UNROLLING
  • Added new possible value for OPTIMIZATION_CAPABILITIES metric: HW_MATMUL
  • Enabled parallel compilation of OCL kernels which improves first ever inference latency. Number of used threads can be controlled by KEY_GPU_MAX_NUM_THREADS plugin option
  • Added new fusion for eltwise pattern from yolo-v5 (non-linear eltwises chain in the middle of fused sequence)
  • Added support for the following new operations:
    • TensorIterator-0 (must be explicitly enabled by setting KEY_GPU_ENABLE_LOOP_UNROLLING plugin option to NO)
    • ScatterNDUpdate-3
    • GatherND-5
    • Gather-7
  • Performance improvements for:
    • permute primitive
    • reorder between some blocked ↔ planar layouts
• MYRIAD plugin:
  • Performance fixes
    • Fixed most of the sporadic performance drops down to 10% by prioritizing runtime threads
  • Accuracy fixes
    • Accuracy and working capability on Yolo-v3 model through ONNX importer were fixed
    • Fixed Yolo-v5 accuracy
  • Operations
    • Fixed Average Pooling operation in case if there are paddings
    • Supported Interpolate operation with batch
  • MyriadX plugin
    • Enable cache support by adding IMPORT_EXPORT_SUPPORT to the list of supported metrics
    • Added option to disable preprocessing check inside the model
    • Fixed an issue when trying to run several MyriadX plugins in two or more threads
    • Fixed an issue related to challenges with importing precompiled blob with batch
    • Unlimited amount of DDR memory in the graph compiler and check it on a device directly
    • Fixed an issue related to challenges with loading a model when MyriadX plugin is being used with HETERO plugin
    • Fixed a memory leak in case there is a failure while booting a device
    • Fixed a memory leak in case a device can’t be booted from the first time, but successfully booted later
    • Fixed batch detection/removal logic in MYX plugin
  • Firmware
    • Fixed an issue when cache eviction would corrupt the read transfer payload
    • Fixed an issue on allocating memory for hardware operations when a device has more than 512Mb DDR memory
    • Fixed a hanging in case of transferring an input tensor with single size
  • myriad_perfcheck tool is deprecated and will be removed in 2022.1. Please, use benchmark_app tool instead
  • myriad_compile tool is deprecated and will be removed in 2022.1. Please, use compile_tool instead
• HDDL plugin:
  • The same performance, accuracy, operations, and firmware fixes as in MyriadX plugin
  • hddl_perfcheck tool is deprecated and will be removed in 2022.1. Please, use benchmark_app tool instead
  • myriad_compile tool is deprecated and will be removed in 2022.1. Please, use compile_tool instead
  • Added SCALAR layout handling 
  • Aligned functionality with IE API
• GNA plugin:

  • Introduced support for Fake Quantize layers which enabled GNA support for models after Post Training Optimization Tool

  • Extended support for Convolution and Eltwise operations which require splitting into several operations to be supported by GNA (to satisfy buffers size requirements)

  • Added support for MatMul operations which are mapped to GNA operations with the batch size greater than 8

  • Fixed bugs with LSTM cell unrolling

  • Fixed export/import of networks which require saving of inputs and outputs shape/layout information

  • Fixed concatenation of the layer with itself

nGraph

• Introduced opset7. The latest opset contains the new operations listed below. Not all OpenVINO™ toolkit plugins support the operations.
  • DFT-7
  • Einsum-7
  • Gather-7
  • Gelu-7
  • IDFT-7
  • Roll-7
• Supports of i4/u4 element types.

• Implemented public nGraph transformations:

  • LowLatency2
    LowLatency2 is a new version of LowLatency transformation that completely replaces the previous one. The transformation inserts Assign/ReadValue operations and connects them to TensorIterator/Loop operations, making step-by-step inference possible. The new version adds support for TensorIterator/Loop with multiple iterations and resolves serialization issues after applying the transformation. Available for CPU and GNA plugins.

• Public nGraph API changes
  • Several new constructors have been added for ngraph::function class:
    • Constructors that take a list of Results/Sinks and automatically detect existing Parameters and Variables in the function.
    • Constructors that additionally take a list of Variables.
  • add_variables, remove_variable, get_variables, get_variable_by_id methods have been added.
  • The class VariableContext has been introduced to work with the memory mechanism in nGraph, supported a functionality to initialize/update/reset State. 'evaluate' methods have been updated with EvaluationContext argument.

Post-Training Optimization Tool (POT)

• Introduced the quantization support for GNA via POT SW API. Added the Python speech sample which demonstrates the quantization of models from the Kaldi framework for GNA accelerator. 
• Telemetry support in POT. If the install process records an "accept" to send the telemetry data, the following information is sent by POT: user interface (CLI or API), target device, compression method, drop type and maximal drop used for the AccuracyAware algorithm, accuracy drop and the number of reverted layers after the quantization, the subset size used for collecting statistics, engine type, value of tune hyperparameters and model type parameters.
• Added INT8 support for the following operations:
  • Deconvolution/ConvolutionBackpropData
  • ConvertLike
• Improved the description of INT8 quantization path in OpenVINO documentation
• Extended models coverage: +40 INT8 models enabled

Neural Networks Compression Framework (NNCF)

• NNCF for PyTorch v1.7.0 and v1.7.1 released:
  • Integration with OTE for Instance Segmentation, Custom Object Detection and Horizontal Text Detection cases.
  • Added 7-bit quantization for weights to avoid the saturation issue on non-VNNI CPU
  • Support for pruning of models with FCOS detection heads and instance normalization operations
  • Added a mean percentile initializer for the quantization algorithm
  • Added Adjust Padding feature to support accurate execution of INT4 for VPU
  • Removed the pattern-based quantizer setup mode for the quantization algorithm 
  • Support for PyTorch 1.8.1

Deep Learning Workbench

• Support for TGL iGPU - full support including INT8 Calibration for TGL GPU
• Streamlined support for non-annotated datasets
  • Support for import of user's own images (replacing autogeneration of dataset with random noise) - individually or by directory with Images
  • Augmentation options (horizontal/vertical flip, random pixel clipping, noise injection, contrast/brightness modification) allows to increase the size of non-annotated dataset based on user's images
  • User can run INT8 (Default) quantization with non-annotated dataset, thus able to measure roofline increase in performance even w/o regular dataset
  • User can visualize inference results over images from non-annotated dataset
• OpenVINO JupyterLab extension. User can record actions in DL WB UI and get corresponding CLI commands to use in JupyterLab notebook (or e.g. use them directly in CLI). That is available to all major steps (MO conversion, Model Downloading, Profiling, Accuracy Measurement, INT8 calibration). 
• Decreased DL Workbench image size from 3 Gb to 1.1 Gb. Though some additional target framework installation steps are now being run for any first import of the model of supported framework. 
• Simplified downloading and starting procedure 
• Extended MO conversion options: pipeline configs, support of grayscale images
• Extended Yolo support (yolo3/4 family)
• UI/UX improvements

OpenCV*

• Updated version to 4.5.3.
• Added support of dynamically loaded UI backends (prebuilt GTK plugin included).
• Added support of clDNN OpenCL kernels cache parameter.

Samples

• Added the new Python sample (speech_sample), which demonstrates how to do a Synchronous Inference of acoustic model based on Kaldi* neural networks and speech feature vectors. The sample works with Kaldi ARK or Numpy* uncompressed NPZ files.
• Added support Numpy* uncompressed NPZ files for C++ speech sample.
• Improved C++/C/Python samples README with information of API and Feature coverage.

Open Model Zoo

Extended the Open Model Zoo with additional CNN-pretrained models and pre-generated Intermediate Representations (.xml + .bin). 

Replacing 2021.2 models:

• machine-translation-nar-en-de-0002
• ​machine-translation-nar-de-en-0002
• text-recognition-0014
• text-spotting-0005-detector
• text-spotting-0005-recognizer-decoder
• text-spotting-0005-recognizer-encoder

End-of-lifed: 

• machine-translation-nar-en-de-0001
• machine-translation-nar-de-en-0001
• text-recognition-0013
• text-spotting-0004-detector
• text-spotting-0004-recognizer-decoder
• text-spotting-0004-recognizer-encoder

New:

• common-sign-language-0002
• face-reidentification-0095
• noise-suppression-poconetlike-0001
• text-recognition-0015-encoder
• text-recognition-0015-decoder
• text-to-speech-en-multi-0001-duration-prediction
• text-to-speech-en-multi-0001-generation
• text-to-speech-en-multi-0001-regression
• time-series-forecasting-electricity-0001

The list of public models extended with the support for the following models:

The following models were removed from the list of public models:

Added  new  demo applications:

• Python face_recognition_demo (restored with updated model license)
• C++ G-API gaze_estimation_demo
• C++ image_processing_demo (combines deblurring and image super-resolution cases)
• Python noise_suppression_demo (removes background noise from the speech)
• C++ social_distance_demo
• Python speech_recognition_deepspeech_demo (renamed from speech_recognition_demo and extended with on-line processing mode, to reduce latency)
• Python speech_recognition_quartznet_demo
• Python time_series_forecasting_demo

object_detection_demo extended with support new models, including Yolo-V4-Tiny

Open Model Zoo tools:

OMZ tools implements sending telemetry statistics, when this option is accepted through install.
Model Downloader, Converter and Accuracy Checker now available in Python PyPi openvino-dev distribution

Support for new datasets and tasks was added into Accuracy Checker

Deep Learning Streamer

• GStreamer version updated to 1.18 bringing bug fixes and new features in GStreamer media and IO plugins
• Added support for human pose estimation models. This feature will enable 3D Athlete Tracking (3DAT) SDK built on top of DL Streamer. Added new sample `gst_launch/human_pose_estimation` with pipeline example
• Fixed issue with multi-model pipelines (ex, gvadetect and gvaclassify) on Intel® NCS2 (Neural Compute Stick 2)
• [Preview] Added support for action recognition models. Added sample `gst_launch/action_recognition` with pipeline example
• [Preview] New property ‘device’ in gvatrack (object tracking) and gvawatermark (overlay) elements allows to select CPU or GPU as target device
• Operating system deprecation notice: DL Streamer will drop support for CentOS and will introduce support for Red Hat Enterprise Linux (RHEL) 8 starting release 2022.1 (Q1’22).

OpenVINO™ Model Server

• Binary input data - ability to send inference requests using data in a compressed format like jpeg or png – significantly reducing communication bandwidth.  There is a noticeable performance improvement, especially with the REST API prediction calls and image data. For more details, see the documentation.  
• Dynamic batch size without model reloading – it is now possible to run inference with arbitrary batch sizes using input demultiplexing and splitting execution into parallel streams. This feature enables inference execution with OpenVINO Inference Engine without the side effect of changing the batch size for sequential requests and reloading models at runtime. For more details, see the documentation.  

• Practical examples of custom nodes – new or updated custom nodes: model zoo object detection , Optical Character Recognition and image transformation. These custom nodes can be used in a range of applications like vehicle object detection combined with recognition or OCR pipelines. Learn more about DAG Scheduler and custom nodes in the documentation.   
• Change model input and output layouts at runtime – it is now possible to change the model layout at runtime to NHWC. Source images are typically in HWC layout and such layout is used in image transformation libraries. Using the same layout in the model simplifies linking custom nodes with image transformations and avoids data transposing, it also reduces the load on clients and the overall latency for inference requests. Learn more  

• OpenVINO Toolkit Operator for OpenShift

  The OpenVINO Toolkit Operator for OpenShift 0.2.0 is included in the 2021.4 release. It has been renamed and has the following enhancements compared to previous OpenVINO Model Server Operator 0.1.0 released with 2021.3:

  • The Custom Resource for managing the instances of OpenVINO Model Server is renamed from Ovms to ModelServer.

  • ModelServer resources can now manage additional parameters: annotations, batch_size, shape, model_version_policy, file_system_poll_wait_seconds, stateful, node_selector, and layout. For a list of all parameters, see the documentation.

  • The new Operator integrates OpenVINO Toolkit with OpenShift Data Science —a managed service for data scientists and AI developers offered by Red Hat. The Operator automatically builds a Notebook image in OpenShift which integrates OpenVINO develop tools and tutorials with the JupyterHub spawner.

  • Operator 0.2.0 is currently available for OpenShift only. Updates to the Kubernetes Operator will be included in a future release.

New Distributions

• Containers:
  • Now RHEL 8 runtime Docker image is available on Red Hat* Ecosystem Catalog  container registry with CPU, GPU plugins support.
    • Includes Inference Engine and OpenCV.
    • Supports CPU and GPU devices.
  • Add special images with `_tgl` tag to natively support inference on 11th Generation Intel® Core™ Processor Family for Internet of Things (IoT) Applications (formerly codenamed Tiger Lake) from OpenVINO Docker container.

Preview Features / Support Terminology

A preview feature is a functionality that is being introduced to gain early feedback from developers. You are encouraged to submit your comments, questions, and suggestions related to preview features to the forum.

Known Issues

Included in This Release

The Intel® Distribution of OpenVINO™ toolkit is available in these versions:

• OpenVINO™ toolkit for Windows*
• OpenVINO™ toolkit for Linux*
• OpenVINO™ toolkit for macOS*

Where to Download This Release

• Intel® Software Development Products Registration Center
• Container and Dockerfiles:
  • DockerHub*
  • Red Hat* Quay.io (started from 2021.2 release)
  • Red Hat* Ecosystem Catalog (started from 2021.2 release)

  • DockerHub CI
    DockerHub CI framework can generate a Dockerfile, build, test, and deploy an image with the Intel® Distribution of OpenVINO™ toolkit.
    You can reuse available Dockerfiles, add your layer and customize the image of OpenVINO™ for your needs.

  • In addition, Intel® Distribution of OpenVINO™ toolkit for Linux* is available to install through the

    • APT repository
    • YUM repository
  • Anaconda* Cloud (conda install -c intel openvino-ie4py)
  • Python Packages:
    • https://pypi.org/project/openvino/ (pip install openvino)
    • https://pypi.org/project/openvino-dev/ (pip install openvino-dev)
• Archival packages with product bits can be downloaded here: https://storage.openvinotoolkit.org/repositories/openvino/packages/

System Requirements

Disclaimer: Certain hardware (including but not limited to GPU and GNA) requires installation of specific drivers to work correctly. Drivers might require updates to your operating system, including Linux kernel, please refer to the their documentation. Operating system updates should be handled by user, and are not part of OpenVINO installation.

Intel® CPU Processors

Hardware:

• Intel® Atom* processor with Intel® SSE4.2 support
• Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics
• 6th - 12th generation Intel® Core™ processors
• Intel® Xeon® processor Scalable Processors (formerly Skylake)
• 2nd Generation Intel® Xeon® Scalable Processors (formerly Skylake and Cascade Lake)
• 3rd Generation Intel® Xeon® Scalable Processors (formerly Cooper Lake  and Ice Lake)

Operating Systems:

• Ubuntu* 18.04 long-term support (LTS), 64-bit
• Ubuntu* 20.04 long-term support (LTS), 64-bit - preview support
• Windows* 10, 64-bit
• macOS* 10.15, 64-bit
• CentOS* 7, 64-bit
• For deployment scenarios on Red Hat* Enterprise Linux* 8.2 (64 bit), you can use the of Intel® Distribution of OpenVINO™ toolkit run-time package that includes the Inference Engine core libraries, nGraph, OpenCV, Python bindings, CPU and GPU plugins. The package is available as:
  • Downloadable archive
  • PyPi package
  • Docker image                                   

Intel® Processor Graphics

Hardware:

• Intel® HD Graphics
• Intel® UHD Graphics
• Intel® Iris® Xe Graphics
• Intel® Iris® Xe Max Graphics 
• Intel® Iris® Pro Graphics

Operating Systems:

• Ubuntu* 18.04 long-term support (LTS), 64-bit
• Windows* 10, 64-bit
• Yocto* 3.0, 64-bit
• For deployment scenarios on Red Hat* Enterprise Linux* 8.2 (64 bit), you can use the of Intel® Distribution of OpenVINO™ toolkit run-time package that includes the Inference Engine core libraries, nGraph, OpenCV, Python bindings, CPU and GPU plugins. The package is available as:
  • Downloadable archive
  • PyPi package
  • Docker image                  

NOTE: This installation requires drivers that are not included in the Intel Distribution of OpenVINO toolkit package

NOTE:  A chipset that supports processor graphics is required for Intel® Xeon® processors. Processor graphics are not included in all processors. See Product Specifications for information about your processor.

Intel® Gaussian & Neural Accelerator (Intel® GNA)

Operating Systems:

• Ubuntu* 18.04 long-term support (LTS), 64-bit
• Windows* 10, 64-bit

Intel® VPU Processors

Intel® Vision Accelerator Design with Intel® Movidius™ Vision Processing Units (VPU)

Operating Systems:

• Ubuntu* 18.04 long-term support (LTS), 64-bit (Linux Kernel 5.2 and below)
• Windows* 10, 64-bit
• CentOS* 7.6, 64-bit

Intel® Movidius™ Neural Compute Stick and Intel® Neural Compute Stick 2

Operating Systems:

• Ubuntu* 18.04 long-term support (LTS), 64-bit
• CentOS* 7.6, 64-bit
• Windows* 10, 64-bit
• Raspbian* (target only)

AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1

Operating Systems:

• Windows* 10, 64-bit

Components Used in Validation

Operating systems used in validation:

• Linux* OS
  • Ubuntu 18.04.3 with Linux kernel 5.3
    •  Ubuntu 18.04.3 with Linux kernel 5.6 for 10th Generation Intel® Core™ Processors (formerly codenamed Ice Lake ) and 11th Generation Intel® Core™ Processor Family for Internet of Things (IoT) Applications  (formerly codenamed Tiger Lake)
  • Ubuntu 20.04.0 with Linux kernel 5.4
  • CentOS 7.6 with Linux kernel 5.3
  • A Linux* OS build environment needs these components:
    • GNU Compiler Collection (GCC)* 4.8 (CentOS 7), 7.5 (Ubuntu 18), 9.3 (Ubuntu 20)
    • CMake* 3.13 or higher
    • Python* 3.6-3.7, additionally 3.8 for Ubuntu 20
    • OpenCV 4.5
    • Intel® Graphics Compute Runtime. Required only for GPU.
      • 19.41
      • 20.35 for 10th Generation Intel® Core™ Processors (formerly codenamed Ice Lake ) and 11th Generation Intel® Core™ Processor Family for Internet of Things (IoT) Applications  (formerly codenamed Tiger Lake)
      • 21.29 for 12th Gen Intel® Core™ Desktop Processors (formerly codenamed Alder Lake S)
• Windows 10 version 1809 (known as Redstone 5)
  • A Windows* OS build environment needs these components:
    • Microsoft Visual Studio* 2019
    • CMake 3.14 or higher
    • Python* 3.6-3.8
    • OpenCV 4.5
    • Intel® HD Graphics Driver. Required only for GPU.
    • Intel® Iris® Xe MAX Dedicated Graphics Drivers. Required only for 11th Generation Intel® Core™ Processor with Intel® Iris® Xe Max Graphics.
    • There are "universal" drivers at https://downloadcenter.intel.com/product/202986/11th-Generation-Intel-Core-i7-Processors but we noticed some regressions in drivers 27.20.100.9126 and newer on Tiger Lake, so the recommended one is 27.20.100.9079 (Dec 23, 2020).
    • 30.0.100.9928 for 12th Gen Intel® Core™ Desktop Processors.
• macOS 10.15

  • A macOS build environment requires these components:

    • Xcode* 10.3
    • OpenCV 4.5
    • Python 3.6-3.7
    • CMake 3.13 or higher

DL frameworks used for validation:

• TensorFlow 1.15.2, 2.2.0 (limited support according to product features)
• MxNet 1.5.1

NOTE: Version of CMake specified above is to build OpenVINO from source. Building samples and demos from the Intel® Distribution of OpenVINO™ toolkit package requires CMake* 3.10 or higher (except of Windows where CMake 3.14 is required as the first supporting Visual Studio 2019).

Helpful Links

NOTE: Links open in a new window.

• Home Page
• Featured Documentation
• All Documentation, Guides, and Resources
• Community Forum

OpenVINO™ Logo

To build equity around the project, the OpenVINO logo was created for both Intel and community usage. The logo may only be used to represent the OpenVINO toolkit and offerings built using the OpenVINO toolkit.

Logo Usage Guidelines

The OpenVINO logo must be used in connection with truthful, non-misleading references to the OpenVINO toolkit, and for no other purpose. Modification of the logo or use of any separate element(s) of the logo alone is not allowed.