Note For the Release Notes for the 2019 version, refer to Release Notes for Intel® Distribution of OpenVINO™ Toolkit 2019.
Introduction
The Intel® Distribution of OpenVINO™ toolkit is a comprehensive toolkit for quickly developing applications and solutions that emulate human vision. Based on Convolutional Neural Networks (CNNs), the toolkit extends CV workloads across Intel® hardware, maximizing performance. It accelerates applications with high-performance, AI and deep learning inference deployed from edge to cloud.
The Intel Distribution of OpenVINO toolkit:
- Enables deep learning inference from the edge to cloud.
- Supports heterogeneous execution across Intel accelerators, using a common API for the Intel® CPU, Intel® Integrated graphics, Intel® Gaussian & Neural Accelerator (Intel® GNA), Intel® Movidius™ Neural Compute Stick (NCS), Intel® Neural Compute Stick 2 (Intel® NCS2), Intel® Vision Accelerator Design with Intel® Movidius™ Vision Processing Unit (VPU), Intel® Vision Accelerator Design with Intel® Arria® 10 FPGA Speed Grade 2.
- Speeds time-to-market through an easy-to-use library of CV functions and pre-optimized kernels.
- Includes optimized calls for CV standards, including OpenCV and OpenCL™.
New and Changed in the Release 4
Executive Summary
- Improves performance while maintaining accuracy close to full precision (for example, FP32 data type) by introducing support for the Bfloat16 data type for inferencing using the 3rd generation Intel® Xeon® Scalable processor (formerly code-named Cooper Lake).
- Increases accuracy when layers have varying bit-widths by extending the Post-Training Optimization Tool to support mixed-precision quantization.
- Allows greater compatibility of models by supporting directly reading Open Neural Network Exchange (ONNX*) model format to the Inference Engine.
- For users looking to take full advantage of Intel® Distribution of OpenVINO™ toolkit, it is recommended to follow the native workflow of using the Intermediate Representation from the Model Optimizer as input to the Inference Engine.
- For users looking to more easily take a converted model in ONNX model format (for example, PyTorch to ONNX using torch.onnx), they are now able to input the ONNX format directly to the Inference Engine to run models on Intel architecture.
- Enables initial support for TensorFlow* 2.2.0 for computer vision use cases.
- Enables users to connect to and profile multiple remote hosts; collect and store data in one place for further analysis by extending the Deep Learning Workbench with remote profiling capability.
Backward Incompatible Changes Compared with v.2020 Release 3
- List of Deprecated API, API Changes
- IRv7 is deprecated and support for this version will be removed in the next major version (v.2021.1). It means IRv7 and older cannot be read using Core::ReadNetwork. Users are recommended to migrate to IRv10 to prevent compatibility issues in the next release.
- Removed Inference Engine NNBuilder API. Users are recommended to use nGraph as a replacement to create a CNN graph from C++ code.
- The following Inference Engine public API will be removed in the next release (v.2021.1):
- InferencePlugin, IInferencePlugin and InferencEnginePluginPtr classes. The Core class should be used as a replacement.
- PluginDispatcher class. The Core class should be used as a replacement.
- CNNNetReader class. The Core::ReadNetwork class should be used as a replacement.
- PrimitiveInfo, TensorInfo and ExecutableNetwork::GetMappedTopology. Use ExecutableNetwork::GetExecGraphInfo as a replacement.
- ICNNNetworkStats, NetworkNodeStats, CNNNetwork::getStats and CNNNetwork::setStat. Use IRv10 with FakeQuantize approach for int8 flow replacement.
- IShapeInferExtension and CNNNetwork::addExtension. Use IExtension class as a container for nGraph::Nodes which implement shape inference.
- IEPlugin from Inference Engine Python API. Use Core API instead.
- Data::getCreatorLayer, Data::getInputTo and CNNLayer. Use CNNNetwork::getFunction to iterate over a graph.
Model Optimizer
Common changes
- Fixed Model Optimizer bugs to generate reshape-able IRs of many models with the command-line parameter "--keep_shape_ops".
- Fixed Model Optimizer transformations to set operations name in an IR equal to the original framework model operation name.
- Implemented an optimization transformation to replace a sub-graph with the TanH operation into the GeLU operation.
ONNX*
- Added support for the following operations:
- Clip-11
- CumSum-11
- NonZero-9
- Pad-11
- ReduceL2-11
- ReverseSequence-10
- ROIAlign-10
- Scatter-11
- ScatterElements-11
- Softplus-1
- TopK-10
- QuantizeLinear-10/DequantizeLinear-10 pair when it can be replaced with FakeQuantize operation
- Where-9
TensorFlow*
- Introduced initial support for the TensorFlow* 2.X models, specifically for CNN (or computer vision use cases). RNN and Embedding layers are not yet supported.
- Added support for the following operations:
- IdentityN
- NoOp
- Where
MXNet*
- Added support for the following operation:
- CumSum
Kaldi*
- Added support for the following operations:
- fixedbiascomponent
- fixedscalecomponent
Inference Engine
Common changes
- The primary CPU preprocessing pipeline is now fully optimized for AVX2 and AVX512. Previously, only SSE4.2-based optimizations were available.
- The Core::ReadNetwork method was re-implemented to use readers plugins that are dynamically loaded in runtime:
- Implemented the inference_engine_onnx_reader plugin to read native ONNX models. This plugin depends on the ONNX Importer and uses it to read ONNX models.
- Implemented the inference_engine_ir_reader plugin to read IRs.
Inference Engine Python API
- No changes.
Inference Engine C API
- Added ie_core_read_network_from_memory to ensure encrypted reading.
CPU Plugin
- Introduced support for bfloat16 (BF16) data type for inferencing. This feature improves performance while keeping the same accuracy when compared with FP32 model on targets with native BF16 support (for example, 3rd generation Intel® Xeon® Scalable Processors, formerly code-named Cooper Lake)
- Several operations from opset3 were implemented:
- Bucketize-3
- CumSum-3
- EmbeddingBagOffsetsSum-3
- EmbeddingBagPackedSum-3
- EmbeddingSegmentsSum-3
- ExtractImagePatches-3
- Select, DepthToSpace, SpaceToDepth operations were updated to be conformant with latest opset.
- INT8 inference performance enhancements for popular use-cases: NLP (BERT), super-resolution, person detection and pose-estimation.
GPU Plugin
GPU Plugin
- Supported several new operations:
- CumSum-3
- EmbeddingBagOffsetsSum-3
- EmbeddingBagPackedSum-3
- EmbeddingSegmentsSum-3
- BatchToSpace
- Decomposition for the following operations was disabled, so now they are supported directly:
- ShuffleChannels
- DepthToSpace
MYRIAD Plugin
- Product Change Notice: Intel® Distribution of OpenVINO™ toolkit
Change Notice Begins July 2020 Change Date October 2020 - Intel® Movidius™ Neural Compute Stick support is discontinued.
- 3 LTS is the last one with Myriad-2 HW support.
- Supported new operations:
- NonZero
- ScatterElementsUpdate
- Broadcast
- Added I32 data type support in many arithmetic operations
- Improved performance of existing operations:
- LSTMCell
- LSTMSequence
- Gather
- TopK
- OneHot
- OpenCL compiler, targeting Intel® Neural Compute Stick 2 for the SHAVE* processor only, is redistributed with OpenVINO. OpenCL support is provided by ComputeAorta*, and is distributed under a license agreement between Intel® and Codeplay* Software Ltd.
HDDL Plugin
- Same new operations and optimizations as in MYRIAD plugin.
- OpenCL compiler, targeting Intel® Vision Accelerator Design with Intel® Movidius™ VPUs for the SHAVE* processor only, is redistributed with OpenVINO. OpenCL support is provided by ComputeAorta*, and is distributed under a license agreement between Intel® and Codeplay* Software Ltd.
GNA Plugin
- GNA library updated to v2:
- Limitation on the number of layers is softened
- Several networks can now be created simultaneously on the GNA device
- Removed dependency on Intel® C++ Compiler runtime on Linux
- on Windows: available via Windows Update
- on Ubuntu: see the link in the documentation for the GNA plugin
- Extended support for Permute layer:
- all variants of the layer that do not require memory reordering ("trivial permutes") are now supported
- a non-trivial permute is now supported when at least one of the reordered dimensions is <=8
- Added support for a number of new layer combinations
- Support for GNA specific parameters in the C++ benchmark app
- Support for dynamic switching of execution mode (SW vs HW)
- Up to 100 input layers are now supported instead of 10
- Memory optimizations
FPGA Plugin
Product Change Notice
Intel® Distribution of OpenVINO™ toolkit for Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA
Change Notice Begins | July 2020 |
Change Date | October 2020 |
Intel will be transitioning to the next-generation programmable deep-learning solution based on FPGAs in order to increase the level of customization possible in FPGA deep-learning. As part of this transition, future standard releases (i.e., non-LTS releases) of Intel® Distribution of OpenVINO™ toolkit will no longer include the Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA.
Intel® Distribution of OpenVINO™ toolkit 2020.3.X LTS release will continue to support Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA and the Intel® Programmable Acceleration Card with Intel® Arria® 10 GX FPGA. For questions about next-generation programmable deep-learning solutions based on FPGAs, please talk to your sales representative or contact us to get the latest FPGA updates.
nGraph
- Moved nGraph source code to the OpenVINO™ toolkit repository.
- Removed dependency on Inference Engine transformations library to the Inference Engine. Now the library only depends on nGraph.
- Separated ONNX importer from nGraph, so now it is a separate dynamic library which depends on nGraph library.
- Introduced opset3. The new opset contains the following new operations (not all OpenVINO™ toolkit plugins support new operations):
- Assign-3
- Broadcast-3
- Bucketize-3
- CumSum-3
- EmbeddingBagOffsetsSum-3
- EmbeddingBagPackedSum-3
- EmbeddingSegmentsSum-3
- ExtractImagePatches-3
- GRUCell-3
- NonMaximumSuppression-3
- NonZero-3
- ReadValue-3
- RNNCell-3
- ROIAlign-3
- ScatterElementsUpdate-3
- ScatterUpdate-3
- ShapeOf-3
- ShuffleChannels-3
- TopK-3
Neural Networks Compression Framework (NNCF)
- NNCF for PyTorch is completely open-sourced under OpenVINO™ toolkit repository
- NNCF v1.3.1 for PyTorch has been released:
- Automatic mixed (INT4+INT8) precision quantization (HAWQ-powered) has been released. Two models have been provided to evaluate performance gain, accuracy metrics for compressed models are as follows:
Model Compression Acc1@Imagenet
(FP32/Compressed)MobileNet V2 46.6% INT8
53.4% INT471.81%/70.89% ResNet-50 44.8% INT8
55.2% INT476.13%/76.30% - Implemented and integrated to NNCF Soft Filter Pruning and Geometric Median Pruning algorithms. Requested pruning rate (PR) can’t be achieved in some cases as accuracy drop of 1% is preserved. However, performance (inference on CPU) always scales similarly to achieved pruning rate.
Model name Requested PR Actual PR MACs FPS resnet18 30% 15.8% -16.9% +18.4%
resnet34 30% 21.7% -22.1% +25.1%
resnet50 30% 28.3% -30% +33.1%
googlenet 30% 36.4% -41.9% +23%
- HW configuration files support has been added to the tool in order to enable quantization parameters tuning for specific HW targets. The API and format of the configuration files are close to that of the Post-training Optimization Tool - just extending them with NNCF specifics.
- Release supports standard ONNX opset INT8 export, latest PyTorch, TorchVision and CUDA runtime versions. Apart of that, Docker support and CPU-only training setup are enabled.
- Automatic mixed (INT4+INT8) precision quantization (HAWQ-powered) has been released. Two models have been provided to evaluate performance gain, accuracy metrics for compressed models are as follows:
- NNCF has been used to quantize and fine-tune a number of models from the Transformers-based family: BERT-large and DistilBert. INT8 models all have <1% accuracy drop and use symmetric quantization that should fit well to CPU performance-wise:
Model Dataset Accuracy (FP32/INT8)
Notes BERT-large (uncased, WWM)
SQuAD v1.1 93.26%/92.60% (F1)
Fully symmetrically quantized
Release to OMZ
1.5x perf gain vs FP32 on CPUBERT-large (uncased)
SQuAD v1.1 90.87%/90.41% (F1)
Fully symmetrically quantized
Model baseline for MLPerf submission
Quality metrics on par with Nvidia INT8DistilBERT (uncased)
Yelp (SST-2-like)
95.50%/95.14% Fully symmetrically quantized
POC for customers (Roblox)DistilBERT (uncased) SST-2 91.10%/90.30% Fully symmetrically quantized
Post-Training Optimization Tool
Post-Training Optimization Tool
- Added support of quantization (Default, Accuracy Aware) for cascaded models. The quantization path requires support and models configuration through Accuracy Checker (AC) configs:
- MTCNN: coarse detector (MTCNN-p) to detector/refining (MTCNN-r) to detector/output (MTCNN-o)
- Action Recognition: encoder (classification backbone, 16 frames) to decoder (multi-head self-attention based)
- Text Spotting: detection+segmentation+embeddings (mask-RCNN-like) to encoder (conv net on embeddings) to decoder (GRU + attention)
- Introduced mixed quantization (for example, symmetric quantization for weights and asymmetric quantization for activations), which can be used as a quantization scheme for Default and Accuracy Aware algorithms to get better accuracy. 8 models out of all are quantized using this scheme, default quantization scheme is still symmetric.
- Improved annotation-free quantization capability
- Enhanced overall usability, which allows users to integrate the quantization workflow with a model validation pipeline by using an API in order to get the model quantized
- Added new documentation of Post-Training Optimization Best Practices and Low Precision Optimization Guide
- The list of layers that are being quantized has been extended to support NormalizeL2. This brings performance benefits for few models like SSD, person-detection-action-recognition (see S-curve below, INT8 2020.4 vs 2020.2 on CLX).
Deep Learning Workbench
- Added support for profiling on remote hosts and targets available through SSH protocol
- Added support for Semantic and Instance segmentation models
- Added several usability improvements, including improved group inference configuration, improved history visualization, and a clearer system business indication
OpenCV*
- SIFT algorithm has been moved from opencv_contrib repository after patent expiration
- Enabled parallel IPP acceleration for GaussianBlur in OpenVINO builds
Examples and Tutorials
- Speech libraries and end-to-end demos demonstrating speech recognition pipeline are now included to CentOS distribution.
Open Model Zoo
- Extended the Open Model Zoo with additional CNN-pretrained models and pregenerated Intermediate Representations (
.xml
+.bin
):- [NEW] bert-small-uncased-whole-word-masking-squad-0001
- [NEW] bert-large-uncased-whole-word-masking-squad-fp32-0001
- [NEW] bert-large-uncased-whole-word-masking-squad-int8-0001
- [NEW] person-detection-0100
- [NEW] person-detection-0101
- [NEW] person-detection-0102
- [NEW] person-detection-0106
- [NEW] person-reidentification-retail-0265
- [NEW] yolo-v2-tiny-vehicle-detection-0001
- [NEW] vehicle-attributes-recognition-barrier-0042
- [NEW] weld-porosity-detection-0001
- person-reidentification-retail-0267 - Replaces the person-reidentification-retail-0249 model.
- person-reidentification-retail-0270 - Replaces the person-reidentification-retail-0300 model.
- Removed models:
- person-reidentification-retail-0031
- face-reidentification-retail-0095 - has been temporarily removed
- The list of public models extended with support of the following models:
Model Name Framework faceboxes-pytorch PyTorch fast-neural-style-mosaic-onnx PyTorch googlenet-v2-tf TensorFlow hbonet-1.0 PyTorch hbonet-0.5 PyTorch hbonet-0.25 PyTorch mobilefacedet-v1-mxnet MXNet mobilenet-v3-large-1.0-224-tf TensorFlow mobilenet-v3-small-1.0-224-tf TensorFlow open-closed-eye-0001 Caffe2 pelee-coco Caffe resnet-34-pytorch PyTorch retinaface-anti-cov MXNet retinaface-resnet50 MXNet retinanet-tf TensorFlow ssd-resnet34-1200-onnx ONNX vehicle-reid-0001 Caffe2 - Added new demo applications:
- Classification
- Bert question answering
- Object detection faceboxes
- Object detection retinaface
- Whiteboard inpainting
- Open Model Zoo tools
- Functionality to resume download, in case if the user doesn't have a stable Internet connection, which is added to the Open Model Zoo Model Downloader tool
- Quantization script extended with support public models trained on COCO and Wider Face datasets
Deep Learning Streamer
- Added support for post-processing of RCNN, YoloV2 and YoloV3-tiny models in gvadetect
- Provided an option to compact metadata representation attached to a video frame in order to reduce latency
- Added a new sample for gvametapublish element. The sample outputs the metadata to file (default=stdout), MQTT or Kafka
- Added a new property ‘json-indent’ in gvametaconvert to pretty-print json with indentation specified to make the syntax more legible
- Added two new tutorials in Jupyter Notebook format, for the developers to try DL Streamer on Intel® hardware using the Intel® DevCloud for the Edge: Advanced Tutorials
- Preview in open source repository: GStreamer element gvaaudiodetect for audio event detection and the corresponding sample pipeline
For more information on DL Streamer, please refer to the DL Streamer tutorial, API reference, and samples documentation at OpenVINO Inference Engine Samples. For the preview feature, please visit DL Streamer open source project located at GStreamer Video Analytics (GVA) Plugin repository on GitHub.
New and Changed in the Release 3 LTS
Executive Summary
- Introduced the Long Term Support policy with this release. Please, read more details here: Long Term Support Release.
- This release has a separate Release Notes - OpenVINO v.2020.3 LTS and further updates for LTS will be posted on that page.
New and Changed in the Release 2
Executive Summary
- Introduces the new Deep Learning (DL) Streamer component, which uses the GStreamer* multimedia framework. This component enables the creation and deployment of analytic pipelines across Intel architecture from edge to cloud.
- Includes bug fixes and stability fixes in memory leak issues in the Intel® Threading Building Blocks (Intel® TBB) library.
- Improves support for noise suppression use cases by adding new layers and bug fixes to the Intel Gaussian & Neural Accelerator (Intel GNA) plug-in.
- Adds support for new FP32 and INT8 models to enable more vision and text use cases:
- Original models: 3D U-Net, MobileFace
- Low precision (that is, INT8) models: EAST, OpenPose, RetinaNet, and FaceNet
- Introduces new pretrained models on Open Model Zoo to reduce time to production, which include:
- Multi-camera tracking for person reidentification
- Cascade R-CNN for person detection
- Published new performance benchmarks.
Backward Incompatible Changes Compared with v.2020 Release 1
- List of Deprecated API, API Changes
- Part of the Inference Engine shared library has been moved to the Inference Engine Legacy shared library, which contains obsolete routines needed for Intermediate Representation (IR) v7. You should link custom applications against both libraries, which is done automatically if you use the CMake* interface.
Model Optimizer
Common Changes
- Implemented generation of a compressed OpenVINO IR suitable for INT8 inference, which takes up to 4 times less disk space than an expanded one. Use the --disable_weights_compression Model Optimizer command-line parameter to get an expanded version.
- Implemented an optimization transformation to replace a sub-graph with the Erf operation into the GeLU operation.
- Implemented an optimization transformation to replace a sequence of Split-Concat operations to the Interpolate operation.
- Fixed a number of Model Optimizer bugs to generate reshape-able IRs of many models with the command-line parameter --keep_shape_ops.
- Fixed a number of Model Optimizer transformations to set operations name in an IR equal to the original framework model operation name.
ONNX*
- Added support for the MeanVarianceNormalization operation if normalization is performed over spatial dimensions.
TensorFlow*
- Added support for the TensorFlow* Object Detection models version 1.15.X.
- Added support for the following operations:
- BatchToSpaceND
- SpaceToBatchND
- Floor
MXNet*
- Added support for the Reshape operation with input shape values equal to -2, -3, and -4.
Inference Engine
Common Changes
- Refactored the Inference Engine Extension API. The new API allows to create Network with custom nGraph operations and load custom CPU execution kernels for custom nGraph operations.
- Added the Inference Engine Transformation Library, which contains nGraph transformation for the conversion of opset2 operations to opset1, and opset1 operations to the legacy operation set.
- Added the Inference Engine LP transformations Library, which contains low precision transformation routines. The library is linked directly to plugins and not a dependency of the Inference Engine shared library.
- Improved the support of AVX2 and AVX512 instruction sets in the CPU preprocessing module.
Inference Engine C API
- Added support of I420, NV12 blobs and added the dedicated hello_nv12_classifiction_c sample.
CPU Plugin
- Removed the mkltiny dependency. This allowed to significantly decrease the binary distribution size required for inference on CPU.
- Added several new operations:
- GeLU
- Swish
- BatchToSpace
- SpaceToBatch
- Continued optimizations for the INT8 pipeline that resulted in 1.2x geomean speedup compared with the 2020.1 release on about 140 topologies.
- Significantly improved load time for INT8 models.
GPU Plugin
GPU Plugin
- Added support for the following new operations:
- Swish
- VariadicSplit
- Transpose
- SpaceToDepth
- ConvertLike
- Select
- Added 3D-spatial dimensions support for the Resample primitive.
- Fixed all known memory leaks.
- Implemented performance optimizations for grouped convolutions, topk primitive, and models with batch size less than 1.
- Implemented minor load time improvements for models with huge layers count or weights size.
MYRIAD Plugin
- Aligned VPU firmware with Intel® Movidius™ Myriad™ X Development Kit (MDK) R11 version.
- Added support for 3D convolutions with restriction on grouped convolution size to 1.
- Added support for 3D pooling.
HDDL Plugin
- Linux* kernel version 5.3 (and below) are supported now.
- Due to the compatibility issue, the ION driver cannot be installed on Ubuntu* with a kernel version higher than 5.0 (included) and CentOS*, fall back to use shared memory.
GNA Plugin
- Added support for the following layers:
- Abs
- Exp
- Log
- Sign
- Softsign
- Unsqueeze
- Fixed problems with some layer combinations.
- Fixed export and import of some models.
- Fixed a problem with configuration key validation.
FPGA Plugin
- Intel® Vision Accelerator Design with an Intel® Arria 10 FPGA (Mustang-F100-A10) speed grade 1 bitstreams and Intel® Programmable Acceleration Card (Intel® PAC) with Intel® Arria® 10 GX FPGA are not included in the Intel Distribution of OpenVINO toolkit 2020.2 release.
- If you use Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and Intel Programmable Acceleration Card (Intel PAC) with Intel Arria 10 GX FPGA, we recommend continuing to use the Intel Distribution of OpenVINO toolkit 2020.1 release.
nGraph
- Introduced opset2. This opset contains the following operations that were not included to opset1:
- BatchToSpace
- GELU
- MVN
- ReorgYolo
- ROIPooling
- SpaceToBatch
- Removed the nGraph Python* API.
- Enabled building nGraph ONNX Importer low-level API as a part of the nGraph shared library. This is a preview feature, which enables creating the nGraph Function directly from an ONNX model in runtime without running the Model Optimizer.
Post-Training Optimization Tool
Post-Training Optimization Tool
- Additional models supported:
- Added 25% more quantized and executed models (145 vs 116, Open Model Zoo scope), all within 1% of accuracy drop.
- Enabled RetinaNet, Open Pose, Open Model Zoo/Human-pose (Single, 3d-pose), Mobile Face Det, and EAST.
- Enabled the most of new Open Model Zoo models except the pipelined ones and asl-recognition-0004
- Usability Improvements:
- Implemented the annotation-free Default quantization (Simplified mode):
- The mode allows quantizing a CNN without the Accuracy Checker support.
- The mode accepts a set of images (unannotated dataset), performs default preprocessing and runs the Default quantization.
- The mode is NOT applicable for complex models, which work with many inputs, specific pre- and postprocessing, dataset different from a set of images.
- Added the Quantization recipe (quantization_parameters section) to IR (*.xml) to show quantization parameters.
- Implemented the annotation-free Default quantization (Simplified mode):
- Implemented the new Bias Correction algorithm to improve the accuracy of quantized models.
- The algorithm is slower but more precise than the previous version of Bias Correction. For example, it allows to get Mobilenet-v2/CF, mobilenet-v1-0.25/TF, image-retrieval-0001, text-detection-0004, text-recognition-0012 , person-reidentification-retail-0248, and other models fully quantized with accuracy drop below 1%.
- Previous Bias Correction algorithm is now called FastBiasCorrection. To activate the new one, set the quantization parameter "use_fast_bias": false.
- Post-Training Optimization tool generates a compressed Intel Distribution of OpenVINO toolkit intermediate representation suitable for INT8 inference, which takes up to 4 times less disk space than an FP32 one. Use --keep_uncompressed_weights to get an expanded version.
Deep Learning Workbench
- Extended support for the INT8 Post-Training Optimization Tool. Now pretrained models can be calibrated to the INT8 precision with an annotation-free dataset (directory with images), which enables you to calibrate to lower precision without the need for a dataset - inputting images is sufficient. Only the Default (Maximum Performance) algorithm currently supports this capability.
- Integrated the Deployment Manager, which allows users to get the most minimal package size for deployment after experimentations and fine-tuning using the DL Workbench
- Added support for the Intel Vision Accelerator Designs target (on Linux only).
- Separated calibration and validation datasets, which enables you to use different datasets for calibration and accuracy/performance measurements.
- Implemented multiple usability and experience improvements, including a new look and feel for the comparison mode.
- Enabled a new authentication mechanism using tokens, which provides additional security protection for users' models and datasets.
OpenCV*
- Version changed to 4.3.0.
- nGraph API is used by default in Inference Engine backend API in the dnn module. The NN Builder API is deprecated, but have not been removed yet. You can enable it with the cv::dnn::setInferenceEngineBackendType function.
Examples and Tutorials
- Added sample to demonstrate nGraph C++ API. The sample demonstrates how to execute an inference using ngraph::Function to create a network. The sample uses the LeNet classifications network as an example.
Open Model Zoo
- Extended the Open Model Zoo, which includes additional CNN-pretrained models and pregenerated Intermediate Representations (.xml + .bin):
- [NEW] face-detection-0100
- [NEW] face-detection-0102
- [NEW] face-detection-0104
- [NEW] face-detection-0105
- [NEW] face-detection-0106
- [NEW] unet-camvid-onnx-0001
- [NEW] yolo-v2-ava-0001
- [NEW] yolo-v2-ava-sparse-35-0001
- [NEW] yolo-v2-ava-sparse-70-0001
- [NEW] yolo-v2-tiny-ava-0001
- [NEW] yolo-v2-tiny-ava-sparse-30-0001
- [NEW] yolo-v2-tiny-ava-sparse-60-0001
- [NEW] instance-segmentation-security-1025
- asl-recognition-0004 - Replaces the 2020 Release 1 model.
- person-reidentification-retail-0248 - Replaces the 2020 Release 1 model.
- person-reidentification-retail-0249 - Replaces the 2020 Release 1 model.
- person-reidentification-retail-0300 - Replaces the 2020 Release 1 model.
- text-spotting-0002-detector - Replaces the 2020 Release 1 model.
- text-spotting-0002-recognizer-decoding - Replaces the 2020 Release 1 model.
- text-spotting-0002-recognizer-encoder - Replaces the 2020 Release 1 model.
- faster-rcnn-resnet101-coco-sparse-60-0001 - Replaces the 2020 Release 1 model.
- handwritten-japanese-recognition-0001 - Replaces the 2020 Release 1 model.
- icnet-camvid-ava-0001 - Replaces the 2020 Release 1 model.
- icnet-camvid-ava-sparse-30-0001 - Replaces the 2020 Release 1 model.
- icnet-camvid-ava-sparse-60-0001 - Replaces the 2020 Release 1 model.
- Added new demo applications:
- Colorization
- Image Inpainting
- MonoDepth
- Model Downloader tool configuration files are placed in separate per-model folders in order to improve user experience and simplify contribution process (less merge conflicts while developing/merging several models at the same time). The list is extended to support the following public models in Caffe2*, TensorFlow*, MXNet*, and PyTorch* formats:
Model Name | Framework |
---|---|
colorization-v2 | Caffe* |
colorization-v2-norebal | Caffe |
googlenet-v1-tf | TensorFlow |
googlenet-v4-tf | TensorFlow |
gmcnn-places2-tf | TensorFlow |
i3d-rgb-tf | TensorFlow |
midasnet | PyTorch* |
resnet-50-tf | TensorFlow |
resnet-18-pytorch | PyTorch |
rfcn-resnet101-coco-tf | TensorFlow |
yolo-v1-tiny-tf | TensorFlow |
yolo-v2-tiny-tf | TensorFlow |
yolo-v2-tf | TensorFlow |
yolo-v3-tf | TensorFlow |
Deep Learning Streamer
Introducing the Deep Learning (DL) Streamer, now part of the default installation package of the Intel Distribution of OpenVINO toolkit. DL Streamer enables developers to create and deploy complex media analytics pipelines across Intel® architectures from edge to cloud. It provides optimal pipeline interoperability with familiar developer experience of the GStreamer multimedia framework. While enabling the developers to build a flexible streaming analytics pipeline, it also offers optimized inference across Intel architectures, from CPU and iGPU to Intel® Movidius™ Vision Processing Unit (VPU).
Key GStreamer elements provided by the DL Streamer:
Element | Purpose |
gvawatermark | To overlay the metadata on the video frame to visualize the inference results. |
gvatrack | To perform object tracking using zero-term or short-term tracking algorithms. Zero-term tracking assigns unique object IDs and requires object detection to run on every frame. Short-term tracking allows to track objects between frames, thereby reducing the need to run object detection on each frame. |
gvapython | To provide a callback to execute user-defined Python functions on every frame. Can be used for metadata conversion, inference post-processing, and other tasks. |
gvametapublish | To publish the JSON metadata to MQTT or Kafka message brokers or files. |
gvametaconvert | To convert the metadata structure to the JSON format. |
gvainference | To run deep learning inference using any model with an RGB or BGR input. |
gvafpscounter | To measure frames per second across multiple streams in a single process. |
gvadetect | To perform object detection using YoloV2-tiny, YoloV3, and SSD-like models. Outputs the region of interest (ROI) for detected objects. |
gvaclassify | To perform object classification. Accepts the ROI as input and outputs classification results with the ROI metadata. |
For more information, please refer to the DL Streamer tutorial, API reference, and samples documentation. at Intel Distribution of OpenVINO toolkit Samples. DL Streamer is an open source project located at GStreamer Video Analytics (GVA) Plugin repository on GitHub .DL Streamer includes Python and C++ APIs that provide access to inference results produced by the GStreamer elements for user applications written in Python and C++. The sample code for both C++ and Python is available in the samples folder installed by the Intel Distribution of OpenVINO toolkit installer.
New and Changed in the Release 1
Executive Summary
- Intel Distribution of OpenVINO Toolkit 2020.1 includes security bug fixes. Users should update to the latest version.
- Post-training Optimization to enable significant acceleration improvement. Little or no degradation in accuracy using a model quantization; while streamlining the development process with no model re-training nor fine-tuning required.
- Extends support for custom layers on the Intel® Movidius™ Vision Processing Unit (VPU). This enables greater flexibility and customization for dedicated edge, media and vision inference.
- Low-precision runtime for INT8. Including native support for “fake quantization” and support for 3D convolution layers along with reduced memory footprint. Enabling developers to add more deep learning models to their applications.
- Maximized performance optimizations and cross-platform flexibility with the Deep Learning Workbench. Simplify the development and deployment workflow, with support for the Post-training Optimization tool, COCO dataset, new algorithms and improvements in developer experience and interfaces. For more information, watch the webinar on Deep Learning Workbench.
- Support for new demos and models on Open Model Zoo, which includes:
- American Sign Language (ASL) recognition
- Character and text detection with Mask R-CNN
- 3D human pose estimation
- New speech libraries and speech recognition demos are included into distributions for Windows* and Ubuntu* 18.04 OSes. The new components enable and demonstrate end-to-end (speech to text) automatic speech recognition scenario.
- OpenVX* was removed from distribution, please see a separate Intel® Distribution of OpenVX* Implementation page.
- Switched to Linux Ubuntu* 18.04 kernel 5.3 by default due to performance drop on 9th Generation Intel® Core™ processors (formerly Coffee Lake) integrated graphics with kernel 5.0 compared to 5.3 and 4.15 (latency mode).
- Added support for AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1 is on Windows.
- Please, review this documentation page to Get a Deep Learning Model Performance Boost with Intel® Platforms.
Backward Incompatible Changes Compared with v.2019 R3.1
- Inference Engine folder for samples <install_root>\deployment_tools\inference_engine\samples\ now includes C, C++ and Python samples. Old path to the C++ samples was changed from <install_root>\deployment_tools\inference_engine\samples\ to <install_root>\deployment_tools\inference_engine\samples\cpp\.
- Old Calibration tool 1.0 that was introduced with 2019 R1 was removed from the package and replaced by a new one.
- CPU extensions library was removed. Extensions were moved into the plugin.
- List of Deprecated API, API Changes.
Model Optimizer
Common Changes
- Model Optimizer generates IR version 10 by default (except for the Kaldi* framework for which IR version 7 is generated) with significantly changed operations semantic. The command line --generate_deprecated_IR_V7 could be used to generate older version of IR. Refer to the documentation for the specification of a new operations set.
- --tensorflow_use_custom_operations_config has been renamed to --transformations_config. The old command line parameter is deprecated and will be removed in the future releases.
- Added ability to specify input data type using the --input command line parameter. For example, --input placeholder{i32}[1 300 300 3]. Refer to the documentation for more examples.
- Added support for FusedBatchNorm operation when mode=’training’
- Added optimisation transformation to remove useless Concat+Split sub-graphs.
- A number of graph transformations were moved from the Model Optimizer to the Inference Engine.
- Fixed networkX 2.4+ compatibility issues.
ONNX*
- Added support for the following ONNX* operations:
- Abs
- Acos
- Asin
- Atan
- Cast
- Ceil
- Cos
- Cosh
- Div
- Erf
- Floor
- HardSigmoid
- Log
- NonMaxSuppression
- OneHot
- ReduceMax
- ReduceProd
- Resize
- Sign
- Sin
- Sqrt
- Tan
- Xor
- Added support for the following ONNX models:
- ONNX model zoo:
- SSD ResNet
- Mask R-CNN: Refer to the documentation on how to convert the model.
- ONNX model zoo:
TensorFlow*
- Added optimization transformation which detects Mean Value Normalization pattern for 5D input and replaces it with a single MVN layer.
- Added ability to read TF 1.X models when TF 2.x is installed.
- Changed command line to convert GNMT model. Refer to the GNMT model conversion article for more information.
- Deprecated --tensorflow_subgraph_patterns, --tensorflow_operation_patterns command line parameters. The TensorFlow offload feature will be removed from the future releases.
- Added support for the following TensorFlow models:
- Wide and Deep family of models (CPU only)
- MobileNetV3
- Added support for the following TensorFlow operations:
- Bucketize (CPU only)
- Cast
- Cos
- Cosh
- ExperimentalSparseWeightedSum (CPU only)
- Log1p
- NonMaxSuppressionV3, NonMaxSuppressionV4, NonMaxSuppressionV5
- Sin
- Sinh
- SparseReshape (removed when input and output shapes are equal)
- SparseToDense (CPU only)
- Tan
- Tanh
MXNet*
- Added support for the following MXNet topologies:
- YoloV3 model from the GluonCV model zoo.
- MobileFace models (up to the NMS layer)
- Added support for the following MXNet operations:
- UpSampling with bilinear mode
- Where
- _arange
- _contrib_AdaptiveAvgPooling2D
- div_scalar
- elementwise_sub
- exp
- expand_dims
- greater_scalar
- minus_scalar
- repeat
- slice
- slice_like
- tile
Kaldi*
- The --remove_output_softmax command line parameter now triggers removal of final LogSoftmax layer in addition to a pure Softmax layer.
- Added support for the following Kaldi operations:
- linearcomponent
- logsoftmax
- Added support for the following Kaldi models:
- Aspire TDNN: Refer to the documentation on how to infer the model with use of a speech_sample.
- Librispeech nnet3
Inference Engine
Common Changes
- Added C API bridge library that allows you to integrate the Inference Engine runtime and plugins into applications not using and depending explicitly on C++.
- Added support of I420 blobs. Enable preprocessing and use class I420Blob to pass data for inference in I420 format. Inference Engine will convert data automatically to the BGR before inference.
- Added Remote Blob allowing to address memory on remote hardware to build pipelines not depending on host memory
- Enabled multi-socket support for Windows OS
- Re-worked INT8 run-time (fake quantized based int8 flow) so that it provides higher performance and wider topology coverage:
- Integrated new graph representation
- Implemented graph transformations
- Supported both symmetric and asymmetric quantization schemes
- Extended INT8 support to cover additional use cases: 3d-models, semantic segmentation, instance segmentation
- Identical functional coverage (features, models) for both CPU and GPU
- Implemented optimizations needed to get 1.2x performance improvements over approach previously used
CPU Plugin
- Re-designed (for CPU) fake quantized based int8 flow, which created also the foundation to scale to other int8 capable platforms.
- Introduced NUMA* support for the Windows, generally re-designed NUMA awareness (which now relies on the latest Threading Building Blocks (TBB) NUMA awareness feature).
- It is recommended to update all Intel products that use the TBB (like Intel® Distribution for Python*) to avoid possible TBB version conflicts when combining with Intel Distribution of OpenVINO toolkit.
GPU Plugin
- Implemented remote blob for OpenCL and Direct3D* 11 frameworks and support of memory been described in their terms on GPU
MYRIAD Plugin
- Extended support for VPU Custom Layers
HDDL Plugin
- Extended support for VPU Custom Layers
- Linux kernel versions 5.2 and below are supported.
GNA Plugin
- Added support for Tensor Iterator layer
- Added support for Squeeze layer
- Extended support for Permute Layer
- Fixed issue with LSTM Cell layer
- Improved accuracy for TDNN networks
FPGA Plugin
- Introduced offline compilation for FPGA, HETERO:FPGA devices using compile_tool. Compile tool produces binaries which can be imported via Core::Import to target device.
Model Optimizer and nGraph Integration Into Inference Engine
- Intel Distribution of OpenVINO toolkit is now powered by nGraph capabilities for Graph construction API, Graph transformation engine and Reshape, that replace former NN Builder API offering. Please refer to the new tutorials and updates for Model Optimizer and Inference Engine Developer's Guides updates on these new run-time capabilities. The NN Builder API is marked as deprecated and will be removed in the later releases.
- Additionally, we revised operations set used in internal Intermediate Representations and offer an extended set, which covers both former Intel Distribution of OpenVINO toolkit and nGraph operations as well as adding necessary semantics to further speedup models enabling from supported frameworks. The detailed opset specification and new versioning schema for IR-vs-operations are provided in the documentation.
- As result, Model Optimizer can now generate two types of IRs: a) IR V10 (default option) with this extended semantics and b) IR V7 (under --generate_deprecated_IR_V7 option in MO) which is just an evolution of IR V6 from previous release.
- We expect that re-converting the existing (and already supported) models into new semantics (IR V10) should be transparent for end users, if for any reason conversion or inference fail, try generating V7 IR (with option above), if this also does not help please report us about your issue.
Post-training Optimization Tool
The main goal of the tool:
- Transform Deep Learning model into a representation specific for selected type of optimization (e.g. quantization or sparsity) without model retraining
- After such transformation, the model can be inferred efficiently on hardware supporting such optimizations (e.g. execution in INT8 precision)
- The tool is extendable in order to support multiple quantization algorithms:
The tool supports quantization of Intel Distribution of OpenVINO toolkit Intermediate Representation (IR) models for various types of IA hardware:
- List of algorithms supported:
- Default algorithm that is essentially a pipeline running three base algorithms:
- MinMax
- Bias Correction (runs on top of naive algorithm, based on minimization of per-channel quantization error)
- Activation Channel Alignment (applied to align activation ranges)
- Accuracy Aware algorithm that preserves accuracy of the resulting model (keeps accuracy drop below threshold)
- Default algorithm that is essentially a pipeline running three base algorithms:
- Provides hardware-specific configs
- Features per-channel / per-tensor quantization granularity
- Supports symmetric / asymmetric quantization through presets mechanism
- Two default presets are introduced for MinMax, Default and Data free algorithms to simplify user experience
- 'Performance' preset stands for symmetric channel-wise (both for weights and activations) quantization
- 'Accuracy' preset stands for symmetric weights + asymmetric activations quantization
- Per-layer quantization tuning is also possible, some layers can be included into 'ignored scope' to skip quantization for those.
Tool is available at <install_root>\deployment_tools\tools\post_training_optimization_toolkit. After installation and running setup.py, the toolkit is available in the command line by the 'pot' alias.
Deep Learning Workbench
- Added support for Int8 Post-Training Optimization Tool. Two new algorithms (Max performance and Max accuracy) are currently exposed
- Added support for COCO dataset
- Multiple UX/UI improvements including new look and feel for per-layer table view
OpenCV*
- Version changed to 4.2.0.
- Added experimental support for nGraph API of Inference Engine (can be selected through cv::dnn::setInferenceEngineBackendType() call).
- Added Inference and Streaming extensions to G-API: now OpenCV G-API can handle full video analytics pipelines including video capture / decoding, image processing, CV, and DL scoring.
- New tutorials are available for these features: Face Analytics and Face Beautification pipelines.
Examples and Tutorials
- CPU extensions library was removed.
- Inference Engine folder for samples <install_root>\deployment_tools\inference_engine\samples\ now includes C, C++ and Python samples.
- Added end-to-end demos demonstrating speech recognition pipeline: offline demo for wave file transcription and GUI demo for live speech recognition.
- Added samples to demonstrate Inference Engine C-API usage
Tools
- Added feature extractor and decoder tools for speech processing pipelines. Added a tool for conversion of Kaldi* language models to the format used by the decoder tool.
Open Models Zoo
- Extended the Open Model Zoo, which includes additional CNN-pretrained models and pre-generated Intermediate Representations (.xml + .bin):
- [NEW] asl-recognition-0003
- [NEW] person-detection-asl-0001
- [NEW] person-reidentification-retail-0200
- [NEW] resnet18-xnor-binary-onnx-0001
- [NEW] text-spotting-0001-detector
- [NEW] text-spotting-0001-recognizer-decoder
- [NEW] text-spotting-0001-recognizer-encoder
- [NEW] person-reidentification-retail-0103. Replaces the 2019 R3 model.
- [NEW] person-reidentification-retail-0107. Replaces the 2019 R3 model.
- Added new demo applications:
- 3D human pose estimation
- interactive face detection
- multi-camera multi-person tracking
- multi-channel object detection with YOLO v3 support
- American sign language (ASL) recognition
- object detection with CenterNet
- single human pose estimation
- text spotting
Model Downloader tool configuration files are placed in separate per-model folders in order to improve user experience and simplify contribution process (less merge conflicts while developing/merging several models at the same time). The list is extended to support the following public models in Caffe2*, TensorFlow, MXNet, and PyTorch formats:
Model Name | Framework |
---|---|
efficientnet-b0 | TensorFlow |
efficientnet-b0-pytorch | PyTorch |
efficientnet-b0_auto_aug | TensorFlow |
efficientnet-b5 | TensorFlow |
efficientnet-b5-pytorch | PyTorch |
efficientnet-b7-pytorch | PyTorch |
efficientnet-b7_auto_aug | TensorFlow |
face-recognition-mobilefacenet-arcface | MXNet |
face-recognition-resnet100-arcface | MXNet |
face-recognition-resnet34-arcface | MXNet |
face-recognition-resnet50-arcface | MXNet |
densenet-121-caffe2 | Caffe2 |
resnet-50-caffe2 | Caffe2 |
squeezenet1.1-caffe2 | Caffe2 |
vgg19-caffe2 | Caffe2 |
ctdet_coco_dlav0_384 | ONNX |
ctdet_coco_dlav0_512 | ONNX |
brain-tumor-segmentation-0002 | ONNX |
human-pose-estimation-3d-0001 | PyTorch |
single-human-pose-estimation-0001 | PyTorch |
vehicle-license-plate-detection-barrier-0123 | TensorFlow |
Preview Features Terminology
A preview feature is functionality that is being introduced to gain early developer feedback. Comments, questions, and suggestions related to preview features are encouraged and should be submitted to the forum at Intel Distribution of OpenVINO Toolkit Forums.
The key properties of a preview feature are:
- It is intended to have a high quality implementation
- There is no guarantee of future existence or compatibility.
Note A preview feature is subject to change in the future. It may be removed or altered in future releases. Changes to a preview feature do NOT require the a deprecation and deletion process. Using a preview feature in a production code base is discouraged.
Fixed Issues
JIRA ID | Description | Component | |
26561 | Performance degradation for model 'semantic-segmentation-adas-0001' for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) | IE MyriadX plugin, IE HDDL plugin | |
26560 | Performance degradation for model 'deeplabv3' for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) | IE MyriadX plugin, IE HDDL plugin | |
26353 | Accuracy issues on a dataset for deeplab_v3 model for Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) | IE MyriadX plugin, IE HDDL plugin | |
26384 | Performance degradation for model 'refinedet-resnet-101' Intel Neural Compute Stick 2 (Intel NCS2) and Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) | IE MyriadX plugin, IE HDDL plugin | |
26700 | Intel Vision Accelerator Design with Intel Movidius Vision Processing Unit (VPU) hangs with Linux kernel 5.3 | IE HDDL plugin | |
24811 | [CPU-VNNI] Segmentation fault on license-plate-recognition-barrier-0007, efficientnet-b0, person-reidentification-retail-003 models. Benchmark app crashes with segmentation fault at the very end of execution. | IE CPU plugin | |
24517 | Specialize_function adds redundant ops for functions with the Split operation | nGraph Core | |
25417 | Incorrect OL and OH params for the FQ layer | MO TensorFlow | |
23985 | Incorrect command-line parameters specified in the documentation to convert the LM1B model | MO TensorFlow | |
25657 | Memory leaks are possible in the GPU plugin in case of multiple network loading/unloading cycles | IE GPU Plugin | |
25087 | Performance degradations are possible in the GPU plugin on MobileNet-like models | IE GPU Plugin | |
25020 | May produce wrong output data in the case when several topologies are loaded into one CPU plugin. The issue requires several conditions:
|
IE CPU Plugin | |
25940 |
|
All | |
23705 |
|
IE GNA Plugin | |
26585 | Speech sample may display incorrect results of comparison with reference scores for models including LSTM cells | Samples | |
24523 | Low performance of asl-recognition-xxxx model on CPU | IE CPU Plugin | |
23198 | Inference of OneHot layer in constant subgraphs may be incorrect. As a result, GNMT topology may produce incorrect output. | All | |
22108 | Stopping the app during firmware boot might cause device hang for Intel® Neural Compute Stick 2 | IE MyriadX plugin | |
23738 | Core::SetConfig and Core::GetConfig methods work incorrectly with the GNA plugin. | IE GNA plugin | |
29082 | OpenVINO has thread-safety issues. LoadNetwork and other ie::Core methods simultaneously called from different threads may lead to error or crashes. | All | |
28747 | CPU plugin does not work on Windows system with CPUs less then AVX2 instruction set (Intel Atom® processors) | IE CPU Plugin | |
28970 | TF faster-rcnn and faster-resnet101 topologies accuracy deviation on MYRIAD | IE MyriadX plugin, IE HDDL plugin | |
29361 | Performance degradation for model 'text-recognition-0012' and ctpn | IE MyriadX plugin, IE HDDL plugin | |
28683 | Changing system time on Linux might break watchdog and interrupt the inference | IE HDDL plugin | |
25723 | TF rfcn_resnet101_coco low accuracy on dataset |
|
Known Issues
JIRA ID | Description | Component | Workaround |
---|---|---|---|
#1 | A number of issues were not addressed yet, please see "Known issue" section in the Release Notes for Intel® Distribution of OpenVINO™ toolkit 2019 | All | N/A |
21670 | FC layers with bimodal weights distribution are not quantized accurately by the Intel® GNA Plugin when 8-bit quantization is specified. Weights with values near zero are set to zero. | IE GNA plugin | Use 16-bit weights in these cases. |
25358 | Some performance degradations are possible in the GPU plugin on GT3e/GT4e/ICL NUC platforms | IE GPU plugin | N/A |
24709 | Retrained TensorFlow Object Detection API RFCN model has significant accuracy degradation. Only the pretrained model produces correct inference results. | All | Use Faster-RCNN models instead of RFCN model if re-training of a model is required. |
26129 | TF YOLO-v3 model fails on AI Edge Computing Board with Intel® Movidius™ Myriad™ X C0 VPU, MYDX x 1 | IE MyriadX plugin |
Use other versions of the YOLO network or USB connected device (Neural Compute Stick 2) |
nGraph Python API has been removed from the current release due to its incompetence. It will be added back when it is ready with the desired quality. | IE Python API |
Use C++ API |
|
26388 | Low latency (batch size 1) graphs with LSTMCell do not infer properly due to missing state handling. | All | Use deprecated IRv7 and manually insert memory layers into the IR graph. Alternatively, add state tensors as extra input and output nodes and associate their blobs given the IR node IDs after loading the graph. |
24101 | Performance and memory consumption may be bad if layers are not 64 bytes aligned | IE GNA plugin | Try to avoid the layers that are not 64 bytes aligned to make the model GNA friendly |
28006 | The result is incorrect for the concat layer having 3 or more input layers. | IE GNA plugin | Refactor the model so it would not have this layer |
34350 | When an LSTMCell is the output layer of a graph, the output is all zeros. | IE GNA plugin | Manually insert an additional non-functional layer (such as Identity activation) at the end of the IR graph. |
31164 | HDDL Xlink fail to connect more than 32 MYX devices | IE HDDL plugin | Use up to 4 Intel® Vision Accelerator Design with Intel® Movidius™ VPUs |
33146 | Some models fails with batch > 1 on Intel® Neural Compute Stick 2 and Intel® Vision Accelerator Design with Intel® Movidius™ VPUs | IE MyriadX plugin, IE HDDL plugin | Use batch 1 or use other HW (i.e. CPU/GPU) |
32934 | Models modified with deprecated CNNNetwork API may fail to load to Myriad/HDDL plugins | IE MyriadX plugin, IE HDDL plugin | Use IRv10 versions of the models |
33881 | Performance regression on se-resnext and efficientnet models | IE MyriadX plugin, IE HDDL plugin | Use OpenVINO 2020.3 LTS release version |
33810 | On Linux, "can't protect" exception may happen when loading large models with large number of CPU streams to the device | IE MKL-DNN plugin | Add the following line to /etc/sysctl.conf: vm.max_map_count=2097152 Then reload the config with sysctl -p |
Included in this Release
The Intel® Distribution of OpenVINO™ toolkit is available in the following versions:
- Intel Distribution of OpenVINO toolkit for Windows
- Intel Distribution of OpenVINO toolkit for Linux
- Intel Distribution of OpenVINO toolkit for Linux with FPGA Support
- Intel Distribution of OpenVINO toolkit for macOS*
Component | License | Location | Windows* | Linux* | Linux for FPGA | macOS* |
---|---|---|---|---|---|---|
Deep Learning Model Optimizer Model optimization tool for your trained models | Apache 2.0 | <install_root>/deployment_tools/model_optimizer/* | YES | YES | YES | YES |
Deep Learning Inference Engine Unified API to integrate the inference with application logic Inference Engine Headers | EULA Apache 2.0 |
<install_root>/deployment_tools/inference_engine/*
<install_root>/deployment_tools/inference_engine/include/* |
YES | YES | YES | YES |
OpenCV library OpenCV Community version compiled for Intel hardware |
BSD | <install_root>/opencv/* | YES | YES | YES | YES |
Intel® Media SDK libraries Eases the integration |
MIT | <install_root>/../mediasdk/* | NO | YES | YES | NO |
Intel® Graphics Compute Runtime for OpenCL Improves usability | MIT |
<install_root>/install_dependencies/ install_NEO_OCL_driver.sh - helps to install OpenCL Runtime, default location /usr/local/lib/ intel-opencl_*.deb - driver for Ubuntu* intel-opencl_*.rpm - driver for CentOS* intel-* - driver's dependencies |
NO | YES | YES | NO |
Intel® FPGA Deep Learning Acceleration Suite (Intel® FPGA DL Acceleration Suite), including pre-compiled bitstreams Implementations of the most common CNN topologies to enable image classification and ease the adoption of FPGAs for AI developers. Includes pre-compiled bitstream samples for the Intel Programmable Acceleration Card with Intel Arria 10 GX FPGA and Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and speed grade 2 |
Intel OBL FPGA SDK 18.1 |
<install_root>/bitstreams/a10_dcp_bitstreams/* <install_root>/bitstreams/a10_vision_design_sg1_bitstreams/* <install_root>/bitstreams/a10_vision_design_sg2_bitstreams/* |
NO | NO | YES | NO |
Intel® FPGA SDK for OpenCL™ software technology The Intel FPGA RTE for OpenCL provides utilities, host runtime libraries, drivers, and RTE-specific libraries and files | Intel OBL FPGA SDK 18.1 |
/opt/altera/aocl-pro-rte/* <user>/intelFPGA_pro/<version>/* |
NO | NO | YES | NO |
Intel Distribution for OpenVINO toolkit documentation Developer guides and other documentation | Available from the OpenVINO™ toolkit product site, not part of the installer packages. |
NO | NO | NO | NO | |
Open Model Zoo Documentation for models; Models in binary form can be downloaded using Model Downloader | Apache 2.0 | <install_root>/deployment_tools/open_model_zoo/* | YES | YES | YES | YES |
Inference Engine Samples Samples that illustrate Inference Engine API usage and demos that demonstrate how you can use features of Intel® Distribution of OpenVINO™ toolkit in your application | Apache 2.0 | <install_root>/deployment_tools/ inference_engine/samples/* |
YES | YES | YES | YES |
Deep Learning Workbench Tool that can help developers to run Deep Learning models through the toolkit's Model Optimizer, convert it to INT8, fine tune it, run inference and measure accuracy. | EULA | <install_root>/deployment_tools/tools/workbench/* | YES | YES | NO | YES |
nGraph Open source C++ library, compiler and runtime for Deep Learning nGraph |
Apache 2.0 | <install_root>/deployment_tools/ngraph/* | YES | YES | YES | NO |
Post-Training Optimization Toolkit designed to convert a model into a more hardware-friendly representation by applying specific methods that do not require re-training, for example, post-training quantization. | EULA | <install_root>/deployment_tools/ tools/post_training_optimization_toolkit/* |
YES | YES | YES | YES |
Speech Libraries and End-to-End Speech Demos | GNA Software License Agreement | <install_root>/data_processing/audio/speech_recognition/* | YES | YES | YES | NO |
DL Streamer | EULA | <install_root>/data_processing/dl_streamer/* | NO | YES | NO | NO |
Where to Download this Release
System Requirements
Intel® CPU Processors
Hardware
- Intel Atom® processor with Intel SSE4.1 support
- Intel® Pentium® processor N4200/5, N3350/5, N3450/5 with Intel® HD Graphics
- 6th - 10th generation Intel® Core™ processors
- Intel® Xeon® processor E3, E5, and E7 family (formerly Sandy Bridge, Ivy Bridge, Haswell, and Broadwell)
- Intel® Xeon® Scalable processor (formerly Skylake and Cascade Lake)
Operating Systems
- Ubuntu 18.04 long-term support (LTS), 64-bit
- Ubuntu 16.04 long-term support (LTS), 64-bit
- Windows® 10, 64-bit
- macOS 10.14, 64-bit
Intel® Processor Graphics
Hardware
- Intel HD Graphics
- Intel® UHD Graphics
- Intel® Iris® Pro Graphics
Note This installation requires drivers that are not included in the Intel Distribution of OpenVINO toolkit package.
Note A chipset that supports processor graphics is required for Intel®Xeon processors. Processor graphics are not included in all processors. See Product Specifications for information about your processor.
Operating Systems
- Ubuntu 18.04 long-term support (LTS), 64-bit
- Windows 10, 64-bit
- Yocto* 3.0, 64-bit
Intel® Gaussian & Neural Accelerator (Intel® GNA)
Operating Systems
- Ubuntu 18.04 long-term support (LTS), 64-bit
- Windows 10, 64-bit
Intel® FPGA Processors
Note Only for the Intel Distribution of OpenVINO™ toolkit for Linux with FPGA Support
Hardware
- Intel Vision Accelerator Design with an Intel Arria 10 FPGA (Mustang-F100-A10) speed grade 1 and speed grade 2
- Intel Programmable Acceleration Card (Intel PAC) with Intel Arria 10 GX FPGA (Intel PAC with Intel Arria 10 GX FPGA)
Operating Systems
- Ubuntu 18.04 long-term support (LTS), 64-bit
Intel® VPU Processors
Hardware with corresponding operating systems
- Intel Vision Accelerator Design with Intel Movidius Vision Processing Units (VPUs):
- Ubuntu 18.04 long-term support (LTS), 64-bit (Linux Kernel 5.2 and lower)
- Windows 10, 64-bit
- CentOS 7.4, 64-bit
- Intel Movidius Neural Compute Stick (NCS) and Intel Neural Compute Stick 2 (Intel NCS2):
- Ubuntu 18.04 long-term support (LTS), 64-bit
- CentOS 7.4, 64-bit
- Windows 10, 64-bit
- Raspbian* (target only)
- AI Edge Computing Board with Intel Movidius Myriad X C0 VPU, MYDX x 1:
- Windows 10, 64-bit
Components Used In Validation
Operating systems:
- Linux* OS
- Ubuntu 16.04.6 with Linux kernel 4.15
- Ubuntu 18.04.3 with Linux kernel 5.3
- CentOS 7.4 with Linux kernel 5.3
- Raspbian 9
- A Linux* OS build environment needs these components:
- GNU Compiler Collection (GCC)* 4.8, 5.4, 7.5
- CMake* 2.8.12 or higher
- Python* 3.5-3.7
- OpenCV 4.4
- Intel(R) Graphics Compute Runtime (required only for GPU) - 19.41.14441 (19.04.12237 for Ubuntu 16)
- Windows 10 version 1809 (known as Redstone 5)
- A Windows* OS build environment needs these components:
- Microsoft Visual Studio* 2015
- Intel® C++ Compiler 2017 Update 4
- CMake 2.8.12 or higher
- Python* 3.5-3.7
- OpenCV 4.4
- Intel® HD Graphics Driver (required only for GPU)
- A Windows* OS build environment needs these components:
- macOS 10.14
- A macOS build environment needs these components:
- Xcode* 10.3
- CMake 3.9 or higher
- Python 3.5-3.7
- OpenCV 4.4
- A macOS build environment needs these components:
- Raspbian 9
DL frameworks:
- TensorFlow 1.14.0, 1.15.2, 2.2.0 (limited support according to product features)
- MxNet 1.5.1
Helpful Links
- Intel Distribution of OpenVINO toolkit Home Page
- Intel Distribution of OpenVINO toolkit All Documentation, Guides, and Resources
- Intel Distribution of OpenVINO™ toolkit Community Forum
Legal Information
You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.
The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at Intel® or from the OEM or retailer.
No computer system can be absolutely secure.
Intel, Arria, Core, Movidius, Xeon, OpenVINO, and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.
OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos
*Other names and brands may be claimed as the property of others.
Copyright © 2020, Intel Corporation. All rights reserved.