Article ID: 000059815 Content Type: Product Information & Documentation Last Reviewed: 06/06/2023

How can Two Images be Inferenced in the Same OpenVINO™ toolkit Batch?

BUILT IN - ARTICLE INTRO SECOND COMPONENT
Summary

Batch inferencing using OpenVINO™ toolkit

Description

Unable to run inference for two input images in OpenVINO™ toolkit.

Resolution

Refer to Using Shape Inference and note the limitations of using batches.

The Open Model Zoo smart_classroom_demo shows dynamic batching when processing multiple previously detected faces. Refer to the function CnnDLSDKBase::InferBatch in the demo, which is located at smart_classroom_demo/cpp/src/cnn.cpp, line 51.

  • When batch is enabled in the model, the memory buffer of the input blob will be allocated to have room for all batches of images, and the data in the input blob for each image needs to be filled.
  • In the loop over num_imgs, an auxiliary function matU8ToBlob fills the input blob with data for current_batch_size of images, and then sets batch size for infer request and run inference.

for (size_t batch_i = 0; batch_i < num_imgs; batch_i += batch_size) {
const size_t current_batch_size = std::min(batch_size, num_imgs - batch_i);
for (size_t b = 0; b < current_batch_size; b++) {
matU8ToBlob<uint8_t>(frames[batch_i + b], input, b);

}

if (config_.max_batch_size != 1)
infer_request_.SetBatch(current_batch_size); infer_request_.Infer();

Additional information

Using Dynamic Batching

Related Products

This article applies to 2 products