Batch inferencing using OpenVINO™ toolkit
Unable to run inference for two input images in OpenVINO™ toolkit.
Refer to Using Shape Inference and note the limitations of using batches.
The Open Model Zoo smart_classroom_demo shows dynamic batching when processing multiple previously detected faces. Refer to the function CnnDLSDKBase::InferBatch in the demo, which is located at smart_classroom_demo/cpp/src/cnn.cpp, line 51.
- When batch is enabled in the model, the memory buffer of the input blob will be allocated to have room for all batches of images, and the data in the input blob for each image needs to be filled.
- In the loop over num_imgs, an auxiliary function matU8ToBlob fills the input blob with data for current_batch_size of images, and then sets batch size for infer request and run inference.
for (size_t batch_i = 0; batch_i < num_imgs; batch_i += batch_size) {
const size_t current_batch_size = std::min(batch_size, num_imgs - batch_i);
for (size_t b = 0; b < current_batch_size; b++) {
matU8ToBlob<uint8_t>(frames[batch_i + b], input, b);
}
if (config_.max_batch_size != 1)
infer_request_.SetBatch(current_batch_size); infer_request_.Infer();