OpenVINO™ Inference Time Increases When Running Multiple Processes

The inferencing time doubles when running two processes to infer the same model.

• Using OpenVINO™ to infer a model.
• Inference time is about 300ms when running a single process.
• When using two processes, the inference time for both processes becomes 600ms.

1. Add the following configuration in your application.
   Python*: core.set_property("CPU", {ov.properties.hint.enable_cpu_pinning(False)})
   C++ : core.set_property("CPU", ov::hint::enable_cpu_pinning(false));
2. Build the demo.

• The default value of enable_cpu_pinning property is enabled (true or YES) on Linux* and disabled (false or NO) on Windows* and macOS*.
• On Linux, the default setting of enable_cpu_pinning is enabled, which in some scenarios can cause multiple processes to bind to the same CPU core, potentially doubling inference time.
• Setting the property to false unbinds inferencing threads from specific CPU cores, allowing the operating system to schedule threads, which may be beneficial in applications with several parallel workloads.

Refer to Performance Hints and Thread Scheduling for more information.