# OpenVINO™ Toolkit Unreal* Integration

Published: 12/02/2019

Last Updated: 12/02/2019

## Overview

This project was created to show how the OpenVINO™ toolkit library can be incorporated into an Unreal* project. This document discusses the Unreal Engine* project being published and the software architecture behind it.

This project is about integration with a simple, easy to follow project that allows developers to extend this architecture to suit their own needs.

## Who This Project is For

This project is for people who have at minimum a basic understanding of the Unreal game engine and skills with Microsoft Visual Studio*, C++, and CMake*.

Further, it is for software developers who would like to incorporate the Intel® Distribution of OpenVINO™ toolkit into their Unreal projects.

## What You Will Need

### Intel® Distribution of OpenVINO™ Toolkit

It is expected that the reader has familiarity with the toolkit; this document will not teach how to use the OpenVINO toolkit, but rather will show how it can be incorporated into the Unreal game engine. Once the toolkit has been downloaded, installed, and configured, ensure that you can run the samples provided with the OpenVINO toolkit.

## Solution Architecture

The project consists of an Unreal Engine application based on the first person shooter template. Using blueprints, the OpenVINO toolkit is accessed via an Unreal plugin architecture.

Figure 1. High level architectural interaction

### Two Microsoft Visual Studio* Solutions

This architecture consists of two different Visual Studio solutions.

#### OpenVinoWrapper DLL

This solution contains only the wrapper DLL that encapsulates the C-style calls needed to call into the OpenVINO toolkit API. The solution is generated using CMake.

#### OVClassification

This solution gets generated by right clicking the Unreal Engine OVClassification project file. It consumes the OpenVinoWrapper DLL and contains the code for the OpenVINO toolkit plugin.

### OpenVINO™ Toolkit Plugin

We generated this as a plugin to allow for easier integration with other projects. If you want to use the same OpenVINO toolkit features in your other projects you can reuse this plugin. This plugin is used in the level using the Unreal Engine blueprints.

The plugin itself consists of two components:

• A third-party component called OpenVinoWrapper that uses the CMake build system (recommended by the OpenVINO toolkit documentation) to produce a .DLL that uses OpenVINO toolkit APIs and exposes them as C-style APIs to the OpenVinoModule.
• The OpenVinoModule, which exposes OpenVinoWrapper to Unreal Engine’s blueprints (and C++ APIs, if needed).

Starting from the bottom, working up the various aspects of the architecture are discussed in the following sections.

#### Intel Distribution of OpenVINO Toolkit

At the very bottom of the call stack is the OpenVINO toolkit. An in-depth discussion of this toolkit is beyond the scope of this document. If you are new to the toolkit it is recommended that you spend some time ramping up on the technology by reviewing the existing documentation.

#### OpenVinoWrapper DLL

Figure 2. OpenVinoWrapper DLL Microsoft Visual Studio* solution

OpenVinoWrapper is a DLL that wraps functionality of the OpenVINO toolkit.

As the OpenVINO toolkit API uses C++ and the standard template library in its interfaces, memory management between this library and Unreal Engine can cause problems. The simple solution is exposing these APIs as C-style APIs.

The library itself uses CMake (CMakeLists.txt file) to create cross-platform build scripts (in this case, Windows* only).

Not only does CMake build the OpenVinoWrapper.dll, but it also copies locally all files from the OpenVINO toolkit installation so that they can later be packaged with the Unreal game or application. This includes:

• OpenVINO toolkit inference_engine binary redistributable files
• OpenCV binary redistributable files
• OpenVX* binary redistributable files
• Intel® C++ Compiler binary redistributable DLL (Short Vector Math Library (SVML) library)

The OpenVinoWrapper project itself consists of several classes:

• OpenVinoWrapper.h/.cpp—The OpenVinoWrapper class is to be included by the OpenVinoModule. It contains the definitions of the exported functions from the DLL as well as the implementation of the exported functions. The functions that have been exported and callable by the Unreal OpenVinoPlugin are as follows:
	DLLEXPORT bool OpenVino_Initialize(
const char* modelXmlFilePath,
const char* modelBinFilePath,
const char* modelLabelFilePath,
void** pHandle );

DLLEXPORT bool OpenVino_Infer_FromPath(
void* handle,
char* filePath,
char* result,
size_t maxResultSize );

DLLEXPORT bool OpenVino_GetLastError(
char* lastErrorMessage,
size_t maxLength );


It should be noted that the actual inference is not handled by this class; rather, the inference is delegated to the OpenVinoData class.

• OpenVinoData.h/.cpp—This class implements actual calls to the OpenVINO toolkit C++ APIs. The Initialization call will create an instance of this class and pass the handle back to the client (OpenVinoWrapper). Calls to Infer are then delegated to this class.
• ClassificationResult.h—This class formats returned results from binary data returned by the OpenVINO toolkit C++ APIs to a single, multiline string.

#### OpenVinoModule

Figure 3. OVClassification OpenVinoModule source code files

The OpenVinoModule creates an entry point into the plugin that the blueprints can call. In turn, OpenVinoModule will call into the OpenVinoWrapper to interface with the OpenVINO toolkit library.

The module itself consists of:

• OpenVinoClassification—This class is a blueprint interface as an Actor component that can be added to any Unreal Engine’s Actor and called from blueprint.
• FOpenVinoModule—This is Unreal Engine’s Module class that controls loading and unloading of the OpenVinoWrapper DLL and delegates basic calls to the DLL.

### Unreal* Project

This project is based on the Unreal Engine’s first person shooter (blueprints) starter content and contains three main components: FirstPersonCharacter blueprint, Billboard blueprint, and customized first person heads-up display (HUD) and special content folders.

The project (and map) have customized world settings: GameMode, HUD, and Default Pawn Class.

#### Special Content Folders

This project contains two special content folders which, when viewed from Unreal Engine’s Content Browser have no content, but when viewed from Windows Explorer they do, as shown:

Figure 4. Special content folders

• Images, which host images to be analyzed by sending them to the OpenVINO toolkit engine.
• OpenVinoModels, which host the OpenVINO intermediate representation data files we will need to initialize the OpenVINO toolkit.

Both folders are added in Project Settings, Packaging, and Additional Non-Asset Directories to Copy, in order to be able to reference them directly from the file system when built and packaged, as shown:

Figure 5. Showing special content folders in the project properties

These folders and their content will be copied when building and packaging projects, and will be accessible directly from the file system.

#### First Person Character

Figure 6. Modified First Person blueprint

Located in FirstPersonBP/Blueprints/FirstPersonCharacter. All the default functionality was left intact; however, additions were made to the blueprint. In the components section an instance of the OpenVinoClassification C++ class was added. Down in the Variables section, a LastStatus public string variable was added to display results from the classification functionality.

Figure 7. First Person Character blueprint added component and variable

• OpenVinoClassification—Actor component for initialization of the plugin and responding to OpenVINO toolkit plugin classification complete events.
• LastStatus—String variable that will contain results from the OpenVINO toolkit plugin that get displayed in the HUD.

#### OpenVINO Toolkit Initialization

The Event Graph is where the OpenVinoPlugin that will be initialized resides. When the Event BeginPlay triggers, the OpenVinoClassification object’s Initialize function is called. After Initialization, the Last Status string variable is populated with any error or success messages and is printed to the OpenVinoHUDWidget.

Figure 8. Initializing the OpenVINO™ toolkit plugin, Part 1

As can be seen in Figure 8, part of the initialization is to get the paths to the required OpenVINO toolkit intermediate representation files. Once created they are fed back into the Initialize function.

Figure 9. Initializing the OpenVINO™ toolkit plugin, Part 2

#### Handle OpenVINO Toolkit Classification Complete

Once the OpenVINO toolkit classification is complete, the On Classification Complete event is triggered, which passes the classification result formatted as a single, multiline string value. This value is then stored in the LastStatus variable, and printed (in non-shipping versions) to screen and written to the log:

Figure 10. Handling Classification Complete message

#### Billboard_BP

Located in Intel/Blueprints/Billboard_BP, this blueprint was created to handle all the different images to classify. This blueprint is responsible for kicking off an OpenVINO toolkit classification functionality. This is triggered when the user aims the crosshairs at an image and clicks the left mouse button firing a projectile at the billboard triggering an OnComponentHit event.

Each billboard contains one image/texture of a pet to classify as well as a text render component to display the image’s classification name for comparison against the results.

Figure 11. Showing a Billboard blueprint with pet textures applied

Figure 12. Pet Billboard Components and added Variables.

The Billboard_BP is made up of the following:

• Static Mesh—The dynamic material will use this to place itself.
• SpeciesName—Text field to display the PetSpeciesName variable that gets set in the details pane for the blueprint.

Three variables were added to this blueprint:

• PetTexture—Specifies which texture to use from Content/Intel/Textures.
• PetImageFilename—Specifies which .JPG image to use when supplying an image to the classification function. Building of the absolute path is done in the Event Graph of the blueprint.
• PetSpeciesName—String to be displayed below the image. This is for visual representation only and helps the user identify the animal species by name.

#### Construct the Billboard

The construction script creates dynamic material to display the image onto the Static Mesh component. It uses the one and only TextureMaterial. The TextureMaterial is populated by the variable PetTexture, which is a Texture object. It will then get the Pet Species Name variable, which is used to populate the SpeciesName Text Render component, which is displayed under the pet texture.

Figure 13. Pet Billboard Construction script

#### Start OpenVINO Toolkit Classification

When a projectile hits a billboard, the event graph starts the classification process by getting the OpenVinoClassification from the FirstPersonCharacter blueprint. Recall that the FirstPersonCharacter contains the actualOpenVinoClassification module object that knows how to make calls into the OpenVINO toolkit. Once the classification object has been obtained, the event graph creates the absolute path to the specified .JPG file associated to the blueprint. This path is then fed into the Begin Classify Image From Path function.

Figure 14. Pet Billboard Event graph

#### First Person HUD Customization

##### First Person HUD

The first person HUD has been customized to add the HUD Widget blueprint. On the Begin Play event, we are creating a HUD Widget and adding it to the Viewport:

Figure 15. OpenVINO™ toolkit HUD

##### OpenVINO Toolkit HUD Widget

This widget is basically one Text Block stretched over the entire screen:

Figure 16. OpenVINO™ toolkit HUD widget

In the Details tab you will notice that Text is bound to the GetText_0 function, as shown:

Figure 17. Text Block populated by GetText_0 function

##### GetText_0 Function

Selecting Graph in the upper right corner of the BP screen brings up the GetText_0 function. This blueprint function is reading the Last Status result from First Person Character blueprint and returning it to the text box for display:

Figure 18. OpenVINO HUD Widget GetText_0 function

Note that this variable (Last Status) is set to:

• Result of initialization, after OpenVINO toolkit initialization.
• Result of classification, after classification process has completed.

## Get OVClassification Built and Running

The project does not come precompiled and set up. This section guides you through the steps of compiling the OpenVinoWrapper DLL and generating the OVClassification solution files so that you can view the OpenVINO toolkit plugin source code.

Figure 19. Project directory listing

Figure 19 shows the contents of the zip file after it is extracted to your hard drive. It contains the OVClassification Unreal Engine project with the blueprint textures that we covered previously.

### Build the OpenVinoWrapper DLL

Note that prior to building the OpenVinoWrapper DLL you must have completed the installation of the Intel Distribution of OpenVINO toolkit, version 2019.2.242.

Step 1. Open the developer command prompt

Figure 20. Microsoft Visual Studio* menu

Step 2. Run Setupvars.bat

Figure 21. Command to set up OpenVINO™ toolkit variables

Navigate to your OpenVINO toolkit installation location and run setupvars.bat. Running setupvars.bat is required because it sets environment variables that CMake uses to build the Visual Studio solution. Without running setupvars.bat, CMake will fail.

Step 3. Run CMake on the OpenVinoWrapper directory

Navigate to the location where you unzipped the OVClassification project, and then navigate to the directory shown in Figure 22 and run CMake.

Figure 22. Run CMake to build OpenVinoWrapper DLL Microsoft Visual Studio* project

After CMake generates the files you will see a folder structure that now contains the Visual Studio solution files as well as all the supporting OpenVINO toolkit files needed for distribution.

Figure 23. Directory listing after running CMake

Step 4. Compile the OpenVinoWrapper DLL

Now that the Visual Studio solution has been generated, open the solution and build it. Doing so builds the DLL and places it in a location where the OVClassification Visual Studio project solution will find and consume it.

Figure 24. OpenVinoWrapper DLL Microsoft Visual Studio* solution

### Compile the OVClassification Visual Studio Solution

Now that the OpenVinoWrapper.DLL has been created, we can move on to generating the Visual Studio solution files for the main project.

Step 1. Create the OVClassification Visual Studio solution files

Right-click OVClassification to bring up the context menu, then select Generate Visual Studio project files.

Figure 25. Generate OVClassification Microsoft Visual Studio* solution

Step 2. Open the OVClassification Visual Studio solution and compile the app

Figure 26. OVClassification Microsoft Visual Studio* solution

At this point, all the source code should be compiled, allowing you to open the OVClassification Unreal Engine project and run it.

## Open OVClassification Unreal Engine* Project

When you open the project you should see something similar to what is shown in Figure 27. The main scene will have several rows of billboards already set up with images of animals configured on them. At this point the application should be ready to run.

Figure 27. OVClassification Unreal Engine* project

### Running the App

Running the app puts you in first person mode with the default gun and projectile. The classification takes place when a user shoots a billboard. Once the classification is complete, the results are displayed in the upper left corner of the screen.

Figure 28. Showing OVClassification Classification results