This project was created to show how the OpenVINO™ toolkit library can be incorporated into an Unreal* project. This document discusses the Unreal Engine* project being published and the software architecture behind it.
This project is about integration with a simple, easy to follow project that allows developers to extend this architecture to suit their own needs.
Who This Project is For
This project is for people who have at minimum a basic understanding of the Unreal game engine and skills with Microsoft Visual Studio*, C++, and CMake*.
Further, it is for software developers who would like to incorporate the Intel® Distribution of OpenVINO™ toolkit into their Unreal projects.
What You Will Need
Intel® Distribution of OpenVINO™ Toolkit
It is expected that the reader has familiarity with the toolkit; this document will not teach how to use the OpenVINO toolkit, but rather will show how it can be incorporated into the Unreal game engine. Once the toolkit has been downloaded, installed, and configured, ensure that you can run the samples provided with the OpenVINO toolkit.
- Download the Code sample of this Project
- Microsoft Visual Studio 2019 with C++ game development support
- Unreal Engine 4.22
- UnrealVS Extension for Microsoft Visual Studio
- Intel® Distribution of OpenVINO™ toolkit for Windows® 10
- Version 2019.2.242 was used for this project
- CMake 3.14 or higher
The project consists of an Unreal Engine application based on the first person shooter template. Using blueprints, the OpenVINO toolkit is accessed via an Unreal plugin architecture.
Figure 1. High level architectural interaction
Two Microsoft Visual Studio* Solutions
This architecture consists of two different Visual Studio solutions.
This solution contains only the wrapper DLL that encapsulates the C-style calls needed to call into the OpenVINO toolkit API. The solution is generated using CMake.
This solution gets generated by right clicking the Unreal Engine OVClassification project file. It consumes the OpenVinoWrapper DLL and contains the code for the OpenVINO toolkit plugin.
OpenVINO™ Toolkit Plugin
We generated this as a plugin to allow for easier integration with other projects. If you want to use the same OpenVINO toolkit features in your other projects you can reuse this plugin. This plugin is used in the level using the Unreal Engine blueprints.
The plugin itself consists of two components:
- A third-party component called OpenVinoWrapper that uses the CMake build system (recommended by the OpenVINO toolkit documentation) to produce a .DLL that uses OpenVINO toolkit APIs and exposes them as C-style APIs to the OpenVinoModule.
- The OpenVinoModule, which exposes OpenVinoWrapper to Unreal Engine’s blueprints (and C++ APIs, if needed).
Starting from the bottom, working up the various aspects of the architecture are discussed in the following sections.
Intel Distribution of OpenVINO Toolkit
At the very bottom of the call stack is the OpenVINO toolkit. An in-depth discussion of this toolkit is beyond the scope of this document. If you are new to the toolkit it is recommended that you spend some time ramping up on the technology by reviewing the existing documentation.
Figure 2. OpenVinoWrapper DLL Microsoft Visual Studio* solution
OpenVinoWrapper is a DLL that wraps functionality of the OpenVINO toolkit.
As the OpenVINO toolkit API uses C++ and the standard template library in its interfaces, memory management between this library and Unreal Engine can cause problems. The simple solution is exposing these APIs as C-style APIs.
The library itself uses CMake (CMakeLists.txt file) to create cross-platform build scripts (in this case, Windows* only).
Not only does CMake build the OpenVinoWrapper.dll, but it also copies locally all files from the OpenVINO toolkit installation so that they can later be packaged with the Unreal game or application. This includes:
- OpenVINO toolkit inference_engine binary redistributable files
- OpenCV binary redistributable files
- OpenVX* binary redistributable files
- Intel® C++ Compiler binary redistributable DLL (Short Vector Math Library (SVML) library)
The OpenVinoWrapper project itself consists of several classes:
- OpenVinoWrapper.h/.cpp—The OpenVinoWrapper class is to be included by the OpenVinoModule. It contains the definitions of the exported functions from the DLL as well as the implementation of the exported functions. The functions that have been exported and callable by the Unreal OpenVinoPlugin are as follows:
DLLEXPORT bool OpenVino_Initialize( const char* modelXmlFilePath, const char* modelBinFilePath, const char* modelLabelFilePath, void** pHandle ); DLLEXPORT bool OpenVino_Infer_FromPath( void* handle, char* filePath, char* result, size_t maxResultSize ); DLLEXPORT bool OpenVino_GetLastError( char* lastErrorMessage, size_t maxLength );
It should be noted that the actual inference is not handled by this class; rather, the inference is delegated to the OpenVinoData class.
- OpenVinoData.h/.cpp—This class implements actual calls to the OpenVINO toolkit C++ APIs. The Initialization call will create an instance of this class and pass the handle back to the client (OpenVinoWrapper). Calls to Infer are then delegated to this class.
- ClassificationResult.h—This class formats returned results from binary data returned by the OpenVINO toolkit C++ APIs to a single, multiline string.
Figure 3. OVClassification OpenVinoModule source code files
The OpenVinoModule creates an entry point into the plugin that the blueprints can call. In turn, OpenVinoModule will call into the OpenVinoWrapper to interface with the OpenVINO toolkit library.
The module itself consists of:
- OpenVinoClassification—This class is a blueprint interface as an Actor component that can be added to any Unreal Engine’s Actor and called from blueprint.
- FOpenVinoModule—This is Unreal Engine’s Module class that controls loading and unloading of the OpenVinoWrapper DLL and delegates basic calls to the DLL.
This project is based on the Unreal Engine’s first person shooter (blueprints) starter content and contains three main components: FirstPersonCharacter blueprint, Billboard blueprint, and customized first person heads-up display (HUD) and special content folders.
The project (and map) have customized world settings: GameMode, HUD, and Default Pawn Class.
Special Content Folders
This project contains two special content folders which, when viewed from Unreal Engine’s Content Browser have no content, but when viewed from Windows Explorer they do, as shown:
Figure 4. Special content folders
- Images, which host images to be analyzed by sending them to the OpenVINO toolkit engine.
- OpenVinoModels, which host the OpenVINO intermediate representation data files we will need to initialize the OpenVINO toolkit.
Both folders are added in Project Settings, Packaging, and Additional Non-Asset Directories to Copy, in order to be able to reference them directly from the file system when built and packaged, as shown:
Figure 5. Showing special content folders in the project properties
These folders and their content will be copied when building and packaging projects, and will be accessible directly from the file system.
First Person Character
Figure 6. Modified First Person blueprint
Located in FirstPersonBP/Blueprints/FirstPersonCharacter. All the default functionality was left intact; however, additions were made to the blueprint. In the components section an instance of the OpenVinoClassification C++ class was added. Down in the Variables section, a LastStatus public string variable was added to display results from the classification functionality.
Figure 7. First Person Character blueprint added component and variable
- OpenVinoClassification—Actor component for initialization of the plugin and responding to OpenVINO toolkit plugin classification complete events.
- LastStatus—String variable that will contain results from the OpenVINO toolkit plugin that get displayed in the HUD.
OpenVINO Toolkit Initialization
The Event Graph is where the OpenVinoPlugin that will be initialized resides. When the Event BeginPlay triggers, the OpenVinoClassification object’s Initialize function is called. After Initialization, the Last Status string variable is populated with any error or success messages and is printed to the OpenVinoHUDWidget.
Figure 8. Initializing the OpenVINO™ toolkit plugin, Part 1
As can be seen in Figure 8, part of the initialization is to get the paths to the required OpenVINO toolkit intermediate representation files. Once created they are fed back into the Initialize function.
Figure 9. Initializing the OpenVINO™ toolkit plugin, Part 2
Handle OpenVINO Toolkit Classification Complete
Once the OpenVINO toolkit classification is complete, the On Classification Complete event is triggered, which passes the classification result formatted as a single, multiline string value. This value is then stored in the LastStatus variable, and printed (in non-shipping versions) to screen and written to the log:
Figure 10. Handling Classification Complete message
Located in Intel/Blueprints/Billboard_BP, this blueprint was created to handle all the different images to classify. This blueprint is responsible for kicking off an OpenVINO toolkit classification functionality. This is triggered when the user aims the crosshairs at an image and clicks the left mouse button firing a projectile at the billboard triggering an OnComponentHit event.
Each billboard contains one image/texture of a pet to classify as well as a text render component to display the image’s classification name for comparison against the results.
Figure 11. Showing a Billboard blueprint with pet textures applied
Figure 12. Pet Billboard Components and added Variables.
The Billboard_BP is made up of the following:
- Static Mesh—The dynamic material will use this to place itself.
- SpeciesName—Text field to display the PetSpeciesName variable that gets set in the details pane for the blueprint.
Three variables were added to this blueprint:
- PetTexture—Specifies which texture to use from Content/Intel/Textures.
- PetImageFilename—Specifies which .JPG image to use when supplying an image to the classification function. Building of the absolute path is done in the Event Graph of the blueprint.
- PetSpeciesName—String to be displayed below the image. This is for visual representation only and helps the user identify the animal species by name.
Construct the Billboard
The construction script creates dynamic material to display the image onto the Static Mesh component. It uses the one and only TextureMaterial. The TextureMaterial is populated by the variable PetTexture, which is a Texture object. It will then get the Pet Species Name variable, which is used to populate the SpeciesName Text Render component, which is displayed under the pet texture.
Figure 13. Pet Billboard Construction script
Start OpenVINO Toolkit Classification
When a projectile hits a billboard, the event graph starts the classification process by getting the OpenVinoClassification from the FirstPersonCharacter blueprint. Recall that the FirstPersonCharacter contains the actualOpenVinoClassification module object that knows how to make calls into the OpenVINO toolkit. Once the classification object has been obtained, the event graph creates the absolute path to the specified .JPG file associated to the blueprint. This path is then fed into the Begin Classify Image From Path function.
Figure 14. Pet Billboard Event graph
First Person HUD Customization
First Person HUD
The first person HUD has been customized to add the HUD Widget blueprint. On the Begin Play event, we are creating a HUD Widget and adding it to the Viewport:
Figure 15. OpenVINO™ toolkit HUD
OpenVINO Toolkit HUD Widget
This widget is basically one Text Block stretched over the entire screen:
Figure 16. OpenVINO™ toolkit HUD widget
In the Details tab you will notice that Text is bound to the GetText_0 function, as shown:
Figure 17. Text Block populated by GetText_0 function
Selecting Graph in the upper right corner of the BP screen brings up the GetText_0 function. This blueprint function is reading the Last Status result from First Person Character blueprint and returning it to the text box for display:
Figure 18. OpenVINO HUD Widget GetText_0 function
Note that this variable (Last Status) is set to:
- Result of initialization, after OpenVINO toolkit initialization.
- Result of classification, after classification process has completed.
Get OVClassification Built and Running
The project does not come precompiled and set up. This section guides you through the steps of compiling the OpenVinoWrapper DLL and generating the OVClassification solution files so that you can view the OpenVINO toolkit plugin source code.
Figure 19. Project directory listing
Figure 19 shows the contents of the zip file after it is extracted to your hard drive. It contains the OVClassification Unreal Engine project with the blueprint textures that we covered previously.
Build the OpenVinoWrapper DLL
Note that prior to building the OpenVinoWrapper DLL you must have completed the installation of the Intel Distribution of OpenVINO toolkit, version 2019.2.242.
Step 1. Open the developer command prompt
Figure 20. Microsoft Visual Studio* menu
Step 2. Run Setupvars.bat
Figure 21. Command to set up OpenVINO™ toolkit variables
Navigate to your OpenVINO toolkit installation location and run setupvars.bat. Running setupvars.bat is required because it sets environment variables that CMake uses to build the Visual Studio solution. Without running setupvars.bat, CMake will fail.
Step 3. Run CMake on the OpenVinoWrapper directory
Navigate to the location where you unzipped the OVClassification project, and then navigate to the directory shown in Figure 22 and run CMake.
Figure 22. Run CMake to build OpenVinoWrapper DLL Microsoft Visual Studio* project
After CMake generates the files you will see a folder structure that now contains the Visual Studio solution files as well as all the supporting OpenVINO toolkit files needed for distribution.
Figure 23. Directory listing after running CMake
Step 4. Compile the OpenVinoWrapper DLL
Now that the Visual Studio solution has been generated, open the solution and build it. Doing so builds the DLL and places it in a location where the OVClassification Visual Studio project solution will find and consume it.
Figure 24. OpenVinoWrapper DLL Microsoft Visual Studio* solution
Compile the OVClassification Visual Studio Solution
Now that the OpenVinoWrapper.DLL has been created, we can move on to generating the Visual Studio solution files for the main project.
Step 1. Create the OVClassification Visual Studio solution files
Right-click OVClassification to bring up the context menu, then select Generate Visual Studio project files.
Figure 25. Generate OVClassification Microsoft Visual Studio* solution
Step 2. Open the OVClassification Visual Studio solution and compile the app
Figure 26. OVClassification Microsoft Visual Studio* solution
At this point, all the source code should be compiled, allowing you to open the OVClassification Unreal Engine project and run it.
Open OVClassification Unreal Engine* Project
When you open the project you should see something similar to what is shown in Figure 27. The main scene will have several rows of billboards already set up with images of animals configured on them. At this point the application should be ready to run.
Figure 27. OVClassification Unreal Engine* project
Running the App
Running the app puts you in first person mode with the default gun and projectile. The classification takes place when a user shoots a billboard. Once the classification is complete, the results are displayed in the upper left corner of the screen.
Figure 28. Showing OVClassification Classification results
Download the Code Sample
At the bottom of this article, you can find the code sample of this project in a zipped file format that you can download and use in your project.
This article and code sample should help you integrate the Intel Distribution of OpenVINO toolkit into your Unreal Engine applications. I have talked about the dependencies and the architecture, and have walked you through the Unreal Engine blueprints. It is now up to you to take it and develop your own applications.
Product and Performance Information
Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.