Vista House is a virtual reality experience that demonstrates the use of photogrammetry technology to create 3D assets from 2D source materials, then incorporate them into a Unity* 3D scene.
Why Create This Project?
This project shows how photogrammetry can be used to generate assets on a large scale which can then be brought into a virtual reality (VR) scene in Unity. Vista House in the Columbia River Gorge, Oregon, United States, was chosen as a photogrammetry subject because its location made it easy to isolate from other nearby buildings and structures.
Virtual Reality Hardware Compatibility
The project is designed to be compatible with the PC-based Oculus Rift* and HTC Vive* VR hardware.
Capture the Imagery
Photography partner, Third Rock Studio, obtained permission to fly drones around Vista House and capture the images needed to run through the photography software.
Equipment
Third Rock Studio used Inspire* 1 Pro and Inspire* 1 RAW drones to photograph Vista House, on which were mounted a Zenmuse* X5R camera fitted with a 3-axis gimbal and 15mm f/1.7 lens.
Image software
Photoshop* software was used for manipulation of the source image, and the 3D model of Vista House was created using Agisoft PhotoScan* 1.4.2 Standard Edition software.
Build the VR Application
Software tools used
Unity* 3D software version 2018.2.8f1 was used to build the VR application for this project as well as SteamVR* Unity Plugin v1.2.3 and Visual Studio* 2018.
Unity* Project Layout and Structure
Unity project assets
Figure 1. Screenshot showing the Unity* project assets.
Prefab
Contains the FBX model with teleport points.
Scenes
Only one scene was created for the application, named "VistaHouse".
SteamVR* Unity plugin
This folder is for the SteamVR Unity plugin that was downloaded from the Unity Asset Store*.
Assets
The following asset files were saved directly in the Assets folder:
- Vista_pano: The panoramic image taken around Vista House.
- VistaHouse: The 3D FBX* model of Vista House created in the photogrammetry software, in this case Agisoft PhotoScan 1.4.2 Standard Edition software.
- VistaHouseTexture: The texture generated by the photogrammetry software to go on the 3D model.
- VistaPanoramicMaterial: The material used to populate the 6-sided skybox cube.
Unity project hierarchy
Figure 2. Screenshot showing the Unity* project hierarchy.
VistaWithVRandTeleporting
The Assets or Prefab folder contains a single prefab for this project. The initial idea for this project was to try and put a panoramic image of the surrounding area around the model. To make a long story short, I grew tired of trying to scale and reposition the model and readjusting the teleport points so I created a prefab making it easy to at least rotate and move the model as needed to try and align it with the surrounding panoramic image.
Everything in the prefab are existing SteamVR components with the exception of the Vista House model itself.
The files are located as follows:
- Player: The SteamVR prefab located in Assets/SteamVR/InteractionSystem/Core/Prefabs.
- Teleporting: The SteamVR prefab located in Assets/SteamVR/InteractionSystem/Teleport/Prefabs.
- TeleportPoints: The file of collected SteamVR prefab TeleportPoints which resides in Assets/SteamVR/InteractionSystem/Teleport/Prefabs
- VistaHouse: The FBX model of the Vista House as generated by the Agisoft PhotoScan 1.4.2 Standard Edition photogrammetry software.
I won’t go into a lot of detail on how the SteamVR components work, that is beyond the scope of the document. SteamVR has documentation describing how the teleporting works. On a high level, the prefab was created by dragging the SteamVR prefabs as well as the Vista House model onto the VistaWithVRAndTeleporting empty game object. A TeleportPoints empty game object was created to be used as a container and several SteamVR TeleportPoint prefab objects were added and dragged into place. Behind the scenes, SteamVR automatically wires up the teleporting functionality for you, so nothing else needs to be done.
After everything was put in place, it’s the normal Unity3D process to create a prefab, dragging the entire VistaWithVRAndTeleporting hierarchy structure into the Assets/prefab folder.
Put It All Together
Working with photogrammetry and trying to wrap the model in the real-world surrounding environment is a challenging task. Given the amount of time I had to work on this project, I was not able to find an easy resolution to this challenge. This following gives you an idea of what to expect and the challenges faced.
The model
After securing the required permits to fly a drone around Vista House, a clear early morning was selected for the photography. Once the footage was captured, the images were processed through the Agisoft Photoscan photogrammetry software, and the FBX model was exported out.
From there, the model was brought into open source 3D creation software to remove unwanted information, such as the camera data. The choice was made not to manipulate the model, so users could see what was possible without any 3D editing.
This exercise demonstrates that, on a scale this large, 3D cleanup will almost certainly be necessary to create crisp sharp lines and flat textures. The textures on Vista House itself work well because the structure is made of stone blocks. However, there is reduced 3D quality in places such as the parking lot, the steps leading up to the structure, the railing around the wheel chair ramp, and the parking lot stone wall.
Adding the teleport points
Once the three SteamVR prefabs were placed into the hierarchy, the next stage was a simple matter of adding teleport points to the model which would allow the user to teleport around the parking lot.
Figure 3. The assigned teleporting points allowing the user to move around the model.
The environment
Figure 4. Screenshot showing the difficulties in accurately aligning the panoramic image with the model.
One of the project goals was to wrap the real-world environment around the model so that the parking lot and entrance would line up with the panoramic imagery giving the illusion that a user was actually there visiting the structure and environment.
Attempts were made to make this a reality by first using a skybox. I created the VistaPanoramicMaterial and assigned the vista_pano image to the cubemap field.
Figure 5
The challenge with this approach is that you can’t position or scale the model in such a way that you can get the surrounding environment to align with the model. I suspect that the only way this method would work is if an image was taking from the exact center location of the Vista House structure at ground level which for obvious reasons can’t be done.
The second idea was to create a large sphere, place the model inside that and project the panoramic image to the interior of the sphere. Again, this proved to be larger challenge than it was worth. The panoramic image was distorted and the model still was unable to be rotated and placed in such a way that it aligned up correctly.
Within the scope of this project, there is not yet a completely satisfactory solution to the puzzle of how to align the panoramic image with the 3D model. However, it's likely this can be done given time and the right approach. One approach may be to use a series of strategically photographed flat images. Within Unity3D place a series of panels around the model and place those images onto the panels. This too would likely introduce its own set of challenges, however it does sound like a potential solution.
Closing Notes
This article provides some useful pointers for combining a 3D model generated from photogrammetry, as well as realistic expectations of what the final model will look like. Photogrammetry is a very powerful technology that makes it possible to take a series of flat 2D images, feed them into software, and generate a 3D model. Applications in the VR space include mapping, tourism, gaming, and industrial uses.
About the author
Rick Blacker is a developer evangelist from the United States. Learn more about Rick and his role at Intel.