The Creative Pipeline for VR Developers and Artists: An Introduction

ID 671618
Updated 1/8/2018
Version Latest



The pipeline of work for creating 3D assets is becoming more and more important. More software and application experiences are moving into the 3D realm as virtual reality (VR) and augmented reality (AR) move toward the mainstream. Whether you are a developer just getting into this space or an artist and creator now finding yourself on a commercial VR project, the learning curve related to 3D workflow can seem daunting. This post is designed to give you a sense of the creative side for 3D assets, so you better understand your options and what areas to study. Listed below are the four key steps in the creative pipeline most applicable to the creative side of VR development. Note that we did not include level design, rigging/weighting, and animation. While these steps are key to game development, they are more specialized skills and not as universally applicable to VR applications. Tools are becoming available that help democratize such workflow.

  • Creating or acquiring 3D meshes: See Sculpting 3D assets in VR using Oculus Medium 
  • 3D retopology and UV unwrapping of 3D assets:
  • Texturing and materials editing of 3D assets:
  • Asset Export/Import and other workflow needed in 3D Engines

Creating  or Acquiring 3D Meshes: 

Whether you decide to download and license 3D assets or build your own, you have many choices during this phase of the workflow. Be aware that even acquiring assets will require knowledge and skill in understanding the cost of mesh topologies, mapping, and textures in your software or animation. Here are some options that will get you at least a base 3D asset to start the process.

  • 3D scanning :  The ability to scan 3D assets is becoming more of a reality. You can use photogrammetry tools from the Apple App Store* or Google Play* store to use a smartphone as a scanner. 
    3D Scanning
    While scanning is a convenient technology, nearly all scanning work requires some 3D topology editing to clean and properly prepare the work for a VR experience (see this serieson scanning and prepping 3D scans for Unity*). Scanning usually results in a mesh with a much higher density than you need and a less-than-optimized material image for the mesh. You likely will still need to work through the editing and texturizing steps to make the scans efficient in a game or animation. In the end, 3D scanning solutions may not save you much time, but they can be a good way to either get started or pull in a very specific asset you need.
  • Purchase store assets: 3D engines stores like those from Unity Technologies and Epic's Unreal Engine Marketplace are a great place to find ready-made assets. You can also find 3D asset stores like TurboSquid* on the Web.  
    3D Stores
    These marketplaces are designed to drive revenue, so if cost is a factor, and depending on your 3D needs, it may be challenging to find something other than crates, zombies, and weapons. The advantage of store assets is that you can purchase much of what you need while saving time on the creative side. But be sure you understand the license and terms of use for any assets you download. If you find yourself gravitating to store assets, we recommend you first spend some time learning about UV maps and texturizing 3D models. You may find it more cost effective to repurpose purchased assets and editing the mesh or materials to make them look different. See the “Professional 3D Modelers” – “Full-purpose 3D editors” section below for tools for the job.
  • Simple 3D drafting tools: These tools allow you to draft and create using 3D primitives (cubes, spheres, and so on), and then shape and extrude them into complex shapes like tables, chairs, industrial tools and hard assets we experience in the real world. One such tool, formerly from Google, is Sketch Up*. The free version allows you to create a lot of amazing things, which can be exported to .DAE files that other applications can use. Such tools are not designed for VR but may be friendly or familiar enough for you to get started in creating custom assets. To get these to work you still need to use a 3D editor like Blender* or Autodesk Maya* to properly unwrap the UVs and texturize the assets.
  • Professional 3D sculpting/modeling tools: For most VR applications you may use, two types of tools are typically used to create the assets and experiences: sculpting or full-purpose modeling tools.

    3D Sculpting: 3D sculpting is like painting in 3D. You paint using 3D pixels called voxels, usually starting with a sphere of virtual clay made of them. You add in voxels, and then push and pull them in 3D space to form any object, a process that can be enjoyable and intuitive for artists. Sculpting-only tools include Sculptris* on the free end and ZBrush* and Autodesk Mudbox* on the high end of paid software.  Oculus* Medium is a best-in-class sculptor for VR. See “VR sculpting tools” below.
    3d sculpting
    Full purpose 3D editor: These tools allow you to sculpt as well model, texture, render, animate, and perform other tasks in the 3D workflow. Modeling a 3D mesh is like drawing with chicken wire to create a mesh. A variety of technical tools allow you to extrude, lathe, and do Boolean operations with the mesh to create any type of object with complete precision. Tools include  Blender, which is a free, open source tool, Maya, and Cinema 4D* and 3DS*. All these tools are at the same professional grade.  
  • Tip: If you are just starting out in creating 3D assets for VR, we recommend you get your "sea legs" with Blender. All the skills you learn with Blender are portable to MayaWatch my video introduction to using Blender to get a sense of the application. Also see Beginning Modeling by Blender Guru
  • VR sculpting tools: These are new tools that allow you to create 3D assets by painting in space using a VR system. Using a tool similar to a spray can, you spray 3D pixels voxels in space. 

    You can smooth, refine, and add detail as needed, as well as shape nearly anything you can imagine. The best-in-class solution is Oculus Medium. Follow my link here on using Oculus Medium to sculpt 3D assets.

    Recommendation: Oculus Medium. As a VR developer or artist new to the space, you should have access to a VR head-mounted display to use with Oculus Medium. Even if you develop using VIVE* you can use it by downloading the Revive library. Oculus Medium is an easy and intuitive way to do either basic or advanced modeling of 3D objects. Tied for a second-place recommendation are Blender or a 3D asset store. Overall, your choice between the three may be based on whether you feel free-flow artsy (Medium), like technical drafting precision (Blender), or want to put your wallet to work over your time (asset store). Follow my link here on using Oculus Medium to sculpt 3D assets.

Retopology and UV Unwrapping of 3D assets: 

Retopology is the process of rebuilding a 3D mesh so it's lower in size and designed better for animation. It may seem like an unnecessary step, but it is a key one. Most original meshes, either built, acquired, or scanned have captured a level of detail that makes it look great but is not necessary. Also, most highly detailed meshes can create unwanted creases or folds when animating. To avoid this, the detail you see in a high poly mesh can be transferred out from the 3D structure and into 2D texture maps, allowing the 3D mesh to be a much smaller file with fewer parts to deal with. In the image below the mesh was reduced from 2,200,000 faces to 2,300 faces, yet the output looks relatively the same.

UV unwrapping your mesh is a key part of this process. In order to get a low poly mesh to look like it has detail, you need to wrap the low poly model with information from a detailed model or information you create in material maps. UV unwrapping is the process of unfolding a 3D model into a flat 2D map of all the vertices in the mesh. The result of the unwrapping is called a UV map. The UV map acts as a guide and defines how various material and texture maps lay over the mesh so they fit cleanly. These 2D map files can hold data on different properties such as height displacement, color and transparency, metallicness, roughness/reflectivity, and subsurface scattering.
UV unwrapping

As far as the right tools for the job, you are looking at the same list of professional full-purpose 3D editors to do retopology and UV unwrapping: Blender on the free end and Maya on the paid side.

  • Recommendation: Blender. It is a free tool that can do most kinds of retopology and UV work as you are learning. You can also get free or paid add-ons that simplify the work. Included with Blender is an add-on called BSurface, which allows you to create simple sketch lines of where and how you want to retopologize, It also will calculate a new mesh on the surface of the old mesh. (Stay tuned for an article on this topic. In the meantime, check out Beginners UV Unwrapping Tutorial by Blender Guru)

Texturing and Materials Editing of 3D Assets

This work is the secret sauce to good-looking and realistic 3D assets. If you have ever been blown away by a street scene, or the realism of a vehicle in a game or VR, the trick is in textures and materials editing. This is the part that people tend to miss. They jump from 3D asset to Unity or Unreal and wonder why their stuff doesn't look as great as they see in other applications.

You can do some of the textures and materials editing in the professional 3D editing tools like Blender. Blender has all that you need to set those up, especially if you find existing physically based rendering materials online and want to use those materials on your objects. However if you want to get into the details of editing materials and add or delete things to be the way you want, consider using Substance Painter*.  This tool makes the work easy and allows you to bounce back and forth between editing and adding materials, and then going back to your editor like Blender to add or adjust your model as you need to.
pbr materials

  • Recommendation:  Blender, again, a free tool. However this area is becoming fairly specialized, and tools like Substance Painter are adding significant value. With Blender you can do a lot using the Principled Shader, by assigning image textures as physical properties. You can download material types for anything from moss, to leather, to rusted steel and use them in Blender to make your models look realistic. Substance Painter allows you to paint in texture and details with more ease, control, and flexibility than you can with Blender. This blog will be updated in the coming weeks with a link and post on this process using both Blender and Substance Painter. In the meantime, check out this video that discusses the use of Principled Shader by BlenderGuru

Asset Export/Import and other workflow needed in 3D Engines. 

The final task in the process is to get your 3D assets in your game engine and get it working for you. Unfortunately because there are so many tools and methods to create 3D assets, often the engines have their own proprietary way of structuring the assets with materials and lighting. Often it's a matter of importing, and then assigning the materials you created to the appropriate physics-based materials like metalness, roughness, or displacement. Your approach is really dictated by the 3D engine you choose to develop your VR application. Beyond just reconnecting materials to 3D assets, you will have control in these engines through scripting, allowing you to alter properties or swap out material properties and looks to your assets as needed.
This blog will be updated in the coming weeks with a post on this process for both Unity and Unreal.
We hope you found this article helpful as a starting guide and overview of the core areas of work for creatives, designers, artists, and developers who want to contribute to the space of quality VR experiences.