I have previously posted and demonstrated how VR can be used as an immersive creative tool using VR apps like Tilt Brush (see Painting in Tilt Brush and Democratizing 3D Content). Tilt Brush is truly an amazing tool and experience, however the file format and assets created in Tilt Brush are not the most functional outside of it. But VR work done in Oculus* Medium* is highly functional and compatible with other 3D tools and technology. Oculus Medium is clearly a first rate 3D Scultping application while being completely functional within VR. It is comparable to Mudbox*, ZBrush* and the sculpting features in Blender*. In the end you get a structured 3D asset that can be used in 3D editors, 3D animators, and 3D game engines from Blender, to Maya to Unity and Unreal.
This post will demonstrate how to do this outlining the hardware, the techniques and process you can use to create 3D assets in VR.
Requirements and Skill Level for this work
- Technical Skill Needed: Low
- Creative Skill Needed: Medium to advanced. Should have an eye for freehand drawing, drafting, or scultping forms and assets
- Target PC: VR Capable PC. The following was used for this work
- Intel® Skull Canyon NUC PC with 16GB Ram, Intel® Corei7 and Intel Thunderbolt™ 3 port
- Razor* Core* GPU Enclosure with nVidia* 1070 GPU connect via Intel Thunderbolt™ 3 port
- VR Hardware: The following was used
- HTC* Vive* Steam* VR HMD and tracking system
- ReVive* library, to run Oculus software on HTC Vive
- Image of this hardware below. The PC is the small VHS sized device to the right of the mouse. A single small Thunderbolt cable connects to the GPU enclosure to the right of the PC. You can find a variety of VR capable PC's from Intel here
Why Create / Design In VR
Before we begin, let me first express the opportunity for content creators to consider VR as a medium for their work. If you've read my post you'll know I approach these technologies first and foremost as a creator and artist, not a technologist. And as a long time artist I know many of us have decades of muscle memory built up around sketching and turning what we see in the real world into 2 dimensions representations on paper. We prefer pencils, brushes or pens to a mouse. As well we know the process to go from an idea in our head to a digital 3D asset is difficult or out of reach because as mainstream 3D tools are mechanical and tedious. I don't care how many times I learn how to UV wrap textures on 3D assets or retopologize a 3D mesh, it is still a struggle for me.
You'll have to trust me on the following. Creating 3D assets in VR is as simple and enjoyable as sketching, while allowing you to create the asset directly as you need it for 3D use. Whether you want to draw out fantastic sculpture you'd print, or create a detailed Roman arch for use in animation or a game, VR art tools allow you to create things as easily as you'd sketch them. Additional, you get the added capability of walking and looking around your art as you are creating it. A wondrous experience.
It is my belief that VR opens up the world of digital 3D experiences to all sorts of artists. Where before this was only left to those who took time to understand the complex 3D CAD and GGI tools, VR now opens digital 3D artistry to anyone with a good eye and imagination.
The Basics of Oculus Medium
Oculus Medium is the first good structured 3D art tool to hit the market and hopefully the first of many. While it is an Oculus application, you are not required you have an Oculus VR system to work. An open source library called Revive allows any HTC Vive users to use those systems. If you have an HTC Vive system, you need only download and install the library for GitHub. Follow this link for details.
From an artists perspective, Medium allows you to follow and translate a lot of established techniques artists use. The difference is that you need to know what menus and options to select to do those techniques. After getting 1 or 2 projects behind you it will become intuitive. Here's a process that might seem familiar and follow any work you've done. These are the same steps to follow when working Medium
- Source reference material for your work - Select an import reference images as you need them
- Sketch out fundamentals of your piece - Use a small brush at starting low resolution to sketch the basics of your piece
- Step back, examine your work in progress - Walk around your and rotate art, look at negative space etc.
- Work in detail as you go along - Increase the resolution and add in detail as needed
Medium Controllers and Features: Medium will provide you with some tutorials to get familiar. I recommend going through the first few of these. For those who want to get right in here are the basics
- Left hand palette of tools: The key things you need to select or change are all on your left hand. Looking down you will see that there are 4 options which correspond to the 4 items on an Oculus Touch controller. If you are using a Vive controller these are part of the left track pad. Either way you are using your left thumb to select each
- Home Menu: The home menu is activated if you tap the lower right of the pad or button. This allows you to load in saved projects, exit the application, select tutorials etc.
- Control Panel: If you tap to the upper right with your thumb you will get to the main options menu for your work. This allows you to create a new project, save a project, export and turn on or off features like the mirroring tool
- Palette of Tools. If you swipe the left of the pad up or Joystick up you will see the palette of tools. These are your sculpting tools. This is how you switch from a brush that paints 3D clay verses paints color on the clay. This is also how you are able to add in detail and refine your clay using tools like Smooth and Inflate
- Layers Menu: If you tap or swipe down left you will activate the Layers menu. This is where you can add in layers of work within your project, allowing you to create work that is not disrupted by other layers. This is also where you can increase the resolution of your work, allowing you to sculpt in greater detail
- Right Hand Tool & Tool Options: Your right hand is where you will do the work. If you've selected the Clay tool, it will put out bits of clay in a stream for as long as you pull in the trigger at your index finger.
- Tigger: The trigger located at your index finger is like pulling the trigger on a hose, or sander, or any tool that you point and operate. Should be intuitive :-)
- Resize: You can resize your tool by swiping your thumb on the joy stick or pad up or down. As you do this you will see a green gauge for how big vs small you can make that tool size
- Colors: You can select colors for the tool by selecting the color wheel button/pad in the upper left
- Tool Options: You can adjust features of the tool by selecting the Gear option on the bottom left button or pad location. Once selected you get a pop-up menu allowing you to do things like changing the brush shape, change the tool from adding clay to subtracting clay, or inflating to deflating etc.
- Brush Options: If you are in the Clay tool and you have selected the Tool Options you have the option to select stamps. These stamps are existing models of work from human anatomy to architectural shapes. These can be used to add into your scene for reference or as a base starting point of your sculpt.
- Scene Grabber - The entire scene is grabbable and moveable. Either hand can do this by squeezing the grabber buttons on the handle of the controllers. Once squeeze you now have the scene locked to the movement of the controller. Another squeeze and you let go. You can resize the scene by squeezing both controllers and pulling them apart to make them bigger or pulling them together to make the scene smaller
Sculpting Tips
Start Sculpting With a Small Brush
The start of my project I decided to create an overly masculine face of a goblin like alien. I decided not use reference art as I wanted to see how well I could free form something from imagination. I started to sketch out a masculine big jawed face to start. I was thinking of a certain action hero / California Governor, since he has that look. To do this I first did the following
- Turned on Mirror featured so I could sketch one side only. To do this, using your thumb select the upper right portion of the trackpad. Use your right controller to then point at the Mirror feature and then pull the trigger on your right hand
- Set the brush down to a medium to fine tip size. On the right pad look at the size of the tip, then using your thumb in the right side of the pad swipe it up or down to adjust the size of the brush. You'll see the size gauge in green on the controller, showing you how big vs small you can make the brush.
- Sketching in the base low resolution gives you a whip cream looking piece but that is OK, the basics of what I was trying to do are there. Sculpt away. If you need to under just swipe your left thumb left, or right to redo.
That was enough for me to work with and start and laid a foundation to sculpt out my final form.
Fill Out the Sculpt
From here is started to edit the work to look more alien or goblin like. I cut the nose down and made the cheek bones more pronounced. I smoothed, inflated and deflated until I had a low res version of something to refined. I suggest playing with the following options from your left Controller palette of tools. To get to this palette, use your left thumb and swipe up on the trackpad
- Smooth - use the smooth tool to smooth out the globbiness of your 3D clay
- Inflate/Deflate - If you want to increase the clay out or decreas it down in a spot this tool will do the trick. Once it is selected you can adjust it from being Inflate vs Deflate on your right controller. On your right controller you will see a gear icon on the bottom left of the pad. Hold that down with your thumb and options for this tool will appear allowing you to adjust inflating or deflating the clay
- Move around your sculpt - I highly recommend this to beginners to VR art. Most of us have been trained on 2D art. Once you go 3D you will find you lack the eye and coordination for depth. Your work will be flat. Work each side of your project and you will soon retrain your brain for 3D.
Increase Resolution & Add Details
If you look at the image above you may notice the left base work is is not as detailed as the right work as I filled out my design, especially around the nostrils, mouth and eyelids. These are areas that require more definition than the base resolution. After you get the basic form sculpted out, use the left controller and swipe down. This pulls up the layers menu. I suggest you duplicate the existing layer one. This will create a new layer two.. Using the right controller, turn off layer 1 and turn on and select layer 2.
Then on the bottom right is something that looks like square made of arrows pointing outward. When selected this will increase your project's resolution. I suggest you increase it twice. The system will take a moment or two to increase the resolution of your sculpt. You can swipe back down on the left controller to get rid of the menu.
Refine The Sculpt
From here on out I suggest you mainly use the Inflate Tool. You will use Inflate to lift areas out or to etch in details in other areas. With this tool you have the option to use Deflate by selecting that option in your right hand (bottom left button / position of the right thumb pad). Deflate is like a grinding sander. It will cut away and allow you to etch finer details. Switching it back to inflate will allow you to puff out areas. The next most common tool at this stage is the Smooth tool to sand down and smooth any rough edges. When using these tools you will want to adjust the size of the brush and adjust your distance with the tool to the clay. These will impact how much you deflate, inflate or smooth the surface
This short video demonstrates the deflate, inflate and smooth tools in action at this resolution
Painting
Once you have your 3D asset as you like, I suggest you paint it directly here in Medium. You could use other tools, but Medium is a far more intuitive. On you left controller swipe your thumb up to get to the tool palette. Use your right controller to point at and select the Paint Tool.
When in paint mode the right controller works as it normally does however rather than sculpting you are now spraying paint. Grab the model with your left controller and move it around to paint at any portion, angle or side of your sculpt. At this point you are at a good resolution that you should be able to paint very well any detail and features you'd like
Export
The final step is to take this work and bring it into something else. Using your left thumb pad select the upper right button. Point your controller at Export. With Export Selected be sure to select the Texture options. I suggest 2048 resolution. FBX seems to work with in most applications such as Blender, Maya, Unity and Unreal. so I suggest using it for export but you can play around.
The below images are an example of this sculpt imported to Blender, then rendered using it's Cycles renderer. These took just a couple of seconds and were rendered in near real time with this advanced renderer.
