This is Part 2 of my article, Sampling Surround Binaural Audio for use in Virtual Environments and Games (Part 1). It stands as a primer for advanced sound design techniques for using multichannel spatial audio captured from field recording and sampling, and transforming it into imaginative useful audio assets for games, virtual reality, film, or video. As you may recall from Part 1, I traveled all over the world recording many terabytes of soundscapes, environments, and Foley sounds from Southeast Asia, Europe, and North America. I want to share with you some of my techniques for cleaning up this amalgamation of audio; editing and transforming it with audio plugins and effects, and mastering it for a sound library. Exhibits are provided, using software examples with MAGIX ACID Pro* Next, PreSonus* Studio One* 4, MAGIX SOUND FORGE* Pro 13, iZotope Iris* 2, and ZOOM* Ambisonics Player 1.
Figure 1. Justin Lassen working on his spatial audio sound library Black Fox Society: Vault in his 5.1 Surround Studio.
Audio is an Abstract Thought for Sound Designers
The most important concept to think about as a sound designer, before processing or changing audio, is not to take it too literally. For instance, the sound of a door slamming could end up being a snare drum in a rhythmic musical loop or perhaps a point of contention in a scary montage drenched in reverb effects. A recording of people talking in a room could be modified to sound like a choir of insects humming in a sci-fi spaceship chamber, but none of these sounds have to be taken verbatim. In sound design, anything you dream of can be created from scratch and custom sampling of the world around us. I think of audio recordings, no matter the quality or source, as abstract impulses of frequency with transients and shapes and modulation. This gives me an objective starting point when trying to bring video games, movies, or virtual reality environments to life with living, breathing, aural atmospheres. In short, I try to find music in everything and the soul of a scene or environment in the vibrations of layers of sound.
Figure 2. MAGIX ACID Pro* running a 5.1 surround session with iZotope Insight* monitoring levels of various spatial audio tracks from the Black Fox Society: Vault sound library.
In other words, I like to think of audio as colors for a painting of a landscape in our minds. I bring audio into a session simply as an idea, a basic shape. I can cut colors, frequencies, and tones out of it and stretch, morph, mix, and augment to satisfy the ideas of my vision for the scene or environment of what it could be turned into; similar to the way a visual artist might use layers in Photoshop*. For example, an ambient room of humans chattering and talking could actually end up being a gorgeous musical choir tone with the help of effects, experimentation, and imagination played through a powerful engine like iZotope Iris.
Figure 3. iZotope Iris* 2 is a powerful sample playback engine that you can drop audio and samples into, select frequencies, and design new sounds.
Try it: After loading an instance of iZotope Iris 2 onto a track in your preferred digital audio workstation (DAW), drag a .WAV file right into the interface and you are ready to play with a MIDI keyboard or direct MIDI input on the event in the channel. Iris makes it just that easy. You can immediately start playing with the parameters of your audio file intuitively for unlimited creative results. Bonus, it is actually fun.
Throughout my career, I have become friends with some of the biggest and most award-winning visual artist/digital painters in the world who work on the biggest movies, TV shows, video games, and exhibits on the planet. Some of these friends also feel that photography is very similar to this creative process of abstraction that I am explaining here. My dear friend and concept artist, Alex Ruiz, paints some of the most imaginative landscapes for films and shows, and if you watch him work, he is actually using not only colors, shades, and brushed paint strokes in Photoshop, but also stamps and cutouts from photographs of buildings, clouds, abstractions, and so on (another form of sampling). Everything is brought together to create a bigger picture that has a completely new story or thought. In one example, I composed some music in an improvisational way on stage in front of an audience at an Intel keynote spotlight performance during CES to his speed painting. In this video you see how he brings samples together with his imagination to create The Procession. My type of sound design is very similar to this process that he visualizes so perfectly.
Figure 4. Procession art collaboration between Alex Ruiz and Justin Lassen, showing how visual art is inspired by sampling environments, much like ambient audio design. Music composed by Justin Lassen; painting by Alex Ruiz.
Stay Organized, Name Recordings and Content, and Back Up
After returning from your field recording expedition or sampling audio sources, you will undoubtedly accumulate a large amount of audio (and/or video) data. The way that cameras and recording devices save file names is generic (that is, RECORD001.WAV, RECORD002.WAV, and so on), so make it a point to rename files as their location or content, or groups of waves in folders with location name. During my trip we went to a lot of different countries and cities, so we ended up going with grouping each day’s files together in a folder with the date and city/country, so that I had an overview of which data to dig through during the editing process.
Figure 5. 16 TB studio drive and Zoom H3-VR recording device for 360 audio (and Justin’s 2018 Top Innovator Award to boot).
Back up is probably one of the biggest pieces of advice I give at shows, talks, and events on projects. If you are working on a big project and you come back from the trip and your hard drive crashes, all those plane tickets, hotels, and trip expenses are for nothing. For my trip around the world we brought two separate hard drives and a giant pile of SD cards, and carried them across two people, in case a bag got lost during a flight. This insured that upon completion of the trip, one way or another, we would have the audio and video from this adventure.
Figure 6. Justin Lassen in various countries around the world in contrast with MAGIX’s ACID Pro surround mixing interface (collage).
Each night, even after a long day of traveling or recording, we would break out the computer and download all the audio and video from the recording devices and cameras, and copy them to both separate hard drives, labeling everything properly. This made the editing of the project a lot smoother and organized. Doing it ahead of time on the trip also made for less file management work in the studio.
Note: I emphasize this at the start of my article because these steps will save you headaches and stress during the sound design process, when you just want to start playing and dragging files and audio into sessions.
Figure 7. We literally had thousands of video (.MP4, .MOV, and so on) and audio (.WAV) files captured by the end of the trip around the world. A lot of content to organize and back up.
Converting Ambisonic/Quadrophonic Audio to Binaural/Stereophonic
I captured a lot of ambisonic audio on the trip, so I had a lot of 3D/360 audio to play with during editing. Because my sound library was intended for creative developers in video games, virtual reality (VR), film, TV, and radio, I geared everything toward stereophonic/binaural format for final output in the libraries. I captured everything in 24-bit/96 kHz, so it had a lot of data there for clean conversions. Ambisonic audio transforms quite well for this purpose and still allowed me to create beautiful worlds of audio and sound in stereo/binaural.
Figure 8. ZOOM Ambisonics Player* is an easy, straightforward interface for converting the Ambisonics format to binaural, stereo, or 5.1 mixdowns.
Try it: When you start the ZOOM Ambisonics Player you are shown one interface and no complicated menus. Select Add Files… and find your Ambisonics formatted audio files on the left panel; load them. You can select the files in any order you want, and on the right panel and main interface you are able to change the 3D placement of the audio as well as playback in stereo, binaural, 5.1 surround, or a custom Left Right setting. You can also change the start and end points to process only what you need. Select Export and save your changes. Ambisonics does not have to be hard with tools like this. I was able to record a ton of great audio for the game Monster Jam Steel Titans for THQ Nordic*/Rainbow Studios* this year. This was also one of the tools I used when converting all of the ambisonic audio from the trip back in the studio.
Figure 9. If you load ambisonic audio into Studio One* you can see a better visual representation of the data is saved within the file. It is a .WAV file with four mono channels of audio from the capsules on the ambisonic microphones.
Layering Ambient Audio Sources Together Creatively
This becomes an experimental part, where your personal aesthetics come into play; this is where the art form comes to life. I may have the sounds of a busy night market in South Korea, the echoes of an ancient temple in Taiwan, and a deep, dark tunnel recording in Paris mixed with the chirps and insects of a jungle recording in Thailand and the sounds of ocean waves on a beach in Northern California. To the untrained ear, this might sound like a cacophony of nonsense. However, to a sound designer, these are all just color palettes waiting to be effected and changed into completely different ideas and spaces.
Figure 10. Find sounds from your field recordings that may sound interesting together.
Figure 11. Move the sounds around to find sections of audio that layer well together for ambience. This is subjective but can be very fun.
Next, after finding a thoughtful and specific placement of these binaural or ambisonic sounds in the session, we can begin using software effects plugins to transform each layer and find the strongest parts to emphasize or de-emphasize, pitch, shift, or add special filters and transitions. In this case, I am trying to amplify certain frequencies of each scene with certain tonal (and atonal) effects and processes, as well as a few wild delays and distortions.
Try it: Open AIR in Studio One is a fantastic convolution reverb effect with a ton of awesome built-in impulse responses (IR) for creating atmospheric audio spaces for static content that needs sweetening.
Figure 13. Studio One 4 Mixer and Inserts view for adding audio effects to each layer.
There are so many great plugins that come with both Studio One and ACID Pro that you will be designing creative and unique soundscapes and audio in no time. They can layer in any order you want and can produce an unlimited number of results for your final audio output.
Figure 14. Studio One Professional 4’s built-in factory plugins are very powerful on their own and even more so together.
Figure 15. MAGIX ACID Pro factory plugins are also quite fun and powerful to design with and interchange together for different effects.
Factory Plugins versus Boutique Plugins (DSP)
Do not underestimate what you can do with factory audio effects that come with any given DAW. Most DAW software comes with all the basics necessary to process most audio circumstances: Delay, reverb, compression, gate, limiting, filtering, EQ, distortion, pitch, conversion, and so on. When these are used in different orders, you can come up with some interesting new takes on the same audio sources, textures, and sonic ideas. For instance, I have created an effects chain that can process audio in a way that makes it come to life rhythmically.
Figure 16. Studio One 4 showing a simple effects chain to compress, and add delay and rhythm to an audio source with a limiter on the main out.
Of course, there will always be extra-cool, fancy boutique, or specialty audio processing plugins. These types of plugins are typically created by other developers for use in all DAW software and add cost to your tool set, but can help save time in the long run. A lot of boutique plugins are actually taking very basic processing concepts that you can already do with the built-in effects in specific orders. Whether you use built-in effects or third-party effects suites, when used sparingly and tastefully (or sometimes overly), you can combine them into creative chains that can take your audio to other worlds and places sonically, and create some extraordinary soundscapes.
Figure 17. Studio One 4 showing an advanced effects chain for even more powerful audio processing.
Working with Foley Sounds, Impacts, and One-Shots
Ambisonic and binaural content has its place in sound design for sure, but not to overshadow the creative uses of tried and true monophonic recordings. With things like slamming a door, crunching devices, squishing liquids, dropping objects, scraping metals, and using elements like wind and fire, you can quickly create a large library of found sounds and performed sounds. Microphone placement on this sort of stuff can be freeform and creative. You can have closely mic’d sources with lavalier microphone (lav mics) or larger spaces mic’d with overheads. You can use condenser mics, dynamic mics, ribbon mics, and so on. There are no rules when it comes to sampling audio. If it sounds cool or interesting to you, then you have succeeded. There are so many different kinds of recording and capturing technologies that you can always find a creative and unconventional way to capture the audio performance. You can take all of these types of sounds and mesh them together in unlimited configurations for music, rhythmic, ambient, or for traditional sound effects design.
Designing Rhythms with PreSonus ATOM* and Impact XT*
I released 483 one-shots (.WAV) format in Abstract, and created 13 epic drum kits specifically for the PreSonus ATOM* hardware and Impact XT* sampling software built into Studio One 4.
Figure 18. 13 ATOM*/Impact XT* drum kits made from a selection of the 483 one-shots included in the Abstract audio library.
Mastering All Audio for Professional Libraries with MAGIX ACID Pro*
I have been using ACID Pro since the 90s. It’s a classic for a reason. The looping capabilities and audio processing capabilities of ACID Pro and Sound Forge* Pro are synonymous with being the Photoshop of the audio world, especially when putting together loop-able content for games, VR, movies, trailers, and music. As you can see in this session, I am able to make sure none of the content is clipping or distorted before doing a final rendering of everything at once.
I was able to load hundreds of tracks in the same session without bogging down the system at all. It handled everything awesomely. I was able to tag the files with metadata from the session all at once instead of individually, which makes cataloging for a library faster. I can name each track and do a final export of all tracks at once with the same processing, so everything stays uniform and professional from start to finish.
Black Fox Society: Vault and Black Fox Society: Abstract Released Through PreSonus
These libraries are a culmination of years of travel, sampling, editing, and processing of terabytes of audio and specifically created for musicians, sound designers, composers, theme park and experiential creators, as well as film editors, virtual reality producers, and game developers. Vault is ready-to-go, ambient spatial audio content you can drop into Unity* and an Unreal Engine*, and bring your VR worlds and games to life with all the work done for you. It is just drag and drop exploration. It has over seven hours of high-end audio that brings an emotional and spatial impact to whatever world you are trying to create. Abstract has more rhythmic elements, one-shot samples, and musical tools. Both libraries complement each other. I wanted to create these libraries so that game developers and virtual reality producers had a massive resource for helping their visions come to life. It also saves them the trip of having to travel to more than 22 countries to capture all of this audio on their own. There is so much content in these libraries that you could not listen to all of it in one sitting. Let’s see what you create with it next.
Figure 20. Cover art principal photography by world-renowned artist and respected creator, Renee Robyn Photography. Renee has also done art technology demos with Intel in the past. Justin Lassen and Renee Robyn have collaborated on many projects.
Audio Demos Created with These Libraries
And finally, I present to you the audio demos for both of my spatial audio libraries in all their glory. There are musical demos and ambient demos. Abstract was geared more toward composers, and Vault was geared more toward game developers, virtual reality producers, and film editors.
Figure 21. The 3 "Abstract" demos are musical. The 3 "Vault" demos are ambient.
About the Author
Justin Lassen is a Composer, Remixer, Sound Designer and Visionary. With over 20 years of experience in the music, film, tech, and video game industries, Justin is currently working as an award-winning spatial audio designer for VR/AR/MxR projects. He has lent his production talents to iZotope, Cakewalk*, Intel®, DTS*, Sony*, Disney*, Konami*, Skybound*, Hasbro*, Lakeshore*, Interplay*, the United States Department of Defense*, and many more.
Product and Performance Information
Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.