Technological advances present many opportunities to use location-based entertainment (LBE) in unique and compelling ways. This article addresses solutions for tracking, and other supporting technologies, that can greatly enhance any location-based virtual reality (VR) experience.
Figure 1. LBE can be delivered in an art gallery.
Commercial LBE experiences can be delivered in amusement parks, themed attractions, 4D films, large arcades, and multiplayer interactive games (see figure one).
Figure 2. LBE can be delivered in an arcade.
LBE experiences can even enhance customer enjoyment in bowling alleys, billiard parlors, water parks, casinos, and movie theaters. The scale of these experiences in the commercial world goes way beyond anything that can be provided in a home environment.
Figure 3. LBE can be delivered in a bowling alley.
Another exciting emerging technology is location-based virtual reality (LBVR). LBVR can be geared toward individual use or a social interactive experience. By giving consumers greater knowledge and personalized experiences, LBVR has helped spur wider adoption of VR. This technology offers ways to educate in museums, aquariums, and schools, as well as helping further the preservation of cultural and historical interests through rich information.
Figure 4. LBE can be delivered in a shopping mall.
An immersive environment that supports a person's learning preferences—including audio, visual, and kinesthetic interactions—makes it easier to retain information and provides more absorbing experiences (see figure five).
Figure 5. Various manners in which we can learn with LBVR.
With LBVR, content becomes freely available, liberated from geographical boundaries, giving users access to the sounds, sights, and surroundings of any place in the world. An immersive experience can involve exploring a national park on the opposite side of the country, visiting a rural community in South America, or taking a submersible tour of the ocean floor. Lifelong learning possibilities are expanded, granting access to different cultures, diverse environments, and historical perspectives, from anywhere the user happens to be. Hands-on activities can be included in the immersive environment, enabling fascinating learning exercises (see figure six). LBVR brings the experience to the student.
Figure 6. VR in school.
This article addresses solutions for tracking, based on system choice, as well as tools to improve real-time tracking ability within a space. You'll gain insights into how to pick up minute movements from motion capture suits, use scanning effectively, and employ other supporting technologies.
In the following sections, you'll learn more about optimizing the experience for each device and individual venue to ensure a quality experience within the capabilities of the network. We'll explore the best ways to balance resources, factoring in infrastructure, equipment, and software, and considering the cost, benefit, and complexity of the LBVR solution. Current use cases of real-world implementations demonstrate some ways to negotiate this balance. You'll also get a preview of the technologies that will be reshaping LBVR in the near future.
Solutions for Tracking in Location-Based VR
Location-based VR encompasses a large number of technologies and tracking types. Some of the larger systems use their own specialized tracking system, built around motion capture or similar technologies. Smaller systems use more standardized approaches, such as the visual tracking of Oculus Rift* or the time-of-flight tracking of the HTC Vive*.
The VOID*, an LBVR franchise chain with 17 locations, uses motion tracking in combination with haptic feedback and special effects to produce an interactive virtual experience with social context. This solution delivers an interesting use of multiple systems—with built-in failsafes and failovers—to create a truly rich and unique experience. Most systems don't need to use such expensive, specialized hardware. At the time this article was published, the VOID's system costs were about USD 500,000 per location, a cost that exceeds the average location-based VR setup (see figure seven).
Figure 7. VBVR experiences from The VOID*.
Since mid-2018, using multiple Oculus Rift VR headsets within the same space for walking users is not recommended. Each headset requires its own tracking unit and the infrared (IR) light emitted from each headset can interfere with others in the space (see figure eight). When setting up a Oculus Rift within the same approximate area, using a cell phone camera to locate the IR attenuation from each headset or, alternatively, providing blocking (such as a curtain separating the systems) is recommended.
Figure 8A. Oculus Rift* tracking.
In comparison, using a HTC Vive headset presents fewer limitations and supports sharing of base stations. Signals beamed to the headset and handsets can simultaneously support different experiences and multiple users (see figure eight). This isn't to say that the Oculus Rift is impossible to use in larger scale experiences or that the HTC Vive is free of issues. The Oculus Rift is a less-expensive headset and, when a user is seated, provides a superior experience to the HTC Vive, as long as the headsets are shielded from each other. Doing this can be more difficult than one would expect. For better results, use a HTC Vive headset in complex environments.
Figure 8B. HTC Vive* tracking.
Another option for tracking users within an LBVR experience, which is a bit trickier to implement, is to collocate the experience and real-world objects. An example would be building a maze to scale and allowing a user to walk through it with a virtual overlay. For higher fidelity, the use of timing trackers is often employed. The Superman* roller coaster at Six Flags America on the East Coast uses Samsung Gear* VR. A timing-based tracker in each roller coaster car receives information from the track about the car's position on the track. This information is updated at intervals, allowing the car to speed up and slow down the experience. This matches the VR with the real world and synchs the virtual visuals with the real-world physical experiences, and creates an actively compelling VR experience in which, like a carnival ride, the outcome of the user does not change the experience's story or progression. Most off-the-shelf hardware can provide an amazingly rich experience for entertainment as well as for education (see figure nine).
Figure 9. Experiencing a roller coaster with VR.
Tools for Real-Time Tracking in Large Spaces
Special challenges exist when tracking users in real time in large spaces. The developers of these experiences spend a considerable amount of the main server's power assuming where the users are and where they are going to be at all times. On average, systems use about 80 percent of their processing power on the actual tracking of the subject, and the other 20 percent assuming where the user is. The following section looks at the technology in more depth, to gain understanding of the issues inherent in a system of this complexity. Users often experience VR disorientation or motion sickness as a result of bad tracking, so it's very important to get it right.
In the VOID, users wear special suits that are tracked by both visible and non-visible tracking devices. Visible tracking devices look like little ping pong balls on major joints. Non-visible sensors can include small accelerometers and gyroscopes embedded within the suits. As an example, a user raising his or her arm triggers the camera's detection of movement in the wrist, elbow and shoulder through the use of the trackers. The trackers then calculate the arm's location based on a number of cues, including size and visible intensity of light, often IR or near-IR, bouncing off the tracker. Simultaneously, the user's suit detects the same movements with the non-visible sensors embedded in the suit. These sensors detect the arm thrusting upward with gyroscopic movements in the wrist and, to a lesser degree, the elbow and shoulder. The accelerometers detect that the wrist has the most speed, helping determine that the arm is moving along the shoulder joint. This information is then sent to a main server, which moves a player avatar into the appropriate location and position. Using both of these technologies enhances the experience for the user.
Different factors complicate the effectiveness of a single tracking system in a room of this size. First, assessing the depth of the room and determining the location of the user within the 3D space is difficult. In a visual-only solution a number of assumptions are made based on the tracker data. If even one of the points is off, it might appear that the arm is going backward instead of forward. This commonly occurs when there is poor motion capture data. An even worse scenario is bad data causing the user's head to leap forward by a foot or so. This could not only cause issues with gameplay and enjoyability, it can also cause motion sickness.
From another perspective, the suit solution doesn't usually have issues with users blinking around in a scene, because it has both gyroscopic and accelerometric data to draw on. However, suit precision isn't the best, and it's typically an educated guess where the user is within a 3D space. A simple illustration of how both technologies can work together helps to clarify this point. Take the case in which the user steps forward and the visual tracker sees her move five feet ahead. The system should also detect data from the accelerometers in the suit. If both sensors don't provide the same information, however, the system may leap to the conclusion that the user has teleported. While this type of issue, for the most part, can be taken care of within systems by using deterministic modeling, it's always easier to make systems work with the data they already have, before you introduce assumptions that might cause further issues.
The possibility of occlusion illustrates another reason why a two-party tracking system is a better idea. For example, multiple users could be going through a doorway. While the doorway is entirely digital, multiple users lining up within a small space will obscure several cameras. Despite the ability to monitor the space from another user's suit or at oblique angles, a visual system can only track the users it can see. In this case, the visible systems would almost entirely fail to plot multiple tracking points within the torso area of several users. The accelerometers and gyroscopes in this instance would not be affected at all and would provide failsafe tracking.
Other solutions offer different approaches. Samsung Gear VR headsets can be synched within milliseconds of each other for a shared viewing experience. The best showcase of this version is from a company called Bigscreen. They currently have solutions for multiple platforms including Oculus Rift, HTC Vive, and Windows* Mixed Reality. Their newest offering, Alpha*, lets users share VR videos within the same room. This could amplify previous 4D experiences, which are 3D videos best seen with 3D glasses and experienced with external stimuli, such as simulated wind, fire, rain, or other elements directed at the observer during key parts of the experience. This adds another dimension of physicality to the digital medium.
Figure 10a. Experience a VR cinema on a big screen.
Adding the extra potential inherent in VR can enrich the immersive experience. Tapping into the amygdala, triggering the fight or flight response, or manipulating the vestibular system are among the possibilities. Anything that affects our presence in an environment and alters equilibrium, while blocking vision, contributes to an immersive 4D experience at a whole new level (see figure ten). This type of setup doesn't need to hide the emitters at all, aside from esthetic considerations. Imagine going to a mall kiosk and having a 4D experience where the operator spritzes you with a spray gun or blasts your face with a hairdryer. It also makes olfactory sensory triggers—smells—much easier to engage with, because the dissipation time is shorter and the magnitude of dispersal is limited. This creates a really robust system that can enhance a user's experience.
Figure 10b. Experience a VR cinema on a big screen.
Networking a Shared Experience
Having a single user in a space has its own challenges, but including secondary or even more players introduces further complications, such as multiple cords getting in users' way. Intel has partnered with several different backpack computer companies to create solutions that sidestep many of these issues. The computers are fully self-contained and some, like the MSI VR One* (see figure 11), have built-in HTC Vive breakout boxes for connecting directly to the HTC Vive headset. This has previously been handled by a secondary box connected to the system, but this secondary box creates problems, including cable length issues and fragility, as the breakout boxes come loose at times. An all-in-one solution creates a superior experience. However, even with all these capabilities, creating an effective networked experience is more difficult than one would expect.
Figure 11. The MSI VR One* backpack.
To create the illusion of realism, a user sometimes needs to be tracked along multiple data points; for example, when there is interplayer interaction. These data points can be as simple as using the controllers to provide an inverse kinematics solution. This means that the body movement is dictated by the hands locking into place, which then drives the wrist position, followed by movement of the elbow, arm, shoulder, and so on. Moving your controller back generates system updates to the mesh model, creating a fairly believable character.
This approach can be seen extensively in Mindshow* VR (see figure 12). Mindshow creates a network-shared experience that lets individuals commingle in a single space. On the other end of the spectrum are more intensive genres, such as a first-person shooter game. In these games, users have guns for which the yaw, pitch, and roll of the end of the gun needs to be updated multiple times per second to ensure good gameplay. The next challenge is that many of these games are very immersive experiences. One of the easiest ways to increase engagement is to increase the granularity, providing more in-depth understanding of all movements for precisely tracking characters. This tracking increase often includes things like hands, wrists, elbows, shoulders, heads, hips, chests, feet, ankles, and knees. Because of this, every single character that is visible to another character requires an update time of at least 15 frames per second (FPS) of actual data and another 45 FPS of predictive data, interspersed to provide the best visual quality. If a user is directly connected to a server, that user would have 67 milliseconds (ms) to update every character. The next issue that arises is network latency. The average Wi-Fi connection is pretty slow—not in throughput, but in round-trip time. On average, this round trip takes about 15 ms, resulting in an update time of 52 ms.
Figure 12. Mindshow* VR.
With the average VR experience headset time at 90 FPS (or 11.1 ms), this seems like a simpler task than tracking movement, but there is another challenge. Wi-Fi has issues with something known as jitter. When jitter occurs, the connection time between the end user and the server can be very small, about 2–3 ms, and other times it can be in the hundreds of milliseconds. Obviously, this can cause massive issues if failsafes aren't in place, including, but not limited to users running into each other. Predictive solutions are the only way to ensure an experience that meets the quality needs of the high-end interactive medium. Figure 13 shows the cost and difficulty factors in meeting networking challenges.
Figure 13: Challenges with networking in VR.
How to Structure a Multiuser VR Experience
Many challenges can affect a multiuser experience, but there are also several tricks and tips that can improve the interactive experience. One of the easiest ways to circumvent network latency issues is to use Ethernet-connected systems. The new series of Next Unit of Computing, Intel® NUC 4 × 4-inch mini PCs, provide a reliable platform for small form factor (SFF) connectivity while cost-effectively meeting all the requirements of demanding interactive VR experiences. Having SFF computers is actually a huge boon in a space where real estate can quickly become a premium. Intel NUCs can be placed on the ceiling with cables dropping down from above, reducing the need for a belay system common with other computer setups. This allows for more comfortable gameplay. When cables come from above instead of below, they don't tug on the user as much, which improves headset comfort. The lighter weight also facilitates better immersion in the user's surroundings.
One of the most efficient and immersive multiuser VR experiences is the multiroom configuration, such as the Berlin Wall VR experience at the Newseum, as shown in figure 14. The environment is powered by multiple HTC Vive systems within the same room, separated by curtains. This lets each user have their own dedicated space, but multiple users can be immersed in the same experience at the same time, with fellow contributors adding to the experience. This is the best of both worlds: Modularity and extensibility alongside an extremely immersive experience.
Figure 14. The Berlin Wall VR experience at the Newseum.
Current Use Cases
In addition to the projects discussed so far, a slew of other systems have popped up recently. The VR arcade is one such system, which is becoming more well known. With over 1,000 of these businesses throughout the United States, VR arcades successfully use off-the-shelf systems to engage users who might not want to invest in a system or are still not sure which system is right for them. Although the price has gone down on VR headsets, the cost of video cards has increased, which is now a major expense for VR computers—an unfortunate side effect of the cryptocurrency boom, which also uses these cards. The VR arcade business model gives consumers an opportunity to try before you buy. Most of these venues have food and drink, so there is a social aspect to the visit, as well as a means for engaging customers in VR culture.
Another aspect of VR that is rarely discussed is the consumer adoption of the technology. While this article is aimed and focused on LBE, the increased consumer adoption of VR technology means more opportunities for LBE itself. This includes exclusive content, or content that wouldn't be viable commercially, as well as uses in educational facilities and attractions. Most museums had some sort of VR experience in the late 90s, but as consumer adoption dropped, so did the interest in using the equipment.
Figure 15: VR arcade.
A company known for imaginative designs and colorful projects, Two Bit Circus Foundation, has created several different projects (including their own Steam* Carnival and collaborative immersive experiences with other studios) that define the epitome of LBE. Among the different experiences they announced at the 2017 Vision Summit in California were the Verizon Full Throttle* IndyCar Experience and the Piñata Party*. Their piñata game uses a bat and HTC Vive trackers to deliver a tactile and VR experience in which users have both the strength of real-world touch and a visual experience enhanced with VR technology. Their micro-amusement park spaceship game leads users through an adventure to save the universe in a fully VR multiplayer experience.
Future Tech
As the technology that drives LBE evolves, so will LBE itself. Just decades ago it would have cost millions of dollars to solve these issues, and now we are in the midst of fast-paced growth in a new revolution of VR, augmented reality (AR), and mixed reality (MR) headsets and systems. As the ecosystem continues to grow, developers, manufacturers, and enthusiasts will come up with creative solutions to the challenges that arise.
Before diving into a discussion of future tech, we'll first set the stage with some current examples of innovative technology. One example saved the government millions of dollars: Using off-the-shelf software and hardware to create flight simulators for training. In Austin, Texas, the latest addition to the United States Air Force (USAF) Innovation Hub network, AFWERX, has been launched. Designed for "Connecting Innovators and Accelerating Results," this project works with small businesses and other non-traditional vendors to create opportunities for innovation in collaboration with the Air Force. For less than $10,000 they created solutions that rival their multimillion dollar in-house system. These mobile systems also take up significantly less room than their counterparts. Just slightly larger than a pallet, these simulators can easily be moved around the facility for maintenance and connectivity.
The new solution's advanced capability supports networking of multiple simulators, so that users can perform maneuvers and engage in dogfights simultaneously. This is the true definition of innovation: Pushing the boundaries of the previous technology and adding the capabilities of many other industry verticals. What's more, all this is accomplished while reducing overhead and cost. The most effective thing about the system is its extensibility: More advanced assets can be added in a plug-and-play manner, as can upgrades of the system, all while extending the capabilities of the old simulator. This showcase of LBE extends the uses of the technology and makes it a more affordable solution package. In the coming decade, LBE might become as ubiquitous as video was in the previous decade. Taking the technological advances of multibillion-dollar Goliaths, such as the USAF, and placing them into the hands of the average developer can only boost cross-discipline operations.
Expectations in the LBE field are focused around more machine learning, computer vision, and tactile feedback integrated into virtual worlds. As algorithms decrease in computational cost, so will the use of technologies that once required a supercomputer to run. Think of a truly immersive experience in which the assets in your everyday environment can be interacted with during gameplay, seamlessly. This will clearly be possible in the next few years. You can find an example of this kind of work at Life Scaled Custom EnVRonments on Developer Mesh (see figure 16).
Figure 16. Experience Life Scaled Custom EnVRonments.
With this technology, stand-in objects for everything from museum artifacts to expensive equipment can be visually accurate without the need to be a complete replica of the real thing. They can be little more than weighted foam blocks and the illusion would hold together to support the storytelling. Think of going to a new digital museum that appears to be just a white room. Put on the goggles and suddenly you are inside the Louvre in Paris. Figure 17 shows an example of an MR experience.
Figure 17. MR experience.
Several transformative things can happen when someone is in a blank box VR experience inside a warehouse or other large-scale space with little interest. The massive advances in our understanding of visual input and movement would allow museums of this type to be much smaller than their real-world counterparts. This means that we can have small areas dedicated to transfer and dissemination of historical information at the touch of a button, opening up the possibility that one local museum is suddenly a million different museums. We can also help save many different historical marvels for study and for future generations. Several years ago, the Cairo Museum was looted and many priceless artifacts were lost in the process. Using photogrammetry or other scanning techniques to capture and preserve such precious items and displaying them digitally could allow for safe storage and more reach, as well as use of the digital objects for further study.
One technology that sounds decidedly futuristic is hard light holograms. However, the University of Tokyo recently revealed that touchable holograms are already in development. The holograms provide tactile feedback without the need for gloves or other devices worn by the user. The premise of this technology is that ultrasonic waves perturb the area around the edges of a holographic image. This lets the user perceive the object's presence physically. This remarkable technology is still a long way from commercialization; the current system can only allow users to touch fairly simple objects.
Experiences that were once fictional in games or movies will soon become part of new real-life applications, disrupting the traditional sectors with entertainment paradigms that accelerate their development even more. The future looks amazing from this vantage point with tremendous opportunity for LBE technology to escape the silos of a specialized vertical market and shift into cross-pollination mode with other industries. Entertainment is no longer just for fun; it can now be for training, education, preservation, and so much more.
About the Author
Timothy Porter is a pioneer in visual technology and the chief technical officer (CTO) at Underminer Studios*. He started his career in video games as a technical artist before moving on to endeavors as a serial entrepreneur, and later as a pipeline technical director at Sony* Pictures Imageworks. During his career, Tim has published more than 50 titles on a multitude of platforms. Driven by a mission to utilize entertainment paradigms in leading-edge technology, Tim is well-known for inspiring others to imagine the endless possibilities with technology and executing that vision. He was recognized as a 2017 Top Innovator by Intel.
Underminer Studios is using VR/AR to humanize the digital world, going to fantastic places within education, entertainment, health, and business for more engaging experiences. They design innovative products that help define the future of human-computer interaction; combining the power of analytics and visualization technologies for compelling uses. Learn more about their work with Volumetric Capture.
Resources
- Oculus Rift
- HTC Vive
- The Void immersive VR experience
- Superman Ride of Steel at Six Flags America
- Bigscreen VR
- MSI VR One Backpack PC
- Mindshow VR
- Berlin Wall VR Experience Now Open at the Newseum
- VR Arcades are gaming centers of the future
- Two Bit Circus
- Two Bit Circus Full Throttle IndyCar experience
- Two Bit Circus Piñata Party
- VR Spaceship bridge game at Two Bit Circus LA "micro-amusement park"
- United States Air Force to launch new innovation and collaboration hub in Austin
- Life Scaled Custom EnVRonments on Developer Mesh
References for Photos
- http://www.newseum.org/wp-content/uploads/2014/08/Connect_PressInfo_ID_BerlinWall.jpg
- https://upload.wikimedia.org/wikipedia/commons/b/b5/VirtualityAttraction.jpg
- https://upload.wikimedia.org/wikipedia/commons/4/45/Mixed_Reality_with_a_Virtual_Reality_Headset.png
- https://upload.wikimedia.org/wikipedia/commons/9/9a/Virtual_Reality_Cinema%2C_Amsterdam.jpg
- https://store.steampowered.com/app/382000/Mindshow/
- https://www.youtube.com/watch?v=pDY7aC8xZpA
- https://bigscreenvr.com/press/
- https://www.thevoid.com/press