The sense of Presence

The word that is on everyone’s lips when describing VR (Virtual Reality) is  “immersion”.  A cool word indeed. To achieve this ultimate goal however, developers need to simulate the perception of being physically present in a virtual environment, and we call this the sense of presence. “Presence is a psychological state or subjective perception in which even though part or all of an individual's current experience is generated by and/or filtered through human-made technology, part or all of the individual's perception fails to accurately acknowledge the role of the technology in the experience.”  (The International Society of Presence Research - ISPR)

Creating this level of true immersion is a real challenge. While graphics already made tremendous improvements in the past few years, audio still has a long way to go in regards to reaching its true potential of playing a major role in this new and still evolving field of VR.

Spatial immersion and multi-sensory uses in VR gaming

In console and PC games, immersion can be separated into three categories: narrative immersion (players being invested in a story), tactical immersion (players performing tactile operations that involve skills), and strategic immersion (more cerebral, associated with mental challenges).  In VR however, a fourth category is added: spatial immersion, which convinces the player the artificial world surrounding them is real.  Among the former three categories, spatial immersion is also of great importance when it comes to triggering the sense of presence and delivering a believable and immersive environment.

To position ourselves physiologically, our visual system is used to help guide our bodies within our environment, our ears provide crucial information about our surroundings, and smell provides complimentary surrounding information.

Audio goes beyond vision in the sense that it is not limited to our field of view, but rather provides a 3D-audio feedback (including 360-degree, height and depth) information about our surrounding environment.

In order to create realistic spatial immersion, ideally, we would want to be able to use all of our five senses and have full sensory processing (process of using our bodies effectively within an environment). The problem at the current stage in VR development is that we are unable to utilize all five senses. Sight and hearing are used as the primary senses, touch is not used very effectively, and taste and smell are not integrated at all at this point. With that in mind, audio is perhaps the most powerful medium we can develop in order to “hack” the user’s brain and create a complete 3D-audio experience that offers spatial immersion, as it gives the representation of a realistic space by providing virtual sources (which can’t be necessarily seen in the scene or action) rendered through a simulated acoustic environment.

3D-audio experience to create spatial immersion

The first step to enhancing spatial immersion through audio realism is audio positioning. A pattern emerging from recent studies (Skalski & Whitbred, 2010) showed that better sound positioning in games (surround VS stereo) had a much higher impact on the player’s spatial immersion and enjoyment than visual quality (HD VS standard).

The brain uses subtle differences in loudness, tone and timing between our two ears to localize sound sources in three-dimensional positions (horizontal, vertical, and distance). In VR, viewers are literally in the middle of a scene, so the process of determining the location of sound sources in a simulated acoustic environment is the most important audio improvement towards creating natural listening experiences.

 

Early in media history, we discovered that the human ears pick up audio in three dimensions and the first binaural recording goes back to 1881.

“Binaural recording is a method of recording sound that uses two microphones, arranged with the intent to create a 3-D stereo sound sensation for the listener of actually being in the room with the performers or instruments. This effect is often created using a technique known as "dummy head recording", wherein a mannequin head is outfitted with a microphone in each ear.” (Wikipedia, “Binaural Recording”)

In games, 3D-audio should be created with real-time processing since we can’t anticipate where sound objects will be located corresponding to the player's location and head movements.  So, about a century after its invention, binaural recording is now being revisited with binaural processing, and engineers and investors are working hard to develop the best binaural algorithms suitable for VR.

Indeed, because of its nature in regards to the required use of stereo headphones rather than speakers to replicate 3D positioning (necessary in VR to isolate the player from the outside world and have audio playback that matches head/ears movements relative to the field of vision), binaural has been logically chosen by the professionals as the new standard for 3D-audio processing instead of the usual surround techniques (limited to horizontal and distance audio positioning).

Conclusion

Many audio challenges, considerations, and needs are being exposed as we work with VR.  By exploring these further, we will be able to push the boundaries of the VR and gaming industry altogether.  And, although the gaming industry has been improving visual quality in the last generation of consoles, with the VR development on a rapid rise, the mindset within the VR industry is quick focusing on audio as the VR community is realizing that better audio realism will increase the player’s enjoyment and engagement levels, elevate the state of immersion, and ultimately, the sense of presence.

3D-audio positioning is just one of the areas where audio improvements are being pursued as a direct result of VR developments.  Audio working tools are also in continuous development with new technologies catering to VR needs and its uniqueness when it comes to the mixing techniques required, in order to enhance VR experiences.

Audio is the next medium for the industry to explore and focus on, and VR is certainly helping put audio the spotlight!