After 2 years of Oculus putting their collective brains and talents to solving the “hard problems” of VR displays and low latency head tracking, at Oculus Connect this year, we heard a new topic beginning to garner attention: VR audio. Oculus’ recently revealed ‘Crescent Bay’ prototype is the first from the company to include integrated headphones. Well aware that audio in VR needs to be more than mere stereo for Presence, the company also announced that they’re licensing a 3D spatial audio processing technology called RealSpace 3D Audio from Visisonics.
3D Audio From Stereo Headphones
Audio is absolutely vital to the dramatic impact of almost all forms of entertainment. Movies and games in particular rely heavily on sound to jack into our subconscious in ways that can enhance, transform, or subvert any accompanying visuals. There are few things as emotive as an evocative and powerful movie score or as creepy as an expertly engineered suspense soundtrack.
Mimicking the way we experience sound in everyday life and applying that level of realism to entertainment has been a pursuit that has long since taken a back seat behind improving visuals, but countless techniques and technologies have evolved over the years that attempt to do just that.
Oculus made it clear last month at its inaugural developer event ,’Oculus Connect’, that audio is in their sights. Not only was the new Crescent Bay prototype revealed sporting top-quality integrated headphones, but also a high quality audio pipeline utilising RealSpace 3D Audio, a proprietary 3D audio mixing and processing technology developed by a company called Visisonics.
On stage during Connect, Oculus CEO Brendan Iribe said that the company is working hard on making audio as immersive as the rest of the VR experience.
“…we’re working on audio as aggressively as we’re working on the vision side. We have a whole team ramped up,” Iribe said. “as part of this (audio) initiative, we’ve licensed RealSpace 3D’s audio technology, a high fidelity VR audio system developed over 10 years…”
Visisonics offers a wide array of solutions for capturing, mixing and processing 3D sound stages, including a neat ‘Audio Panoramic Camera’, an 8-inch sphere with 64 mics and 5 cameras embedded for capturing panoramic video with expansive sound to match.
The company has close ties to the University of Maryland, an institution attended (and subsequently dropped out) by Oculus CEO, Brendan Iribe. Iribe recently donated $31 million to the university to fund scholarships and the expansion of their computer science facilities. Visisonics are a spinout company from the school, currently holding exclusive licensing rights to technology developed on campus. They’ve focussed on the capture, mixing and processing of spatial audio, specialising in systems based around HRTF (head-related transfer function)
RealSpace 3D Audio in Action
Clearly Oculus liked what they’d heard in RealSpace 3D’s techniques and algorithms, so I went hunting for examples of the technology in action. Truth be told, no hunting was required as RealSpace 3D not only have 2 demos available for anyone to download and try for themselves but one of them is a VR specific, DK2 enabled version of the now historic Tuscany VR demo.
The ‘standard’ demo is a fairly simple demonstration using a first person view (complete with gun for some inexplicable reason). You’re treated to sound effects and music played first without then with RealSpace 3D processing enabled. Had I left my RealSpace 3D experience at this video, I’d have been distinctly underwhelmed, frankly.
Luckily, the Tuscany demo was an entirely different story.
I’ve listened to a lot of surround sound systems and I’ve experienced a lot of DSP/processed 3D audio techniques plus countless binaural tracks, but this demo genuinely set a new benchmark for me.
The demo is identical to the Tuscany experience we’re all familiar with except that objects within the world act as focal points for demonstrating a facet of RealSpace 3D Audio’s techniques. Approach the fountain (flashing red, somewhat disconcertingly) and you’ll be treated first to the sound of the fountain. Nothing special perhaps, but when you pan your head, the sound of cascading water is anchored in 3D space exactly as expected, including when directly behind you. This effect is somewhat overcooked, but nevertheless impressive.
Wander further and you’ll spot a butterfly marked in red, walk toward it and the thudding sound of a helicopter follows the insect’s path. As it pans over your head, the vertical 
positioning is uncanny and I found myself circling around marvelling at the positional accuracy relative to my view into the virtual world. Very impressive, if somewhat abrupt and in your face.
My favourite by far however comes when venturing into the house itself. You’ll find an somewhat incongruous loudspeaker hovering adjacent to the stairs; blaring music just above head height, the effect brought upon something I can only describe as ‘aural presence’—that is my brain just accepted the audio almost completely. I was in that room, listening to that music and it didn’t just sound realistic, it sounded real. I just walked around the villa, positioning myself between the almost tangible crackling fire and I just sat, slack jawed at how good it sounded.
Gathering myself, I realised that what I was hearing wasn’t just good audio positioning, but the sources seemed to be reflected and occluded as you’d expect in a real space. That is, when you move near to the wall opposite the speaker, the experience is subtly different than if you’re in the middle of the room. Moving outside, the front of the villa, audio seemed to emanate from the upstairs window in a way you’d expect. Now, this could have been the powerful visuals coercing my brain to correct for this, but either way it sounded great.
The demo is somewhat heavy handed in its presentation. The audio sources are faded in too abruptly (although I can see why, given the lack of virtual space), and aside from some incidental environmental audio provided by birds etc. there wasn’t a cohesive sound stage that matched the surroundings. But this is to be expected from a tech demo such as this. I’m looking forward to seeing a properly mixed scene.
As a quick hammer blow to the senses and to illustrate just what can be done with accurate, realistic audio married with VR, it’s probably unrivalled right now. I can certainly see why Oculus decided to license this technology, it shows immense promise.
You can grab the demos over at Visisonic’s Realspace 3D Audio page here. We’ll be endeavouring to discover more about RealSpace 3D Audio and Oculus’ teased audio pipeline as soon as we can.
