There’s no doubt that what Oculus and Samsung managed to achieve with the mobile Gear VR headset is impressive, but one aspect, positional tracking, lags far behind desktop VR headsets. Oculus CTO John Carmack has affirmed his team’s focus on adding this missing piece to an otherwise excellent mobile VR package.
Not that long ago there were many that were skeptical that, in the wake of the largely (and unsurprisingly) disappointing experiences offered by Google’s unashamedly low-rent Cardboard movement, that Samsung could pull off a substantially better experience with its Samsung Galaxy Note 4 powered mobile VR headset, the Gear VR. However, the combination of Oculus’ VR engineering prowess and Samsung’s proven industrial design expertise produced a so-called ‘Innovator Edition’ Gear VR that really did deliver a compelling VR experience.
But, despite managing to pull off low persistence and 60 FPS mobile VR on their very first try, one aspect of the experience lagged behind the rapidly evolving desktop PC VR headsets: positional tracking.
Positional tracking tracks a player’s head as it moves through 3D space, and is a major benefactor to comfort and immersion. Tethered PC headsets have achieved high performance positional tracking using an ‘outside-in’ technique: placing stationary sensors away from the user to watch the movement of their head.
However, this ‘outside-in’ style of positional tracking doesn’t make a lot of sense for mobile VR, whose entire selling point is the ability to take it and use it anywhere (and without wires); you wouldn’t want to carry a wired positional tracking sensor everywhere you go and be forced to set it up whenever you want to use the headset.
So the more appealing method of achieving positional tracking on a mobile VR headset flips to the ‘inside-out’ concept: use sensors on the phone to watch the movement of the environment (and thus understand the movement of the player’s head through space). But it turns out that achieving this with the precision necessary for virtual reality (‘sub-millimeter accuracy’ as you’ll often hear) is particularly challenging and still considered an unsolved problem.
Although we know that Oculus CTO John Carmack had been working on the issue for some time, Oculus and Samsung hadn’t achieved a viable solution by the time they wanted to ship Gear VR.
So Gear VR went out the door with rotational tracking only. Developers have been able to design around this limitation for Gear VR apps quite effectively, but it’s widely agreed that positional tracking would add significantly to the mobile VR experience.
We haven’t heard anything about positional tracking since Carmack was mobbed at ‘Connect’ Oculus’, developer conference last year, prompting one of his infamous corridor mini-lectures. Carmack elucidated then on what may or may not be on the way for mobile virtual reality:
I spent a while on that and I’m confident I can do a good job with stereo cameras, so we’re left with the problem there of; if you put them in the headset then either you need to have processing in the headset and build a whole ‘other system there, or you need to push all the data through USB3 to the phone which is going to take up a lot of power … so it does not look good for making an inside-out tracking system that doesn’t use a whole lot of battery power.
Now, in a response to a question via twitter put to Carmack on the subject of his pet ‘vrscript’ project, he responded that he’s been too busy on other positional tracking work to spend time on it.
As Carmack alludes in the Connect statement above, having a set of stereo cameras, allowing the calculation of depth in the real world, seems a bare minimum here, but after you’ve managed to convince a mobile phone manufacturer to refactor their smartphone hardware to include two, very fast (by mobile standards) imaging sensors and perhaps a co-processor to assist with low power computer vision related calculations, you still have to stream and interpret that data inside an application already overburdened by the demands of mobile VR rendering. Hopefully, the pace of mobile processing speeds with successive generations of phones will assist here, and this is where Samsung’s S7 mobile phone could play a major role.
Carmack seemed frustrated at Connect at how little resources Oculus had devoted to solving this significant problem.
“It bugs me a little bit, we have like 30 computer vision experts at Oculus form the different companies we’ve acquired and none of them just wanna go solve this problem, they’re all interested in their esoteric kinda researchy things while this is a problem I want solved right now,” he said. “I wish someone had spent all last year on it.”
Carmack thinks it’s solvable, but he said at the point “I still don’t have a ton of support within the company.” Perhaps as the consumer Oculus Rift is locked down, with the consumer Gear VR already on the market, Carmack has managed to persuade some of the computer vision resource to work for him these past few months.
So when might we know more about positional tracking for Samsung’s Gear VR? The next major event for the mobile industry is Mobile World Congress and Samsung’s accompanying ‘Unpacked’ event. It seems likely that Samsung, perhaps anticipating a growing tide of interest from its competitors in VR, may tell us more about next gen mobile VR.