Vector Not Raster
Before the MTS demonstration, McCauley explained why he thinks this cameraless approach is important. Primarily it’s about range and cost.
McCauley said that while he was at Oculus and the company was working on their first camera-based system for the DK2, he quickly picked up on the range problem. The company had talked for a long time about room-scale capability, and McCauley didn’t see the camera approach as sufficiently scalable to those distances. He explained that the range of a camera based approach is limited by the image sensor which is raster-based.
The Rift DK2 camera has a resolution of 752×480. The headset of a user sitting just a few feet away can only be seen by a small portion of those pixels (as the view of the headset only takes up a portion of the total pixels that comprise the scene). As you get further away, the headset is represented on fewer and fewer pixels which means the computer has much less data to work with, McCauley says.
You can think of it like this: if at 8 feet from the camera the headset only takes up 94×60 of the 752×480 sensor, it’s essentially like trying to track the headset with a 94×60 pixel camera up close with the headset filling its entire field of view. The further away you move the headset, the lower resolution your camera becomes (in a sense); there’s no effective means of zooming the camera in when the headset is at range so that it can use more of its image sensor.
Several tricks have been devised to counter this reduction in available pixels at range, like dynamically boosting the LED brightness to create a larger light source for the camera to spot, using the flashing of LEDs to glean additional information about the tracked object, and utilizing dynamic exposure of the camera. At a certain point however, the resolution of the camera-based tracking becomes the fundamental range-limiting factor.
The obvious fix then is to increase the resolution of the image sensor, but that racks up cost quickly and USB bandwidth becomes a bottleneck, McCauley says.
So he opted for a vector-based approach; one which would not be stuck with a set resolution, meaning that, in theory, it could track with equal precision at 5 feet or 50 feet. McCauley says that Kris Pister, the professor who pioneered the tracking algorithms in MTS, has used a similar system to track a drone in the air more than one mile away (though I would guess at that range we’re far removed from the realm of ‘lasers you can legally point at a person’).
Because MTS only has to stream the values of the angle of the laser, the solution is very low bandwidth compared to sending and processing a high resolution image at 60Hz or more, says McCauley.
Beyond Proof of Concept
The system isn’t perfect. There were plenty of times where I saw it lose tracking, and it isn’t integrated with any apps at this point so I wasn’t able to actually look into a headset and see how precise the tracking was. But McCauley’s goal is only to demonstrate the concept, and it appears he’s well on the way. There’s still tons of room for optimization to get the system working in tip-top shape. Ultimately though, he doesn’t intend to be the purveyor of MTS.
“I’m gonna let someone else [commercialize it]. What I’m gonna do is put the system together to let someone else try to get this to work. I can get the components… the companies on board to provide the hardware to build the thing and get it debugging in some rudimentary form,” says McCauley. “But to get it actually integrated with an application? I don’t ever intend to do that. I’m just going to make this thing to prove it can be done. That’s the only interest I have.”
When I press him on this, he says he has no interest in spinning up a company for the technology. He seems happy to be taking a break after Oculus, and has plenty of work left to do on his Lola T70. But it doesn’t sound quite like he’s doing this as an academic endeavor, where he’ll simply publish his findings for just anyone. Instead, McCauley is considering looking within his network to find the right partners to make MTS a reality.
“I have access to all the foundries and stuff and the silicon which is high value. And I have enough friends that if I say ‘that thing is gonna go’ or ‘we’re gonna do this’, they’ll be on board,” he tells me. “If you’re at a small startup somewhere—even a medium sized startup—you’ll have a tough time getting people to [take the risk on you to get this built]. All the engineering that goes into making this is an enormous expense, but it’s already kind of done [referring to the foundries that craft the MEMS devices]… to get those kinds of resources is very hard to do for a small company but I’m pretty well connected.”
