One of Facebook’s underlying goals for VR is to use it as a means of connecting distant people. While today friends can talk and play in a range of social VR applications, including Facebook Spaces, the representation of users in VR is still a caricature at best. At F8 this week, Oculus showed work being done on hand-tracking to bring more intuitive control and accurate avatars into VR.
Maria Fernandez Guajardo, Head of Product Management on Core Tech at Oculus, this week shared glimpses of her work, which she describes as turning the company’s various research projects into practical technologies for future products. Beyond hand-tracking, she also revealed the Oculus ‘Half Dome’ prototype, a headset with a 140 degree field of view and varifocal display.
Guajardo detailed a computer-vision based hand-tracking system trained with a self-optimizing machine learning algorithm, which she says achieves tracking that’s “far more accurate than any method before for tracking a single hand, two hands, and hand-object interactions.” Footage which appeared to show the hand-tracking in action (above) also appeared to show detection of snapping gestures.
Guajardo explained that the company used a marker-based tracking system to record hand interactions in high fidelity, and then condensed the recorded data into 2D imagery which allowed them to set a convolutional neural network to the task of uniquely identifying the positions of the markers across a large set of hand pose imagery, effectively allowing the system to learn what a hand should look like given an arbitrary set of marker positions. Ostensibly, this trained system can then be fed markerless camera input of a user’s hands and solve for their position.
By measure of something Oculus labeled ‘Tracking Success Rate’ (and also described as “accuracy,” the company claims to have achieved a rather astounding 100% success rate with single hand-tracking, up from the claimed 90.49% of other hand-tracking methods. The company claims even bigger leaps compared to other methods for two-handed and hand-object interactions.
Hand-tracking can enable rich, controller-less input, which could be especially suitable for mobile VR headsets where carrying separate controllers would hinder portability. While the accuracy achieved with Oculus’ approach is certainly impressive, it isn’t clear what the computational constraints are at this stage.