Sony Interactive Entertainment (SIE) researchers unveiled a prototype motion controller recently which showcases experimental techniques into full hand tracking using machine learning and capacitive sensors.
Vision-based hand tracking systems like Leap Motion are great for tracking hands when you’re not holding anything like a motion controller, which can typically occlude finger movement, therefore breaking the illusion that your hands are really 1:1 represented in virtual reality.
In an effort to bring greater hand presence to VR motion controllers, SIE researchers Kazuyuki Arimatsu and Hideki Mori set out to build a prototype controller which integrates a host of capacitive proximity sensors—much like the ones you’ll find under the hood in Valve’s Index controller (aka ‘Knuckles’) and Oculus Touch.
The prototype houses 62 electrodes which Arimatsu and Mori used to record training datasets from a number of different hand sizes and and poses. Like the Index controller, Sony’s prototype features an open-palm design using a wrist strap.
“We have introduced 2.5D representation to apply convolutional neural network methods on a capacitive image of the curved surface, and two types of network architectures based on recent achievements in the computer vision field were evaluated with our dataset,” the paper states. “We also implemented real-time interactive applications using the prototype and demonstrated the possibility of intuitive interaction using fingers in VR applications.”
The video (above) shows off some pretty impressive results, however Arimatsu and Mori admit there are a few drawbacks to their design currently, namely its lack of support for all hand sizes.
“This could be problematic if we deliver our method to users worldwide. Per-user fine-tuning training [demonstrated in the video] cannot be conducted with our current training method realistically since we require an optical tracking system for recording the training dataset. For better adaptation for various hand sizes, a method for augmenting or synthesizing the capacitive image needs to be studied.”
The current algorithm can’t estimate the shape of a user’s hands, such as the thickness of the finger, which can cause a gap between the actual hand pose and virtual hand pose when performing things such as pinching actions. Users can of course self-adjust when picking up smaller virtual items, although this has admittedly been true for Knuckles as well as the original Oculus Touch controller, which was the first major VR controller featuring a comparatively smaller array of capacitive sensors.
Whatever the case, it’ll be interesting to see if Sony continues forward towards commercialization, be it via Sony-built PC solution or controller intended for use with PS5. We’re still waiting with bated breath on an announcement of the next-gen PSVR, which we really hope ditches the aging PS Move and PS Camera optical combo to say the very least.
