MindMaze, a neurotechnology startup, is developing a simple, low-cost approach to face-tracking called MASK—claimed to be compatible with any VR headset—that can map significantly more life-like expressions onto your virtual avatar. I recently met with MindMaze at their San Francisco working space to test Mask for myself and learn more about it. Since optical, computer-vision based face-tracking is made more challenging in VR due to the headset blocking a significant portion of the face, Mask instead employs low-cost electrodes around the periphery of the headset's foam padding to sense the electrical signals generated when you move muscles in your face. [caption id="attachment_63810" align="aligncenter" width="640"] Photo courtesy MindMaze[/caption] Speaking to MindMaze CEO Tej Tadi, I was told that the embedded sensing hardware is very low-cost and adds little to the headset's overall cost of manufacturing. The challenge is not the hardware, Tadi says, but the software which is tasked with interpreting the input. [caption id="attachment_63813" align="aligncenter" width="640"] Each graph here represents the electrical activity detected by the electrodes in the headset. Decoding this data into something useful is the challenge. | Photo by Road to VR[/caption] The electrodes, which make direct contact with your face through the headset's foam interface, only measure electrical amplitude. With eight electrodes embedded on the prototype version of the Mask that I tried, the only thing the computer "sees" is eight incoming data streams that go up or down on a graph. Tadi says that MindMaze has applied their neurotech knowledge to create an algorithm that can read that data, and, from it, extract a set of facial expressions. Neuro-bamboozling vs. the Real Deal Now I should be clear up front that I'm extremely skeptical when it comes to neuro-bamboozling: that's when companies use 'neuroscience', 'neurotechnology', and other brain buzzwords to make it seem like they're doing something more significant than they really are. One oft-seen example of neuro-bamboozling is when a company might tell you something like, "you can fly a drone with your mind!," which, 99% of the time, means they're going to slap some electrodes on your head and tell you to "concentrate", which will make the drone go up, and then to "relax" which will make the drone go down. It's highly binary, and ultimately not very useful. Nearest I can tell for MindMaze—at least for Mask, as I haven't seen their other products—they are the real deal. When I strapped on the headset to try Mask for myself, even without any calibration, a range of canned expressions that I made were quickly reflected on the face of an avatar that represented me in the virtual world. When I smiled, it smiled. When I frowned, it frowned. When I winked, it winked. It was easily the best calibration-free tech that I've seen of this sort. https://gfycat.com/SlimyHarshBluefintuna I was told the prototype presently supports 10 different facial expressions. Because they are canned poses, which means Mask can only provide approximations of your expressions; it won't be able to capture the unique movements of the face that make you, you, but it does cover the basics. [irp posts="62434" name="Vive to Get Eye-tracking Add-on with Optional Corrective Lenses"] In addition to the 10 current expressions, the team says they're working to suss even more expressions out of the data, including the potential for rudimentary eye-tracking, which wouldn't be precise enough for things like foveated rendering but could be good enough for expression mapping. Continued on Page 2: Not Perfect, But Promising » Not Perfect, But Promising https://gfycat.com/UnacceptableMessyIndianjackal Mask wasn't perfect by any means. I saw the occasional misinterpretation of data which could cause my avatar's face to do something that I wasn't actually doing. When it came to normal blinking (not winking), the prototype didn't seem to pick up my blinks very well. However, Tadi claims that the algorithm gets smarter and more robust over time individually for each user as they use it. How much better it can get over time is the big question. It's going to need to be pretty darn close to 100% because users will need to be able to trust the system to represent them accurately. Facial expressions are a hugely important part of unspoken communication; you wouldn't want to be in an important virtual meeting with a potential new client and look as if you're sneering at them (due to misinterpretation of your facial expressions) when you're actually smiling. And even outside of important meeting scenarios, face-tracking in VR still needs to be highly robust so that it doesn't lead to awkward facial movements that end up dropping your avatar into the uncanny valley. The ability to improve Mask's robustness from 90% to 99% will make or break the applicability of the tech. An Unfair Advantage While computer-vision based face-tracking has the potential to be much more precise—capturing subtle changes in the face rather than just being able to jump between a few poses—even if the headset didn't pose a significant challenge, this electrode-based approach has an advantage that computer-vision based face-tracking physically cannot: prediction. Tadi claims that the Mask algorithm can, via the electrical signals being read from your face, sense the expressions you're going to make "20 to 30 milliseconds" before you make them. If true, that potentially gives an advantage to this method, opening the door to essentially instantaneous mirroring of your expression in the virtual world. [caption id="attachment_63814" align="aligncenter" width="640"] The MASK prototype currently requires electrodes attached to earlobes in addition to those that contact the face, though the creators say these would be built into on-headset headphones[/caption] The electrode/algorithm approach to face-tracking is also interesting because it doesn't require much processing power, Tadi says, and the workload can easily be performance on a VR headset powered by a smartphone. That's in contrast to the computer-vision approach which requires more heavy-duty processing to derive your facial expression from each frame. - - — - - So, Mask seems promising. What's next? Tadi says that the underlying technology is compatible with any headset, and that the company is in talks with all the major players in the market. Ultimately the company seeks not to sell the hardware as an add-on, but offer a simple blueprint for direct integration and then license the algorithm. We could see a headset with this tech integrated as early as holiday 2017. [irp posts="61499" name="Eye-tracking Glasses Give Glimpse into Professional Pianist's Perception"] While I think that VR is destined to one day incorporate face-tracking, it's not yet clear to me exactly how precise and accurate it needs to be in order to be a must-have feature (and one that doesn't risk misrepresenting the user). Given the current state of the industry, bringing headset costs down is a greater priority right now than adding more features (and cost), which could mean tough timing for Mask.