Facebook today revealed some of its latest research and vision for a wrist-worn input device that the company expects to form the basis of AR and VR interactions and haptics in the future. The device, still in a research prototype phase, senses electrical signals in the user’s arm to detect intentional inputs. In addition to functioning as a simple ‘button’ input, the company says the device can even enable accurate, keyboard-less typing, and more.
In a media briefing this week, researchers from Facebook Reality Labs Research shared some of their latest work in developing new input technology which the team believes will form the foundation of interactions for XR devices of the future. The group shared a concept video of what it believes will be possible with the technology.
Beyond the concept video, the researchers also discussed the work happening to bring it to fruition.
Input on the Wrist
The Facebook researchers seem increasingly convinced that a wrist-worn controller is their best bet as an ‘always on’ wearable input device that can enable “ultra low friction” interactions for XR experiences.
Facebook has continued to build atop the wrist-wearable input technology it picked up in an acquisition of CTRL-Labs in 2019.
The heart of the wrist controller is electromyography (EMG) sensors which can detect the electrical signals which control the muscles in your hands. Rather than just course movements, the researchers say that EMG can be used to sense individual finger movements with precision down to one millimeter. In a video shared by the company, Facebook says the movements of the hand shown below are detected entirely with EMG:
While the near-term use-cases of this kind of technology could be to enable an ‘always available button’ which you can use to confirm choices presented to them through contextually relevant AR systems, the researchers say, further out it will be used for manipulating virtual interfaces and objects, and even to type without a keyboard.
“It’s highly likely that ultimately you’ll be able to type at high speed with EMG on a table or your lap — maybe even at higher speed than is possible with a keyboard today. Initial research is promising,” the company writes. “In fact, since joining FRL in 2019, the CTRL-labs team has made important progress on personalized models, reducing the time it takes to train custom keyboard models that adapt to an individual’s typing speed and technique.”
The researchers shared what is purported to be a live demo of this personalized keyboard model in action, using the wearable prototype to enable reasonably fast typing without a keyboard:
Beyond typing, the researchers say that being able to read finger movements from the wrist could also allow users to manipulate objects. Another purportedly real demo shared shows this in action:
Further in the future the team suggests that users may be able to train themselves to do some of these commands without any physical movement at all.
Although the company says it’s wrist-worn device is a “neural” input device, it draws a distinction between neural input and “mind reading.”
“This is not akin to mind reading. Think of it like this: You take many photos and choose to share only some of them. Similarly, you have many thoughts and you choose to act on only some of them. When that happens, your brain sends signals to your hands and fingers telling them to move in specific ways in order to perform actions like typing and swiping,” the company writes. “This is about decoding those signals at the wrist—the actions you’ve already decided to perform—and translating them into digital commands for your device.”
Always-on Haptics
For a device which may be comfortably worn all day, the researchers say, a wrist-wearable is also a great place to deliver haptics.
To that end, the company has been experimenting with different types of haptic technologies for the wrist.
One prototype, called ‘Bellowband’, lines the inside of the device’s wristband with quarter-sized bladders which can lay flat or be inflated to put pressure on the user’s wrist. Different haptic effects can be achieved by using different combinations of the bladders or by pulsing them at different rates.
Another prototype, called ‘Tasbi’ (short for Tactile and Squeeze Bracelet Interface), uses six vibrating actuators around the wrist, along with a sort of tension-based, wrist-squeezing mechanism which can dynamically tighten and put pressure on the user’s wrist.
The researchers say that prototypes like these help the company find out which kinds of haptic feedback technology may be worth pursuing.
Contextual AI
A major part of Facebook’s vision for AR and an “ultra low friction” input approach necessitates AI which can deeply understand the user’s context.
“The underlying AI has some understanding of what you might want to do in the future. Perhaps you head outside for a jog and, based on your past behavior, the system thinks you’re most likely to want to listen to your running playlist. It then presents that option to you on the display: ‘Play running playlist?’ That’s the adaptive interface at work,” writes FRL Research Science Manager Tanya Jonker. “Then you can simply confirm or change that suggestion using a microgesture. The intelligent click gives you the ability to take these highly contextual actions in a very low-friction manner because the interface surfaces something that’s relevant based on your personal history and choices, and it allows you to do that with minimal input gestures.”
That’s largely conceptual for now. While today’s smartphones or smartwatches are able to leverage some clues like, time, location, and connected accessories to infer what actions might be relevant to you, the sort of contextual AI suggestions that Facebook is envisioning will require both an advance in AI as well as the sensor-laden peripherals that can get a real-time understanding of the user’s immediate environment.
More Questions Than Answers on Privacy
Facebook says its goal in building the far future of XR is to build technologies where “the human is the absolute center of the entire experience.”
Achieving the company’s vision will require hardware and software with an intimate understanding of both the user and their environment.
Facebook maintains that it’s committed to transparency throughout the development of these technologies, but admits that it isn’t equipped to asses the broader questions.
“Understanding and solving the full extent of ethical issues [raised by these technologies] requires society-level engagement,” says FRL Research Science Director Sean Keller. “We simply won’t get there by ourselves, so we aren’t attempting to do so. As we invent new technologies, we are committed to sharing our learnings with the community and engaging in open discussion to address concerns.”
Indeed, the company says that a major reason why it’s sharing this information today is to engage the broader tech community on these questions before it moves to take the technology out of the lab and into the market.
“[…] we support and encourage our researchers to publish their work in peer-reviewed journals—and [that’s] why we’re telling this story today. We believe that far before any of this technology ever becomes part of a consumer product, there are many discussions to have openly and transparently about what the future of HCI can and should look like.”
