For immersive technologies like AR and VR to be effective, they need to understand not only about our world, but also about us. But we don’t even understand us all that well yet; as we capture more biometric information on our quest to lose ourselves in augmented and virtual realities, we’re learning more about ourselves too.
Last month we wrote about a Disney Research project which used virtual reality to study human perception. The task—catching a ball—was simple, but the reality behind how the brain and body coordinate to get a simple task done can often confound our intuition.
Now Function has teamed up with eye-tracking specialists Tobii to equip a professional pianist with eye-tracking glasses to see what happens (see the video heading this article). Pianist Daniel Beliavsky sets about playing some complex arrangements, and the results are quite interesting, and also revealing about how much we have yet to learn about our own perception.
In the video from Beliavsky’s perspective, a yellow dot shows what his eyes are fixed on. Rather than looking constantly at his hands to ensure each note is landing correctly, his eyes are darting about, often leading the hands to the next location on the keyboard, but also frequently returning to the point between his hands where he’s able to gather information about both at the same time with his peripheral vision.
This video highlights not just the potential usefulness of eye-tracking for input in immersive technologies, but also the challenges of using the raw data to understand intent. While we can use eye-tracking input for passive things like foveated rendering, using it for interaction is a much more complex problem.
Using the input from Beliavsky’s eyes alone, it may be possible to predict the next likely moves, but because of the complexity of how the brain, hands, and eyes are interacting in this case, doing so may come with extreme difficulty.
If the computer had to guess what the most important thing was to Beliavsky while he was fixated between his hands, it might guess it’s the keys between his hands. But actually Beliavsky was using a combination of muscle memory and peripheral visual queues to make his performance work, while at times the point at which his eyes were directly fixated was not important at all. The raw data in this case betrays the intent of the user. Intent represents a major challenge for the usefulness of biometrics beyond simple passive input and data collection.
The more we learn about human perception, the deeper our virtual and augmented worlds will be able to immerse us. One day, it’s feasible that we might be able to fully emulate input and output of human perception, but today we’re at just the beginning of that journey.
