A project from researchers at Reality Labs, Meta’s XR division, shows a method for interacting with the real-world through augmented reality.
I’m about as seasoned as they get when it comes to the XR space. I see research papers all the time that are interesting but ultimately one more step along a predictable path. It’s rare that I come upon something that I’d never even conceptually considered. But this new Reality Labs research project is exactly that.
In fact, it’s so different that it’s kind of hard to explain.
Researchers at Meta Reality Labs Research and University of Duisburg-Essen devised a method for making real objects virtual so they can be interacted with in real-time. By not only digitizing the objects, but also seamlessly erasing their real counterparts, the method creates a mind-bending blend between the real and virtual worlds. The video below with examples explains it better than I possibly could with text alone:
The researchers—Mohamed Kari, Reinhard Schütte, and Raj Sodhi—call this concept ‘Scene Responsiveness‘ and published their work late last year at the ACM Symposium on User Interface Software and Technology.
As the authors put it, this method creates a “visual illusion that virtual actions affect the physical scene.”
What makes this so interesting? We’ve all seen AR demos where virtual objects seem to float in your room, attach to your walls, or even walk ‘behind’ objects. And we’ve seen virtual objects that seem to abide by the physics of the room by rolling across a table and then correctly falling off when they reach the edge.
But digitizing the real objects in the space to give the user virtual control over them—while erasing the real object from where it once sat—that’s a damn interesting trick. Perhaps even more powerful is when virtual characters in the scene appear to interact directly with objects in the real world.
I quite appreciate that the authors even took the next step and showed how to prevent the illusion from breaking.
Giving the user control over an (apparently ‘real’) virtualized object, but not allowing them to get close enough to actually touch the object, means the expectation of tactile feedback can’t be broken. And another smart detail is preventing users from bumping into things that have been erased from the scene by making them reappear only if they get too close.
What’s been shown in the work is visually imperfect and largely a proof-of-concept that relies on lots of manual, scene-specific work to make possible. But with the rapid advancements of computer vision and artificial intelligence, it’s not difficult to imagine how such a system could be built in an automatic and scene-independent way, and could be very convincing.
The authors have some surprisingly creative ideas for practical applications of this type of AR illusion, including “for health, we imagine as the user reaches out to a physical unhealthy chocolate bar, it morphs into something less appealing such as a spider. Or, the chocolate bar grows legs, runs away, morphs into a banana while running, and rephysicalizes at the location of a physical banana.”
So if, in the future, we find ourselves chasing down a magic chocolate bar only to find that it turned into a banana by the time we reach it—we’ll know just who to blame.