James Andrew, Founder of PixelRouter VR and creator of Z0NE, explores the challenges virtual reality developers face when trying to transcend the boundaries of users’ physical space for the purposes of virtual reality. He shares his experiences at NYU’s MAGNET ‘Holodeck’ and some of his research into creating ‘extra-dimensional’ VR space.
A few months back I had the opportunity to begin collaborating with Dr. Ken Perlin and his team at NYU MAGNET (Media and Games Network), on a project that has come to be known as ‘The Holodeck’.
Ken had a fantastic idea. He combined a professional motion capture stage with Samsung’s mobile Gear VR headset, yielding a 600 square-foot room with full positional tracking. In other words, it’s a research space that’s way ahead of anything close to consumer level. Not only this, but we can add retro-reflective markers to real world objects in the space so that we can populate virtual worlds with real, touchable things, and then interact with them in VR.
Ken has gathered a team of some particularly driven and creative individuals who are working together to devise and create new experiences for the space. When he asked me if I’d like to contribute some of my own ideas, I didn’t have to think very long before saying yes! So I started thinking about what sorts of things I’d like to do in VR that might be possible in this new space.
Just before I got there, the team had undertaken the brilliant and obvious task of creating a virtual representation of the real physical space. As Ken says, “How are you going to create new virtual spaces if you can’t even create the one you are physically in?” I love this idea of using a virtual representation of your real surroundings as the gateway into other virtual places.
What do you see in VR as soon as you put on the Gear VR? You see walls. And there it was, the most obvious first step that I just had to take—let’s do something about these walls. That was the idea behind my first contribution, in the form of an experience I called ‘Sunroof’.
The idea is simple. Let’s roll down the walls like windows, revealing a much more expansive outdoor environment. And let’s make it the surface of a huge lake under sunny skies. And let’s shrink the floor so you find yourself on the middle of a small floating dock. And let’s put a single huge Koi fish in the water, swimming circles around the dock.
Now, something like this can be pretty cool in VR, even without a motion tracked room. Even on the Gear VR alone, it is pretty cool to see walls open up revealing another world. However, I was not prepared for the level of wow that I experienced the first time we tried this in The Holodeck.
You see, having your head tracked perfectly, anywhere you go in the room, and in any position, fundamentally changes this experience and it becomes amazing. Now I was able to walk to the edge of the dock, kneel down, and peer over the edge into the water where I saw the Koi swim by just inches from my face. Now I could lie down in the center of the floor and look up at the sky moving slowly overhead.
This was all very cool, and it represented one way of taking our minds off of the fact that we are inside of a relatively large, but nonetheless walled room. In this case, by contracting the floor into a small dock and surrounding it with water, the walls recede into irrelevance for the duration of the experience. We could go all the way to the edge of the dock and look out into the distance.
But how might we connect virtual spaces so that we could walk from one to another? Technology is not close to addressing the fundamental limitation of the real physical walls. So how can we feel like we travelled from one place to another place?
A few weeks ago Ken introduced a very cool concept to the group, which came out of a MAGNET faculty brainstorming session. Here’s the idea: What if we could place a mirror on the wall in VR, and then when you walk up to the mirror, you see yourself reflected, but in another world? And then, by some gesture like reaching out and touching your mirrored hand, the worlds change places. You now see yourself reflected in the place you just came from, and then when you turn around, you are in the new world that was formerly in the reflection.
I decided that the technical aspects of pulling this off were a little beyond my reach at this point. It requires tracked avatars and some pretty creative coding around the reflections, especially considering the target platform is the Gear VR. Don’t get me wrong, I can’t wait to try this, but my executive producer, who is also me, won’t allow it.
In Ken’s idea, a mirror is on the wall, and it is something we are used to walking up to and then turning away from. This feels natural. I think a key aspect of any good idea about making the Holodeck feel bigger is going to be coming up with creative ways to make your player change direction. We can’t walk more than about twenty four feet in any one direction, so how can we work with this limitation and make it feel the natural thing to do, to turn around?
I began brainstorming other situations where it is natural to want to turn around. And there it was—the elevator. Of course! We are all used to walking into a small enclosure, turning around, and then walking out into a new place moments later.
So here was the idea for my next contribution. Let’s install elevators in the Holodeck! Let’s alternate their location from level to level, encouraging the player to walk across the room, get into and elevator, and then emerge from the elevator at a new level. Then let’s repeat this several times, taking the player higher and higher from the ground floor, all the way up the roof and the wide open sky.
And this is what I did. And I filmed the premier, on location at the NYU Holodeck.
Watch how it all turned out in the video at the top of the article.