Making the Virtual Real World More Real
Since the lighting of the scene is ‘baked’ into the photo-based textures, the raw capture won’t include volumetric light or specular lighting. However, the team has shown how they can layer in some of these effects to make the scenes even more realistic.
In another space that I saw—an abandoned room with an old grand piano at the center—the team had added volumetric light rays streaming through the high windows along the side of the room. Following the beams from the window to their landing point on the opposite wall, they reconnected with the baked lightning from the real scene in a convincing way. With the rays illuminating the (added) dust particles floating about, the decrepit space was yet more convincing.
Another scene I got to see was an abandoned shower and bathing space from an old hospital. Ceramic tiles covered the space from floor to vaulted ceiling. Instead of the walls growing flatter and less real as I approached (as you might expect in a videogame), getting closer actually revealed them to be even more real than they looked from afar, thanks to the team’s photogrammetry process which was accurate enough to capture the mere millimeters of elevation change between the tiles themselves and the sunken grout lines between them.
The realism of the tiles is further enhanced because the Realities team added in-engine specular lightning, showing reflective highlights on the semi-reflective tiles which moves appropriately as you move your head throughout the scene, thus eliminating the baked lightning effect. This is especially impressive up close and at extreme angles where usual techniques used in many non-VR experiences (like bump-mapping) would break down.
Go to Places You Can’t Go
So Realities can capture impressive looking real-world spaces and bring them into VR. Now what?
The company’s current approach is to capture amazing places around the world and allow people to visit as virtual tourists, with a bit of an educational bent. Some of the early user interactions within these spaces involves finding photos positioned throughout the area which, when retrieved, reveal interesting information about the space. The team especially wants to capture places people aren’t normally allowed to go. For instance, all of the scenes I described in the abandoned building were of a place the public isn’t actually allowed.
It’s a compelling idea, especially given the rate at which the team’s photogrammetry technique is improving. I imagine it would be awesome to step inside the White House’s Oval Office, but even if I was allowed to do so, it might not be compelling enough for me personally to warrant a trip to Washington D.C. But it’s almost certainly compelling enough to warrant merely putting on a VR headset if it means I can see a near-identical recreation. And there’s no doubt that the recreated scenes would be accurate enough to almost entirely replicate the experience of walking around an art museum.
As a gamer at heart however, I was definitely thinking of ways these spaces might be used for more-action oriented interactive experiences. With the team already rendering these scenes in real-time in UE4—even with VR’s high end processing demands—it seems like that should be possible. Anyone up for team deathmatch in the Oval Office?
Realities is planning to launch a version of their photogrammetry captures on SteamVR for free. Timing on that release is not yet determined, but the company will be showcasing their work at GDC this coming week at the Graphine booth (Main Hall, booth 429).
