Last week news broke that Sony would be demonstrating an augmented reality ‘Prototype SR’ head mounted display at the Tokyo Game Show. This HMD is based on the new HMZ-T2 which Sony announced at the beginning of the month. Toshi Nakamura of gaming news site Kotaku got a chance to experience the demonstration that Sony put together with the HMZ-T2 Prototype SR. The reaction? “I’ve seen the future of virtual reality, and it is terrifying.”
Sony apparently raffled off a number of limited slots for Tokyo Game Show goers to sit in on the demo, though the gaming press likely got in without a ticket.
The HMZ-T2 Prototype SR has a camera mounted on the front which passes the video from outside into the displays of the unit. Nakamura explains that Sony exploited this digital divide between one’s eyes and reality to create a visceral experience. Instead of true augmented reality, where extra information is laid over top of a live scene, Sony utilized a combination of pre-recorded footage and live-view to deceive the viewer. After being shown a video of Resident Evil: Retribution to set the mood, things got frightening:
After the trailer, the screen returned to the video feed of the room where the guide proceeded with the demonstration. At this point, the lights in the room suddenly began to flicker, and the guide seemingly grew ill and collapsed. While I knew this portion was a pre-recorded sequence featuring the same guide, I was able to suspend my disbelief. The ability to move my head and look around made the whole thing all the more real. The guide then rose again as a zombie and lunged at me, causing me to jump. What can I say, it was a good scare. The display went black and I once again returns to a live feed of the room with the real guide.
The trick was that upon returning to the view of the room after the trailer, the scene switched from a live view to a pre-recorded sequence, hopefully without the viewer noticing. Nakamura says that the recorded sequence was captured in stereoscopic 3D and recorded with a 360 degree camera, allowing the viewer to look around the scene convincingly; although, because the Prototype SR only uses a single camera, the live view couldn’t feature 3D video like the pre-recorded parts were — I feel like this would have been an immersion-breaking giveaway.
After the zombie sequence there were apparently a few more scenes:
In further demonstrations, the guide continued to show a sequence of mixtures of live and prerecorded images, all of which was visually fascinating to see (At one point, the guide reached out her hand for me to touch to show that she was really there). By blending both the live and prerecorded 3D images, the whole sequence was like a highly immersive AR game.
Of course there are limitations to the experience, as Nakamura notes. For one, the pre-recorded scenes are not true augmented reality, they are simply video recordings. This means that if you reach your hand out you won’t be able to see it in front of you (because it was not there when the scene was recorded). Then there’s the HMZ-T2 itself which has a relatively small 45 degree field of view which is less immersive that something like the Oculus Rift [head mounted display comparison chart here].
The takeaway for me however is when Nakamura says, “Nevertheless, even the with current restrictions, the possibilities are astounding. I’d never seen anything like it. There were moments during the demonstration where I literally could not tell what was real and what was not,” [my emphasis]. Achieving that level of sense-trickery where you must literally ask yourself ‘is this real?’ means that we’re well on the way to highly immersive virtual reality and augmented reality experiences.
Sony has said that it has no current plans to release the Prototype SR version of the HMZ-T2, but it is great to see that they are experimenting with these types of experiences. If the Oculus Rift makes a big splash in the game market, as I expect it will, Sony better be ready to compete!