Interactive Virtual Reality Movies Will Combine Film and Games to Create Highly Immersive Narratives

As games and films come ever closer together, I see a new genre of entertainment being born: the interactive movie. First, an inventory of where games are today:

Many of the latest games are already incorporating real-world actors into their environment by using motion capture, a technology that allows real-life body movements to be captured and used in place of hand-animation. This brings an impressive level of realism to a digital world. A behind the scenes video of Uncharted 3 (2011) gives a good idea of the process behind motion capture if you are unfamiliar:

Some of the latest blockbuster games are taking it one step further by using the voices and faces of actors to achieve better human facial animation and emotion than hand-animation and voiceovers can provide. L.A. Noire (2011) used a technology called MotionScan which utilizes 32 high definition cameras to capture the head-performance of an actor from every angle, creating a believable 3D model that has perfectly synchronized animation and audio. Rockstar, the developers behind L.A. Noire, added the information captured from MotionScan directly to the game without any adjustments. Here you can see how the faces are captured:

As video games are becoming increasingly movie-like (from using real actors, to surpassing movie sales), I foresee a merger of the two genres — the interactive movie.

The first form of this genre will likely place you in pseudo-control of the camera. I envision putting on a head mounted display with integrated head-tracking, and being able to literally look around the scene. How much more immersive would if be if you weren’t simply watching something happen to others, but you felt like you were in the midst of the scene itself? This could be done with high definition panoramic cameras (more on the film side), or by creating the entire interactive movie virtually (more like a game).

Rather than seeing a couple argue emotionally in the kitchen from some predefined angle, imagine feeling like you’re sitting at the table right next to them. The emotion of the argument would be greatly enhanced thanks to you being quite literally ‘in medias res’.

Imagine how much more terrifying a scary movie would be where, thanks to surround sound, you can look in the direction of every creak, groan, and scream, or hear faint footsteps creeping up behind you only to turn and find that there is nothing there.

I just recently read a great article about VR by Jim Blascovich; he argues that virtual reality is being expressed with new technology, but the concept itself is ancient. Aside from Blascovich’s interesting perspective, the article opened with an anecdote of Blascovich trying an early virtual reality prototype. The year was 1995, and although the graphics and head-tracking were likely not a quarter as good as they are today, Blascovich describes an impressive level of immersion when he donned an HMD and was asked to walk across a plank that was perched atop a pit:

Looking down, I was startled by the appearance of a pit, about 3 meters square, in the ground in front of me. I approached the edge of it gingerly and looked down. Being careful not to fall in, I estimated the pit to be about 10 meters deep.

My colleague told me to walk a narrow plank that spanned the pit if I dared. At first, I couldn’t move my feet. There was no way I wanted to do it, even though I knew that the lab’s floor was physically there and I wouldn’t actually fall down the shaft. After several false starts and a fall down the shaft causing me to catch my breath, I made it across the proverbial plank, arms extended outward and parallel to the ground, wobbling with fear even though I kept repeating to myself, “It isn’t real. There is no pit.” I simply could not consciously control my fear response.

How many times have you seen a movie where adventurers are exploring through a jungle when they come upon a rickety bridge spanning hundreds of feet of gorge, with a lethal drop below? Normally, the view you get in a movie is some aerial shot as the actors move across the bridge, but what if you were right there behind the adventurers, able to look down at your feet and see the terrifying chasm beneath you? That movie is suddenly far more immersive than it would be otherwise. This immersion provides a new tool for story-tellers to evoke emotion that plays on our own experiences, instincts, and fears. Again, we have the technology to make this happen today.

Early forms of interactive movies will likely be of the ‘on-rails’ variety (ie: you’ll be able to look around, but your path will be chosen for you). As the genre further developers, I can see the interaction quickly going one step further; allowing you to make decisions within the virtual environment and alter the path of the experience and the story.

Games and movies already provide all of the tools necessary to make interactive movies:

Games: virtual environments, dynamic surround sound, allowing the user to interact with the world.

Movies: cinematics, immersive stories, great actors, widespread appeal.

You can see where the two will intersect, and it won’t take a lot of work because we’re nearly already there. All it needs to go mainstream is an affordable and high-performance HMD (Sony is getting us started on that front), and you’ll start to see highly immersive interactive movies popping up in no time. I, for one, cannot wait.