Ever wanted a full-size movie theater of your very own? A new Oculus Rift compatible project promises to grant your wish, and then some. We talk to Christer-Andre Fagerslett (aka namielus) the author of a new ‘Virtual Cinema’ project which leverage’s the immersive power of the Rift and aims to provide the ultimate movie viewing experience.
Virtual Reality Movies with the Oculus Rift
Ask an enthusiast what he or she wants from Virtual Reality, you’ll find plenty of fantastical worlds and unearthly experiences on the resulting list. However, once 3D HMDs become commonplace, more mundane places and activities might well find themselves replaced by virtual reality alternatives.
One of the lesser discussed possible uses for an effective 3D HMD (such as the Oculus Rift) is using it to watch your favourite movies in ghost-free, high FOV 3D, emulating that unmistakable feeling of being at the big screen at your local Multiplex. Certainly that was the primary goal for Sony’s recent ‘HMZ’ offerings, the Hi-res HMDs that were designed soley for HD media consumption.
But what if that desire to recreate the Cinema experience was taken to its logical extreme? Christer-Andre’s homebrew application targeting the Oculus Rift not only seeks to fool your brain into believing you’re enjoying a first rate movie experience, complete with fully-realised virtual theaters, but to emulate the human element that can make going to the Movies such a ‘unique’ experience.
To find out a little more about what this project promises and its future applications I spoke to Christer-Andre himself to get his thoughts:
Christer-Andre: My name is Christer-Andre Fagerslett (Yes I know my last name is funny in English). I saw a video of Sony HMZ-T1 being used for Skyrim, found it amazing and started searching for information. Stumbled upon Palmer Luckey’s project, and thats was probably August 2012. I don’t remember where I first tried a Rift, but here I am with my own DIY version developing content for it full time.
RtoVR: Clearly you’re a VR enthusiast. What was it that got you interested in Virtual Reality? Are you an Oculus Rift Kickstarter backer?
Christer-Andre: Yes I am a Kickstarter backer. I’m an 80’s kid, and growing up I felt like VR was in the near future. There was Vr5 on TV, Lawnmover Man, all kinds of gameshows with vr-themes. You also had things like the VirtualBoy [Nintendo] and Powerglove [Nintendo NES]. Stuff that was cool in concept but not quite there in practice.
I fell in love with the idea of being inside the game, becoming whatever I wanted wherever I wanted. As I grew older, my interest/hopes for VR faded and became a distant memory. Until now.
RtoVR: Are you a programmer by trade or just a hobbyist?
Christer-Andre: I would say I am neither, I am learning programming as I go – and definitely need to hire someone more able. Figuring out one line of code takes time for me, and I am learning so much new stuff in a short time span I often have to figure it out all over again the second time I read the same line of code.
Only reason that I am doing this all on my own is that up until now I have had no choice. Programming is really not my thing, it hurts my head. I like to work more as a concept designer , 3dmodeller and “idea man”.
RtoVR: How do you feel VR will change the way we consume media?
Christer-Andre: It will take our attention back and focus it. Currently you have people checking Facebook on their phone while watching movies, dividing their attention between multiple media. When you are immersed in what you are doing, the artist can take back that control. It’s ironic, given that a HMD blindfolds you and gives you a view into a purely digital world that if you put VR-headsets on each student in an actual classroom you would have much better control of their attention. i.e. to do something else, they would have to remove the HMD.
Also, I think this is the time where the work of 3d-modellers are truly appreciated but also more scrutinized. Especially when the goal of positional tracking is achieved, you are going to have people on their knees in the game looking at little flowers and small details.
Goodbye getting away with sprites!
RtoVR: Tell us a bit about your Oculus Rift Virtual Cinema project – what were your goals and objectives (or were you just messing about for fun?)
Christer-Andre: I was just messing around with a virtual lounge concept (it’s still in the works). I started a thread about it on MTBS3D on September 3rd 2012.
At that point I had never written a line of code, and never really tried Unity. I was just making 3D-models, and rendering them in Indigo Renderer.
As the interest/response grew, I realized this really has potential. Since then some big names have contacted me and expressed interest.
It’s been a steep learning curve. I have been looking at different scripts, modifying and butchering them to get to the point where it has level of support for Rift worth showing. I feel confident that I will be able to have a alpha demo at launch, but once the SDK is out I am probably going to remove all the rinkydink scripting and use the tools provided by Oculus.
I found out, that unless I had a demo worth showing – nobody would bother joining my team. The cinema concept as a standalone became the most popular, so I want to use that as a step to my further goals. I am finally at the point now where people are actually interested, and I have a small amount of funding.
The next step now is to go on Kickstarter, and try to get interested in my Virtual Cinema and some pretty cool stretch goals.
RtoVR: What platform / engine / tools are you using for development and why?
Christer-Andre: I started out using Unreal Engine, but ran into severe limits in terms of codec/container support. I could play bink-files, and that’s it. The people I hired, they could not implement head-tracking, the side-by-side was squeezed and it was just a loss of money for me. Sadly.
Then I decided to go with Unity3D, it already has a wider support but needs some extra scripting to enable browsing your drives for files directly from within the game.
I am using TrackIR5 for head-tracking, (positional tracking limited to mimic Rift tracking). However, I have ordered [a] hillcrest [tracker] out of necessity, because I am helping another games company get ready for showing their rift enabled game at GDC.
Also, renderheads will provide you with better codec support.
RtoVR: The project doesn’t limit itself to ‘merely’ watching movies in a virtual auditorium. Tell us more about the Interactive elements (i.e. the heckling).
Christer-Andre: Voice chat is a must, gesture tracking by either Kinect, Leap Motion or similar is an big goal. I was hoping I could do something to record a persons comments and gestures and then let you send that to a friend so that he can watch a movie with your ranting somewhere in the audience. Maybe even let you share it to the public. I am looking into various methods/software for photogrammetry, so that you can take photos or video of your face and get a realistic looking 3Dmodel of yourself with textures as an avatar.
Other than that, there will be a standalone Virtual Cinema that can be considered a media player however, I am also making a larger building that has museums and arcades, office buildings and stuff.
Currently [I’m] making a VR museum where you can walk around looking at different vr-devices from the 70s and up until now.
A worry here is licensing though. This is a very early pre-alpha render of my arcade, since then I have added hundreds of arcade machines. I was hoping to stream each game to the individual arcade machine through MAME. That’s going to be a challenge with all the large companies breathing down my neck.
Also, one of the Reward tiers in [our] kickstarter is going to be having a permanent room in my VR-building that you can sketch yourself and we will make it in 3D. It’s going to be a cool way of marketing that is completely optional for the user. Nothing intrusive. If you want to walk into the room and look at what they have there, you can. If not, just move along.
This is the very first render I made showing the concept of the whole building. Its noisy and unpolished but I like it because it shows how it all started. You can see the cinema inside there, but let me emphasize that there will be a standalone cinema with multiple themes that is not part of that building.
RtoVR: How do you go about mapping video inside the virtual world? Are there technical limitations on the video CODECs, bitrate, container (i.e. MKV / AVI etc.) or resolution?
Christer-Andre: The limits are only in your hardware. Assuming the consumer rift has a resolution that makes it worth while, you can go beyond 1080p. However, this currently involves streaming video into a texture in the game. This puts a heavy load on the cpu if the resolution/bitrate is high. If your computer struggles to play high bitrate 1080p, forget about playing it in my virtual cinema. For now.
You can play any container, codec and bitrate you can play in MediaPlayer. If you have a fast enough capture device, you can even connect your NES, Atari, Sega or whatever you want by s-video etc and pipe that through a texture onto the big screen.
Streaming video by inputting URL’s [are] also gonna work, so my hope is to enable services streaming live games like GOMTV, Ustream etc.
There are some limited support for stereoscopic 3d-video on the screen itself. My hope is to support bluray 3d playback simply by putting a disk in the drive, and getting that authentic pop out effect you know from 3D-movies. When the stereo-3d is all over your field of view its more subtle and natural. Like real life. Of course this is a good thing.
However, some people seem to be missing that popout effect. The only way to provide that the same way its done in movies, is to have a frame of reference it can pop out from in relation to the “real world”. The real world in this case is the virtual cinema, and the frame of reference is the virtual screen. But it does make it closer to the way you are used to watching 3D movies.
Using DirectShow, I could in theory just stream the entire desktop into a texture. However, its clumsy and less than ideal. If desktop streaming is what you want, go with IBEX or similar. Even Gmod. What I am working on is a sleek, intuitive user interface that is specifically designed to work as a mediaplayer from within the game. I am confident we can get emulators working smoothly, like MAME etc. What remains is tying all this to a nice and intuitive GUI.
RtoVR: How do you plan to handle surround sound audio? Will discrete audio channels from the movie soundtrack be positioned in the world and moved according to head rotation?
Christer-Andre: At the moment, I have not given much effort to that aspect. The sound is in the same no matter where you turn.
However, I just got in contact with a guy who’s job is to create/implement/optimize audio for games in Unity. I am hoping that throwing money on that company will give me some breathing room. At the very least they should be able to advise me on who to hire for these things.
Also, hoping to be able to have each surround channel be represented in the virtual space located where they are supposed to be in the real world. I was told that as long as the audio listener turns with the camera, you can position speakers like a 7.1 setup and output each channel to the correct speaker. So that turning your head away from the virtual screen gives the same result as in the real world.
RtoVR: You mention that you see this as a ‘multiplayer’ project, i.e. people can join you in your Virtual Cinema. Do you see this as something people will do more of in the future? How do you see the feature evolving?
Christer-Andre: This is the most fundamental aspect of my project. In terms of the future.
My plan is to expand this way beyond videos. I plan on having a fairly expensive motion capture rig, like Naturalpoint Optitrack or similar. Alternatively something like Kinect 2.0 if its fast enough. This way I can capture full body motion and gestures from a few people, and having the majority of the audience use the very cheap Oculus Rift with head-tracking only.
Basically you can have anything from a lecture (with the teacher manipulating gigantic 3Dmodels hovering above the audience) to a rock concert you can watch live but then replay forever from any perspective.
If this sounds complicated, its very much achievable with current tech. The content provider (school, business etc) rents or buys a more capable/expensive motion capture rig, and deliver this content to users that are using Oculus Rift.
Eye contact is a powerful thing, and with high-def graphics it starts making sense to take that into account. Imagine a teacher speaking to his class, maintaining eye contact with every student. (within reason) This could impact how much you pay attention to what is being said even further.
If we can mount a Leap Motion to the helmet, or the chest of the user to enable hand/arm tracking – the potential for affordable VR is through the roof. And that’s the very near future.
Our thanks to Christer-Andre for sparing his time and for supplying the exclusive video. You can see more of his work and get in touch with him over on his YouTube channel.
