Smart Solution Makes Handheld VR Cinematography Easy with a Real Virtual Camera

VFX professional Kert Gartner has been pioneering new ways to make capturing virtual worlds as compelling as filming in the real world. His latest technique makes a real virtual camera out of an extra Vive controller and a smartphone.

You may have actually seen Gartners work before, he was a major part of the new Space Pirate Trainer trailer production, and the mixed reality Fantastic Contraption trailer. His latest trick is a little bit of genius that expands upon much of his prior work with VR cinematography.

To get the feel of ‘real world’ cinematography in VR, you need to be able to manipulate the virtual camera just like real world cinematographers manipulate real cameras. Often times that means handheld cameras that are up close to the action. Gartner has been emulating that handheld camera look by smartly using a third Vive controller to manipulate a virtual camera inside the virtual world. But filming this way is difficult because the controller doesn’t have a viewfinder, meaning that seeing what it’s pointed at requires looking at some distant monitor on the periphery of the of the action.

Now Gartner has solved that problem by using a smartphone to act as the viewfinder for the virtual camera, allowing him to see exactly what the virtual camera sees as he films:

The foundation of the technique requires a custom build of whatever VR experience is being filmed; one which is able to project multiple camera views in a single frame, including an angle which will be the view of a virtual camera which is “attached” to an extra Vive controller. Rendering out a 4k frame divided into multiple views means that each view can still be captured at a sharp 1920×1080 by cropping away the other views. Capturing all views at once gives the editor the ability to chose from multiple viewpoints of the same shot in post processing, without the cinematographer needing to do multiple takes from different angles.

The next step is to affix a smartphone to the Vive controller. In this case, Gartner whipped up a simple rig with a vertical grip for easy capturing, but I imagine it would be almost as easy to connect the smartphone directly to the Vive controller (just don’t block the tracking receivers!).

From there, Gartner employs the jsmpeg-vnc utility which allows him to stream a selectable crop of the monitor to a web server which allows it to be viewed easily by visiting a URL on the smartphone. This gives him a viewfinder which shows exactly what the virtual camera sees.

Altogether, the rig now works like a handheld camera, except instead of filming the real world, his viewfinder shows a portal into the virtual reality world. Even rendering multiple views at 4k and running the screen streaming and capturing software, Gartner says the whole system is still hitting the crucial 90 FPS figure (no doubt thanks to a powerful PC managing it all).

Praise to Gartner indeed for this smart solution, though many will know that technique isn’t brand new; the use of real-time digital visuals with a virtual camera and viewfinder has been seen in high-end film and game production, notably on Avatar (2009), Halo Reach (2010), and Uncharted 4 (2016) but Gartner’s system is tremendously less expensive, using just a PC, HTC Vive, extra Vive controller, and some streaming software; that puts it in the realm of practicality for many VR developers.

We expect to see this technique further developed for the production of high-end VR trailers, and even for big livestreaming VR shoots. Hooking the Vive controller and smartphone viewfinder up to a Steadicam rig could give a cool high-end film look. We can also imagine a more realistic virtual camera model offering complex adjustments like focal length, exposure, and manipulation of lens characteristics, all controllable in real time from the touchscreen of the smartphone viewfinder.