After having teased the tech toward the end of last year, we've finally gone hands-on with HypeVR's volumetric video captures which lets you move around inside of VR videos. Inherent Limitations of 360 Video Today's most immersive VR video productions are shot in 360 degree video and 3D. Properly executed 360 3D video content can look quite good in VR (just take a look at some of the work from Felix & Paul Studios). But—assuming we can one day achieve retina-quality resolution and geometrically perfect stereoscopy—there's a hurdle that 360 3D video content simply can't surmount: movement inside of the video experience. With any 360 video today (3D or otherwise) your view is locked to a single vantage point. Unlike real-time rendered VR games, you can't walk around inside the video—let alone just lean in your chair and expect the scene to move accordingly. Not only is that less immersive, it's also less comfortable; we're are all constantly moving our heads slightly even when sitting still, and when the virtual view doesn't line up with those movements, the world feels a less real and less comfortable. Volumetric VR Video Capture That's one of a number of reasons that HypeVR is working on volumetric video capture technology. The idea is to capture not just a series of 360 pictures and string them together (like with traditional 360 cameras), but to capture the volumetric data of the scene for each frame so that when the world is played back, the information is available to enable the user to move inside the video. At CES 2017, I saw both the original teaser video shot with HypeVR's monster capture rig, and a brand new, even more vivid experience, created in conjunction with Intel. With an Oculus Rift headset, I stepped into that new scene: a 30 second loop of a picturesque valley in lush Vietnam. I was standing on a rock on a tiny little island in the middle of a lake. Just beyond the rock the island was covered in lush wild grasses, and a few yards away from me was a grazing water buffalo and a farmer. https://gfycat.com/VeneratedSizzlingAiredale Surrounding me in the distance was rainforest foliage and an amazing array of waterfalls cascading down into the lake. Gentle waves rippled through the water and lapped the edge of my little island, pushing some of the wild grass at the water's edge. It was vivid and sharp—it felt more immersive than pretty much any 360 3D video I've ever seen through a headset, mostly because I was able to move around within the video, with proper parallax, in a roomscale area. It made me feel like I was actually standing there, in Vietnam, not just that my eyes alone had been transported. This is the experience we all want when we imagine VR video, and it's where the medium needs to head in the future to becoming truly compelling. Now, I've seen impressive photogrammetry VR experiences before, but photogrammetry requires someone to canvas a scene for hours, capturing it from every conceivable angle and then compiling all the photos together into a model. The results can be tremendous, but there's no way to capture moving objects because you can't capture the entire scene fast enough to record moving objects. HypeVR's approach is different, their rig sits static in a scene and captures it 60 times per second, using a combination of high-quality video capture and depth-mapping LiDAR. Later, the texture data from the video is fused with the depth data to create 60 volumetric 'frames' of the scene per second. That means you'll be able to see waves moving or cars driving, but still maintain the volumetric data which gives users the ability to move within some portion of the capture. The 'frames' in the case of volumetric video capture are actually real-time rendered 3D models of the scene which are playing back one after another. That not only allows the viewer to walk around within the space like they would a VR game environment, but is also the reason why HypeVR's experiences look so sharp and immersive—every frame that's rendered for the VR headset's display is done so with optimal sampling of the available data and has geometrically correct 3D at every angle (not just a few 3D sweet spots, as with 360 3D video). This approach also means there's no issues with off-horizon capture (as we too frequently see with 360 camera footage). Continue Reading on Page 2 >> Overcoming Limitations What HypeVR has today is a promising look at what VR video should really look like, but there's some clear production and distribution limitations that will need to be solved before it can be used to make the sort of cinematic VR content audiences want. Volumetric Shadows & Panning You can think about this like normal shadows: when an object move past you and is illuminated by a light from the front, a shadow is cast behind the object because the light can't pass through it. The location of the shadow depends on the location of the light and the object. Since HypeVR is capturing the depth of the scene using LiDAR (which bounces lasers off of objects to determine how far away they are), objects block the laser and cast 'volumetric shadows'. Instead of a dark area, these volumetric shadows leave holes in the geometry behind them because the camera is unable to see behind the object (just like the light can't shine behind the object). If you stood completely still at the exact capture point, this wouldn't matter at all, because you'd never see the volumetric shadow behind the captured objects. But since users can move around within the space, they will potentially see behind those objects and thus see the volumetric shadows. The HypeVR content I saw had no volumetric shadows, which means the company has found (for these specific pieces of content), effective ways of dealing with them. The easiest way is perhaps hand-fixing the geometry to fill the holes in post-production. That could be relatively simple (in the case of a person walking by a few yards away from the camera), or extremely difficult (in the case of putting the camera in the middle of a dense forest with nearby trees and plants casting all manner of volumetric shadows across the scene). Another way is to insert a 3D model (which could be captured from the real world with photogrammetry) into the space after capture so that there were never shadows in the first place. That's easy because the entire scene is rendered as 3D geometry already, much like a videogame environment, so inserting more 3D objects is relatively straightforward. And that last one is probably where volumetric video capture will ultimately go in the future—a fusion of live-action scene captures combined seamlessly with CGI models in heavy post production (similar to what we see in today's blockbuster film production). Moving the capture rig is likely to complicate things further, as it will cause volumetric shadows to pan across the scene. HypeVR tells me they haven't yet shot moving-camera tests to see how their tech handles panning. If it turns out that the rig can't be moved while filming, it may or may not matter, depending upon how important moving cameras are to the language of VR filmmaking; presently, many 360 video productions for VR are shot with static cameras. Download Sizes and Data Handling HypeVR's capture rig uses 14 RED cameras which each shoot at 6k and 60 FPS. There's also the LiDAR data being captured in each frame. It takes time to process and reconstruct all of that data into its final form (currently around 6 minutes per frame, locally). After it's all properly rendered, the experience needs to get from the web to the user somehow, but at 2GB per 30 seconds of capture, that's going to be tough. 2GB per 30 seconds of video is actually a massive improvement over where HypeVR was not long ago, with 30 seconds of capture clocking in at a whopping 5.4 terabypes. Making volumetric capture technologies like HypeVR work will require still better compression to create realistic file sizes. Faster consumer internet connections will likely also play a key role in making this sort of VR video streamable so it can start right away without the need to wait for a huge download. - - — - - There's definitely challenges to solve when it comes to volumetric video capture, but that's normal for any new film production technology. HypeVR has demonstrated that it's possible to achieve the sort of compelling live-action VR video that everyone wants, while mitigating or eliminating its limitations. Employing the technology in increasingly complex productions will require smart direction and probably the development of new techniques to achieve practical production times and a seamless end product, but if the company's demo scenes are any indication, the results may very well be worth the effort.