Overcoming Limitations
What HypeVR has today is a promising look at what VR video should really look like, but there’s some clear production and distribution limitations that will need to be solved before it can be used to make the sort of cinematic VR content audiences want.
Volumetric Shadows & Panning
You can think about this like normal shadows: when an object move past you and is illuminated by a light from the front, a shadow is cast behind the object because the light can’t pass through it. The location of the shadow depends on the location of the light and the object.
Since HypeVR is capturing the depth of the scene using LiDAR (which bounces lasers off of objects to determine how far away they are), objects block the laser and cast ‘volumetric shadows’. Instead of a dark area, these volumetric shadows leave holes in the geometry behind them because the camera is unable to see behind the object (just like the light can’t shine behind the object).
If you stood completely still at the exact capture point, this wouldn’t matter at all, because you’d never see the volumetric shadow behind the captured objects. But since users can move around within the space, they will potentially see behind those objects and thus see the volumetric shadows.
The HypeVR content I saw had no volumetric shadows, which means the company has found (for these specific pieces of content), effective ways of dealing with them.
The easiest way is perhaps hand-fixing the geometry to fill the holes in post-production. That could be relatively simple (in the case of a person walking by a few yards away from the camera), or extremely difficult (in the case of putting the camera in the middle of a dense forest with nearby trees and plants casting all manner of volumetric shadows across the scene).
Another way is to insert a 3D model (which could be captured from the real world with photogrammetry) into the space after capture so that there were never shadows in the first place. That’s easy because the entire scene is rendered as 3D geometry already, much like a videogame environment, so inserting more 3D objects is relatively straightforward.
And that last one is probably where volumetric video capture will ultimately go in the future—a fusion of live-action scene captures combined seamlessly with CGI models in heavy post production (similar to what we see in today’s blockbuster film production).
Moving the capture rig is likely to complicate things further, as it will cause volumetric shadows to pan across the scene. HypeVR tells me they haven’t yet shot moving-camera tests to see how their tech handles panning. If it turns out that the rig can’t be moved while filming, it may or may not matter, depending upon how important moving cameras are to the language of VR filmmaking; presently, many 360 video productions for VR are shot with static cameras.
Download Sizes and Data Handling
HypeVR’s capture rig uses 14 RED cameras which each shoot at 6k and 60 FPS. There’s also the LiDAR data being captured in each frame. It takes time to process and reconstruct all of that data into its final form (currently around 6 minutes per frame, locally). After it’s all properly rendered, the experience needs to get from the web to the user somehow, but at 2GB per 30 seconds of capture, that’s going to be tough. 2GB per 30 seconds of video is actually a massive improvement over where HypeVR was not long ago, with 30 seconds of capture clocking in at a whopping 5.4 terabypes.
Making volumetric capture technologies like HypeVR work will require still better compression to create realistic file sizes. Faster consumer internet connections will likely also play a key role in making this sort of VR video streamable so it can start right away without the need to wait for a huge download.
– – — – –
There’s definitely challenges to solve when it comes to volumetric video capture, but that’s normal for any new film production technology. HypeVR has demonstrated that it’s possible to achieve the sort of compelling live-action VR video that everyone wants, while mitigating or eliminating its limitations. Employing the technology in increasingly complex productions will require smart direction and probably the development of new techniques to achieve practical production times and a seamless end product, but if the company’s demo scenes are any indication, the results may very well be worth the effort.
