Rendering specialist Otoy has as of late taken to deep exploration of light fields as a medium for rendering high fidelity virtual reality scenes. While the company’s computer-generated ray-traced light fields are great for viewing CGI envrioments, what happens if you want to capture the real world? Well, Otoy has been working on that too, and we’re the first to see their progress in action.
For some time, light fields to me have been about as utterly incomprehensible as quantum physics—you can see the results in action, but without understanding the underlying principles, it’s hard to grasp what it really means, and importantly, how it can be used. After spending some time in Otoy’s array of VR demos, however, I’m just starting to wrap my head around why the company is focusing their energy so intently on light fields for VR and AR.
Key to even thinking about light fields is first understanding what they are. I’ve had people try to explain them to me as a scene captured in many angles by an array of tiny cameras, or ‘all the light rays in the scene’. But without the benefit of being able to stick your head into a light field yourself, it’s hard to derive any meaning from those phrases.
See Also: First Look – ‘Batman’ Gear VR Experience Featuring OTOY’s Incredible 18k Cubemap Video Tech
So let me first start with the best way I can think to explain what a light field is, at least in my head: A light field is like a volumetric pixel of 3D space. Let me elaborate.
Let’s suppose we have a magic lightfield capturing device that looks like a cardboard box that’s 1 foot square. Take that box and hold it anywhere in the room you’re currently sitting in. Now press the magic capture button on the side of our magic box. *CHHCKK* (my best text impression of a camera shutter sound). You’ve snapped a volumetric pixel that’s exactly as big as the box.
So what did you actually capture? Well, you now have a portable view of the world as seen from anywhere and at any angle, inside of that box. When you stick your head inside of the pixel you just captured, it’s as if your head is in the same 3D space that the box had filled. Anywhere you look from inside of that box area will show you the world accurately represented—in any direction, from any position, and with depth—as though you head was transported to the volume that the box had occupied when you took the snapshot. If you move your head outside of the area of the box, the world goes blank (conceptually similar to the edge of a flat photograph, but sceen here in more dimensions), because you’ve only captured what the world looks like from any point inside of that box.
I’m still not sure if that will make any sense to someone who hasn’t seen a light field for themselves, but hey, I tried.
Anyway… Otoy has been working with computer-generated light fields for a little while now, but lately they’ve been developing a method for capturing light fields from real life scenes. I got to stick my head into one of the very first captures using the HTC Vive.
Remember that magic light field-capturing cardboard box I described earlier that you could place somewhere and then capture a volumetric pixel of that area? Well that’s exactly what Otoy is doing. Ok well… not exactly. The capture rig they’re using actually consists of two DSLR cameras which whirl around on a spinning mount. But that they accomplish is the same—they capture the view as though you head was right in that volume of space (in this case, its a sphere instead of a box, because of how the cameras spin around). That means that you can see the scene accurately no matter how you rotate or move your head within that space (including positional movements). Visually it’s a lot like having that volume of the real world right there with you, even though its physical manifestation in reality is elsewhere.
Ultimately the output, as seen through the HTC Vive headset looks a lot like 360 degree photographic views you may have seen before (albeit on the higher quality end of the spectrum, though not quite matching the exemplary quality of the company’s CGI light fields), but with the key benefits of stereoscopic depth, positional tracking (parallax), and the ability to recreate the view accurately no matter how you roll your head (shoulder to shoulder). Other 360 degree stereoscopic camera systems have trouble with that last part since the lenses creating the stereo pairs are aligned side-by-side which doesn’t match the orientation of your eyes when your head is tilted). Excitingly, these live-captured light field views have a simple path to even higher quality as the major limitations to the fidelity is the resolution of the capture cameras and the screens that the scenes are viewed on, both of which will improve ‘automatically’ and independently of Otoy.
Performance also seems to be a benefit of Otoy’s particularly brand of light fields. The company says that their viewer runs at 2,000 FPS on the PC and 120Hz on mobile, meaning miles of headroom before increasing display refresh rates become a bottleneck.
And while filesize has typically been pointed to as a major drawback of light fields, Otoy says that they can fit a 1 meter light field cube into 5-36 Mb. Large compared to a simple 360 degree photo of the same view, but way worth it for the benefits of positional tracking and proper stereoscopy.
See Also: 2nd Round Winners of ‘Render the Metaverse’ VR Panoramic Contest Announced, Available Now on Gear VR
Indeed there are some downsides to the light field—all of which Otoy says they’re working on. For one, the file sizes of these scenes can be impractically large for many use-cases. 5-36 Mb isn’t bad if you just want to look at one view, but as you increase the size of the light field, the filesize can increase quickly. You can add more discrete light fields, but the size will add up quickly compared to other methods of scene capture.
Imagine for instance a house tour with 20 views; on the low end of Otoy’s light field size figure, 5 Mb, that’s 100 Mb—not too bad, as long as you’re on a WiFi connection. But let’s just consider the average of the range they gave (20.5 Mb): that would put a 20-view house tour at 410 Mb which can be quite restrictive to the way people might want to use light fields, even if it only means a minute or two of waiting. Streaming, however, could be an easy enough answer.
Another issue is that these are still scenes only, for the time being. Otoy says they have a working solution for video in CGI and live-captured light fields, though we haven’t seen it just yet. I’m not yet sure how they plan to maintain practical filesizes for video in live-captured light fields, but they’re well aware that it’s an issue to be tackled.
One interesting upside to Otoy’s live-captured light field is that their particular process doesn’t require much equipment. Compared to a 10-GoPro 360 camera rig which would run you $5,000 for the cameras alone, you could presumably slap a high-end consumer DSLR onto Otoy’s setup for a fraction of the cost (the dual-camera capture rig in the video above apparently only uses one of the two cameras, the other, is just for balance {an expensive counterweight at that!}).
Up to this point, Otoy’s light fields have been inaccessible except to those who know how to use their complex rendering tools. But now it looks like Otoy is on a path to capturing these high quality volumetric pixels with increasing simplicity. In the not too distant future we may very well have something as easy to operate as the magic light field capture device I wrote about above.
Disclosure: OTOY is a Launch Partner of the upcoming VR Launchpad event which is co-organized by Road to VR.
