Limitations
Although 360 capture is slowly becoming available in more and more applications, it still faces a number of limitations compared to traditional monitor-based capture tools like Fraps, OBS, and ShadowPlay.
Requires application support
With regular capture tools, it’s easy to capture video of virtually any application. The application itself does not need to provide any special functionality to support this. Currently, 360 capture is only available in apps in which the developers add special support for it, and access to the source code of the project is required.
In principle, the same kind of injection technology used by tools like vorpX and Vireio, which can manipulate the camera in certain DirectX titles, could be used to capture 360 images, but no one has yet implemented this. This approach would not suffice for 360 video, since it cannot be recorded in real-time, and would also struggle with artifacts from screen-space effects.
No real-time video
Traditional monitor-based capture tools capture applications at full resolution and frame rate in real-time during play with little overhead. As noted above, 360 video must be very high-resolution, at least 4K pixels wide. At least 6 different views of the scene must be rendered per frame (more for stereoscopic capture). Additionally, in VR it’s essential to maintain a high framerate on the display to ensure a comfortable experience. All this results in a capture framerate of less than 10 FPS.
Since this is unacceptably low, the only alternative is to render the video offline. For on-rails experiences like Welcome to Oculus, Senza Peso, Colosse, or most roller coaster apps, where the player is merely an observer, this is simple: just step one frame at a time and capture each frame, then proceed to the next frame. Audio is added in postprocessing.
For other applications where the player is actively controlling the action, it is necessary to implement a frame-by-frame replay system which captures the player’s actions and then replays them afterwards while capturing each frame. Ensuring that the replay reproduces the original scene exactly may be tricky in certain applications that include elements of randomness or examine wall clock time.
Stereo equirectangular format
The stereo equirectangular format, with separate left-eye and right-eye images, is easy to implement efficient viewing software for, easy to compress and store using existing image and video formats, and is compact enough to download and stream. However, even with all the rendering fixes outlined in previous sections, this format still has fundamental limitations.
For one, it cannot properly display stereo views where your head is tilted with your ear on your shoulder (roll). Convergence will not be possible if you attempt this. Another example: if you lean your head so far back you can see behind you, your view will be reversed. Even with your head pointed straight forward, if you use your eyes to look left or right, you will not see the correct view toward the edge of the frame, and this issue becomes more severe as HMD FOV increases. The distance between the eyes (IPD) is “baked in” to the image, which will produce a distorted sense of scale for those with unusually extreme IPD values.
Finally, it can’t provide any kind of positional tracking – in addition to preventing the viewer from leaning left, right, forward, or back while viewing, this also causes incorrect parallax during head rotation, because the eyes are rotating around a fixed point between the eyes, rather than around the center of the neck as they do in real life. This manifests as a perceived motion of nearby objects during head rotation, even in static scenes where those objects should remain still. (It may be possible to mitigate parallax problems with the light-field techniques described before, but this may also exacerbate artifacts.)
Where is the player looking?
In a traditional video, it’s easy to see where the original player was looking during gameplay. In a 360 video, the viewer takes over all head movement, and there is no longer any clear way to tell which way the original player was looking. There’s much room for experimentation in this area, but here are a few ideas:
- World-space UI and spatial audio: An arrow might indicate which way the player was looking; a square or circle may highlight the portion of the scene they were looking at; the portion of the scene they’re looking at may light up as they look at it, as though their eyes were shining light on the scene like a flashlight; a spatial audio source such as a hum sound may be located in the direction they are looking.
- Third-person camera with head animation: A viewer is, in some sense, another person hanging out with the original player, so being inside their body is weird, even in a first-person game. By placing the viewer outside the original player’s body, you can watch their body and head turn as they explore and look around, while also having the freedom to look around yourself. A third-person camera also offers the ability to smooth out motion and reduce discomfort when moving around. However, as with all third-person cameras, they can get confused or move unpredictably in cramped environments, like a tiny hut in Minecraft.
Conclusion
Capturing of real-time rendered content in engines like Unity and Unreal Engine 4 offers exciting possibilities for VR developers and gamers to share their creations and experiences while letting their friends be just as immersed as they were, with full freedom to look around. The tools and techniques are available now to begin integrating this into a variety of applications, but a number of daunting obstacles remain before 360 capture will be as fast and straightforward as video capture is today.
All rights to all original content in this article waived under CC0 Public Domain Dedication.
