It’s notoriously difficult to transmit exactly what’s happening in a VR game without actually strapping on a headset yourself. And while mixed reality capture basically addresses this by letting you see the action from the comfort of a traditional monitor, it oftentimes requires purpose-built green screen stages with external cameras—great if you have it, but it’s time, space and cost-intensive. Owlchemy Labs, the studio behind the critically acclaimed VR game Job Simulator (2016), however recently revealed an ingenious way of watching and interacting with users in VR which may be as simple as pulling the smartphone from your pocket.
Owlchemy Labs recently unveiled research on an experimental mobile app, dubbed ‘Mobile Spectator’, that uses an Android smartphone running Google’s ARCore to track the device’s position in physical space; while the VR headset user sees a floating phone, rendered within the game, the smartphone user has a window into the virtual world, letting you watch a live feed of the VR player, snap pictures, and even interact by doing basic things like tossing beach balls.
The company says in a blog post that they took a “different approach from a traditional multiplayer networking solution” to create the unique third-person view. The phone and PC communicate directly via WiFi. After an initial calibration, its AR-calculated position is sent to the PC, which then places an additional camera in the VR scene—Mobile Spectator’s point of view. The PC renders a frame from this camera, encodes the frame, sends it back to the phone, and decodes it there.
In effect, this lets you walk around the VR user’s virtual environment and freely take in the scene at any practical perspective while the VR user goes about playing the game. While it’s not mixed reality capture (example video at the bottom of the article), giving someone a window into a VR game that they can control helps address the same fundamental task of dissolving barriers between the prospective user and the VR content. As an example, there’s also implications for local asymmetric play too, as a smartphone user might see something rendered invisible to the VR player, or have to give them a key item.
The experimental app was developed using the studio’s upcoming game Vacation Simulator; the studio says Mobile Spectator is however “a formidable undertaking,” which includes performance overhead to the PC with additional rendering and video encoding. The smartphone itself also has its work cut out for it, as it must simultaneously run ARCore and decode video, which all adds noticeable latency.
Although Owlchemy Labs hasn’t announced any official plans to include the app’s functionality in Vacation Simulator, what we’ve seen thus far is really promising.
Owlchemy Labs has been a steadfast pioneer both in object-based interactions vis-a-vis Job Simulator, but also some of the first in-engine mixed reality capture techniques which made creating the sort of reality-bending mashups a quicker and easier process.