Futuremark, the 3D performance benchmarking company behind 3DMark, is working on VRMark, a virtual reality specific test the capabilities of your gaming PC and VR headset, using some ingenious hardware.
PC gaming and benchmarking have gone hand in hand for many years now. For some hardcore hardware enthusiasts, proving how capable your ‘rig’ was, seemed somehow more important than playing the games it was built to run. 3DMark from Futuremark is one of the longest running so-called ‘synthetic’ benchmarking suites, with each iteration of their software accompanied by a set of cutting-edge visual rendering and calculation tests designed to cripple any PC not technically worthy.
For virtual reality however, if your gaming system is running anything other than optimally, the resulting experience isn’t merely irritating, it can make you ill.
Now, just in time for the release of the first wave of commercial VR headsets, Futuremark are ready to enter the virtual reality space too. VRMark is a benchmark designed to ensure that your virtual reality hardware stack, from PC to HMD, can deliver a comfortable and compelling VR experience, and we’ve got our hands on the hardware it uses to gather its data.
VRMark faces a challenge unlike its well established benchmarking stable mates however. 3DMark can calculate how fast your CPU and GPU is able to crunch numbers to generate a 3D scene ready to send to a traditional display, entirely within the PC being tested. However, the response time of the display components within the VR headset itself, a critical factor dictating user comfort and immersion, cannot be so easily determined.
For VRMark, Futuremark is working on a solution to these benchmark quandaries using a neat collection of hardware components to gather metrics externally. By measuring the photons emitted by a VR headset in response to draw calls or display control requests made by the benchmark software, that data can be mapped and used to gauge relative performance of the hardware. Whilst we’re not permitted to write about the individual instrumental components in detail, Futuremark have allowed us to show you them.
The kit comprises a USB-attached oscilloscope, photo-sensitive meter also connected into the scope. This metered ‘loop’ allows photons generated by the VR headset (with said photo sensitive sensor placed on one of the headset’s lenses) to be detected and measured in response to actions called for by the benchmark. For example, the latency between a draw call being initiated and the resultant image displayed by the headset. Crucial stuff for sure, but the team have already gone way beyond this baseline metric, claiming they’re besting display manufacturers own tests in some areas.
Futuremark’s James Gallagher tells me “This new version of VRMark now finds the key latency events automatically and reports these results in the UI. We’ve implemented our own bespoke event finding algorithm that significantly outperforms the method commonly used by panel manufacturers,”. Gallagher also added “You can measure and report rise time, fall time, response time, pixel persistence, frame persistence, frame latency, and total draw latency. The result chart now shows the timing of vsync events. And named ranges in the monitoring data highlight specific spans used in the results.”
This set of hardware is designed for specialist media use, given the requirement for extra hardware that isn’t particularly cheap. This early development version of the software includes one part of the testing suite – focused on determining the VR headset’s display capabilities. So where does Futuremark see VRMark being pitched, in terms of target audience?
“The general way it will work is that VRMark will have high-level and low-level tests,” Gallagher says, “High-level tests look at the performance of the whole VR system for various VR content types. These tests will answer questions like, “Can this smartphone handle VR photo and movie content?” or, “Is this PC capable of running the latest VR games with a high and consistent frame rate?” If there is a consumer version, it will include these high-level performance tests since no other equipment is required other than the device being tested.”
“VRMark‘s low-level tests measure specific aspects of hardware performance such as latency, image quality, and implementations of asynchronous time-warp. These tests tend to require external hardware, like the kit we supplied to you. These tests will only be available to professional customers. Working with our parent company, UL, we also plan to offer lab-based verification and certification for manufacturers and software developers.”
I suspect if there was anyone out there in any doubt as to Futuremark’s commitment to ‘serious’ and above all useful virtual reality metrics gathering, the above statements may have changed their minds. If not, let me tell you that having used the latest in-development version of VRMark along with its pre-requisite hardware, it’s an impressive set up.
Measuring how comfortable a VR experience certain hardware combinations can provide is an immensely nuanced area however. Engineers, game designers and programmers are still learning about visual and gameplay cues which are or are not conducive to a ‘good’ experience. Ultimately, every user is different. Their physiology and their psychological response to immersion dictate how sensitive they are to the minute discrepancies between virtual worlds displayed and their physical senses (that inner ear is a tough one to fool). But if Futuremark can provide an accurate way to measure and report on the chain from software to hardware, this is at least a very good start.
As for that high-level experience VRMark are aiming to produce, the company released an optional Beta update to their 3DMark benchmark on Steam just before the holidays. Although it serves only as a preview for where Futuremark intend to go with High-level benchmarking, I’d highly recommend you follow these instructions to go check it out. It’s actually one of the best introductory VR experiences I’ve seen in quite some time. The developers have cleverly lifted assets from previous 3DMark releases and plonked them into a museum environment, complete with dioramas of those benchmark scenes for you to peer into and around whilst in motion. It’s impressive stuff. However, how such an experience can be measured, when user input (specifically head motion) will vary every time a ‘live’ benchmark is run.
We’ll be diving more deeply into VRMark testing capabilities and posting some results in the not too distant future. In the mean time, it’s clear that Futuremark are taking their bid for supremacy in the potentially important VR benchmarking space very seriously indeed. With the software evolving rapidly and with the first consumer VR headsets just months away, we’re looking forward to seeing how the new headsets measure up.
Disclosure: Futuremark provided testing hardware and software used in this article.
