First Look: Basemark’s ‘VRScore’ Benchmark Arrives With a Unique Solution for Accurate Testing

Basemark has officially launched its virtual reality performance benchmark, VRScore and the package comes complete with an ingenious, low-cost hardware assisted solution to help ensure results are accurate.

I’ve written previously about how tough a problem benchmarking virtual reality hardware and software accurately really is. VR’s tight integration between hardware, application and drivers mixed with platform provider’s varying approaches to performance optimisations, means getting at numbers are indicative or even useful is extremely difficult.

We took a look at Futuremark’s early solution to gathering motion to photon performance which featured a bewildering and expensive array of light sensitive sensor and oscilloscope, an effort to try to close the loop on the latency between application rendering a scene and the user inside VR experiencing it. The solution was, perhaps unsurprisingly, abandoned with Futuremark instead recommending their VRMark software be used largely experientially between hardware setups in order to gauge relative performance. A solution which leaves much to be desired from an analytical stand point.

Enter Basemark, a Helsinki based company who have made a name for themselves across a wide array of performance measurement solutions. They’re perhaps best known however for their widely used Basemark X, their solution to measuring graphics performance on mobile devices. Their new solution is called VRScore and it promises to provide an analytical approach to measuring the performance or your PC hardware and chosen VR headset.

The VRScore package comprises the base performance testing software itself (which comes in both DX11 and 12 guises), and the VRTrek – a device which features optical sensors and is hooked up to your PC’s soundcard Mic-in socket.

The software itself is broken into three sections:

System Test: Purely a test of your PC’s rendering grunt, this runs the Crytek developed ‘Sky Harbor’ VR experience, itself running on the company’s CryEngine software, and measures the resulting frame rate throughout the experience. The benchmark is rendered in stereoscopic 3D and uses pre-baked head motions, presumably recorded from an in-house playtest, to simulate VR headset usage. Each frame from the experience is rendered sequentially as quickly as possible and that sequence run multiple times. The faster you complete the loop, the faster your PC is. This clearly does nothing to gauge performance in relation to your VR headset, for that you’ll need to move to the full test.

VRTrek Test: This is the more interesting of the two benchmark modes as it leverages Basemark’s unique selling point, the VRTrek sensor device. VRTrek tackles a difficult problem (measuring the latency between images being rendered at the PC and displayed on the VR headset) rather elegantly. This benchmark runs the same looped experience as the System Test above, but this time displayed through the headset, ready for latency measurement.

The VRTrek device contains two photosensitive sensors mounted on a height adjustable perspex stand. Tweak the height of the sensor for your VR headset and position them to sit in front of each lens. Then you simply attach the single audio cable to an enabled Mic 3.5mm jack on your PC’s sound card and you’re ready to go.

Why the sound card hook up? Well this is the clever part. The Trek plugs into a PC via that 3.5mm jack (via a spare Microphone-In port); using the PC to coordinate timed flashes on the VR headset’s display, the Trek then sends analogue measurements of brightness right through the microphone port. The VRScore benchmark takes these readings and compares timing of the flash command and the actual flash to determine latency. The device allows the VRScore benchmark to detect dropped frames, frame latency and even duplicate frames received by the VR headset.

Here’s a brief breakdown of technical specifications for the VRTrek device:

VR Experience Mode: Simply gives you the chance to try out the Sky Harbor benchmark experience for yourself and it is well worth doing so. Road to VR‘s Ben Lang was so impressed with VRScore‘s little demo after trying it at last year’s GDC, he remarked that Crytek had “accidentally made the most spectacular cinematic VR short I’ve ever seen”. While expectations and qualitative bars in VR experienced have been raised in the last year, Sky Harbor still remains an extremely impressive demonstration of how transportative VR can be, especially when built with the level of production design present here.

VRScore supports HTC Vive, Oculus Rift and OSVR’s HDK as of writing, for the purposes of this review we used an Oculus Rift.

Other features of note, are the ability to engage NVIDIA Multi-Resolution Shading and Lens Matched Shading, both extremely interesting technologies which aim to reduce GPU rendering burdens for better overall performance (currently DX11 only). You’re also able to run the benchmarks at non native HMD resolutions. We didn’t get the chance to test, but this looked to give the option to test super-sampling.

Alas, due to a shipping delay, our review VRTrek device hadn’t made it to us in time for this article, but we’ll be putting the system through its paces this week for some more detailed feedback. In the mean time, Basemark looks like to have struck a good balance between accessibility, spectacle and analytical credentials with VRScore, we’ll let you know if it performs as expected soon.

VRScore will be launching initially with its Corporate version, with both the Professional and Free versions following in Q2 2017. The differences between the editions are broken down on Basemark’s site here.

Disclosure: Basemark supplied Road to VR with a copy of VRScore and one VRTrek device for evaluation.