Ever wonder how VR tracking systems really stack up? A research and engineering group called the Virtual Dimension Center (VDC) in Fellbach, Germany conducted a study recently that reveals just how accurate some of the top VR headset’s are when it comes to tracking their respective motion controllers.
When most people compare VR tracking accuracy, you typically get a response about how it feels in relation to a known quantity, like when you play Beat Saber across two different headsets to see how they match up. That works when trying to help someone understand how they might perceive the overall experience, but it does little in the way of objectively proving the point.
According to the test conducted by VDC Fellbach, HTC Vive Cosmos’ inside-out optical system consistently shakes out as the least accurate of the bunch.
Note: This article refers to the stock HTC Vive Cosmos, and not Vive Cosmos Elite, which ships with a SteamVR tracking-compatible faceplate.
How It Was Measured
As first reported by German publication MIXED (German), VDC Fellbach measured the tracking systems on Oculus Rift S, Oculus Quest, Windows Mixed Reality, Valve’s SteamVR (aka Lighthouse) tracking, and HTC Vive Cosmos. Ostensibly as a sort of control, the study also included the industrial-grade tracking cameras from ART, which routinely outperformed even the rock-solid external SteamVR laser-based tracking system.
The objective was to quantify each system’s input accuracy, measured in millimeters, which was done by attaching each controller to a robot arm with an adaptor. A 3D-printed head model was used to hold the target headset while the robot arm moved the VR controllers 50 times between two points on the X-axis, set 500 millimeters (~20 inches) apart, or about an arm’s distance away.
The robotic arm, which was connected to the laptop via LAN, was controlled by an in-house software using a modified variant of OpenVR. This allowed the testers to not only reliably know the actual position of the controller, but also the reported position of the controller, capturing data in the X, Y and Z-axis during each headset’s trial.
The Results
Below you can see the standard deviation of the results for position one, which is closest to the headset. The study also includes the maximum deviation and mean, however they all point to more or less the same relative conclusion.
SteamVR tracking, which uses laser-based ‘outside-in’ beacons, is the most accurate consumer-grade solution of the bunch. Notably, ART’s outside-in optical tracking is so accurate that its bar isn’t even visible on the far right of each chart.
Windows Mixed Reality performs better than we expected in position one, coming in neck-to-neck with Oculus Rift S. HTC Vive Cosmos lags behind by a wide margin with its 2-3 mm deviation. This changes somewhat when the respective controllers move to position 2 however.
Outside-in tracking remains more or less the same regardless of the distance between the headset and the controller, however inside-out tracking takes a notable hit in accuracy as it moves farther away from the headset.
In one instance, Windows Mixed Reality underperforms by nearly 3 mm from dead zero, beating out Vive Cosmos by nearly a millimeter in accuracy.
The charts above don’t tell the whole story though. According to the study, HTC Vive Cosmos performs “worst,” with deviations that are “sometimes above 5 mm,” and that it was even possible to visually detect Vive Cosmos’ deviation while the measurement was being carried out.
Granted, Vive Cosmos has benefitted from multiple software updates since its late 2019 launch, which was so bad at the time that we actually postponed our initial review until HTC addressed our concerns over tracking. According to this study though, it seems that well into late 2020 that Cosmos is still underperforming in the tracking department.
The testers admit their findings may be influenced by “differences in the hardware components used and different software,” however it’s unlikely a research group capable of creating such a testing platform in the first place would simply forget to update their drivers. At least we’d hope not.
– – — – –
If you speak German, or don’t mind flexing your Google Translate skills, you can check out the full paper to see VDC Fellbach’s findings for yourself. A special thanks goes out to our friends at MIXED for pointing us to the news.