John Carmack, the famed developer who helped put the Oculus Rift on the map, has been doing plenty of virtual reality research as of late. Carmack is particularly interested in latency. Specifically, he’s interested in reducing latency to a point that looking around inside of a virtual reality world with a VR headset feels genuinely real. Recently he’s published a paper discussing “latency mitigation strategies.”
Low latency is vital in order to feel like you’re really inside of a virtual reality game. It isn’t as simple as slapping a fast head tracker into a head mounted display and calling it a day. There’s a pipeline of hardware that needs to work in symphony to correctly render the game world according to your head movements, and the round trip through that pipeline can take a significant amount of time.
If the end-to-end latency isn’t low enough, it will mean that your view into the virtual reality world feels sluggish. Carmack writes:
A total system latency of 50 milliseconds will feel responsive, but still subtly lagging. One of the easiest ways to see the effects of latency in a head mounted display is to roll your head side to side along the view vector while looking at a clear vertical edge. Latency will show up as an apparent tilting of the vertical line with the head motion; the view feels “dragged along” with the head motion. When the latency is low enough, the virtual world convincingly feels like you are simply rotating your view of a stable world.
Too much latency not only breaks the illusion of virtual reality, but can also contribute to motion sickness and even make games difficult to play. Minimal latency is essential for a virtual reality experience and it’s a problem that’s not yet solved.
Oculus Inc. said at CES 2013 that they have their end-to-end Oculus Rift latency down to 80ms — though it’s likely that they’re continuing to work that figure down.
In the paper, published at #AltDevBlogADay, Carmack proposes several measures that could be taken to reduce end-to-end latency to acceptable levels. According to him, anything under 20 ms is largely imperceptible. And while it is possible to reach that level with current hardware, it will take some careful work to make it happen.
According to Carmack, in an ideal world sensor data is sent to the computer from the head mounted display and then…
The operating system immediately processes the update, and immediately performs GPU accelerated rendering directly to the framebuffer without any page flipping or buffering. The display accepts the video signal with no buffering or processing, and the screen phosphors begin emitting new photons within microseconds.
“In a typical VR system,” he continues, “many things go far less optimally, sometimes resulting in end to end latencies of over 100 milliseconds.”
It isn’t the hardware itself that’s the issue, it’s simply that modern games don’t require such levels of latency optimization, so the industry has largely avoided techniques that could be used to reduce it.
The paper goes on to describe, at a rather technical level, what steps can be taken to reduce end-to-end latency. If you are a developer looking to make virtual reality games for the Oculus Rift, you’d do well to read this as a primer on some of the latency challenges that you’ll face.
