NVIDIA Research has demonstrated a new method of foveated rendering which the company says is nearly invisible to users. Researchers behind the new approach say the method can drastically reduce rendering workload, allowing for more detailed virtual reality scenes.
The human eye only really detects significant detail in a tiny area at the center of our field of vision, thanks to a small pit in the back of the retina called the fovea. Our peripheral vision, while useful, sees things like color and movement, but very little high fidelity detail. For instance, look one inch to the left of this paragraph, and see if you can read any of the words in it. I’ll wait.
You’ve probably found that it’s nearly impossible. Foveated rendering takes advantage of this fact and supposes that, when it comes to CGI, we can get away with rendering the area outside of the fovea at lower detail, thereby being able to focus more of the available computing power in that smaller area for richer scenes.
Foveated rendering isn’t new. It’s been around for some 20 years, according to Nvidia. But contemporary techniques resulted in objectionable artifacts like flickering or blurring which users could see too easily. So Nvidia researchers sat down to try to understand what it is that our peripheral vision is and isn’t adept at seeing.
The results of their work, soon to be published in a paper titled Perceptually-Based Foveated Virtual Reality, found that our peripheral vision is best at seeing things like color, contrast, edges, and motion. Thus, the ideal foveated rendering technique should maintain those aspects of the scene while reducing high fidelity detail. Nvidia says they used this knowledge to design a new rendering algorithm that allows for greater reduction in detail without users being able to detect it.
“When we started this project, the researchers working on it knew if foveated rendering was turned on or not. By the end, even they have to ask [whether or not foveated rendering was enabled],” said Aaron Lefohn, one of the Nvidia researchers who worked on the project.
Click to enlarge photos in the gallery above. Look at the clock highlighted by the red square. While looking at the clock, use your arrow keys to switch back and forth between the ‘blur’ foveated rendering and ‘contrast preserving’ foveated rendering. Notice the different of clarity in your peripheral vision.
The results are impressive compared to other methods. By maintaining the sorts of details that our peripheral vision is best at seeing, the rendering technique restores a feeling of clarity to the areas outside of the center of our vision. With this method, Nvidia says they’re able to reduce pixel shading performance by 2x-3x compared to a scene not using foveated rendering. Those performance savings can be used instead to make the detailed part of the image even sharper and more realistic.
“There’s this insatiable demand for more pixels and more performance [in virtual reality]. The benefits we can get from foveated rendering are tremendous and growing,” said Lefohn.
Nvidia will be demonstrating the new foveated rendering method at SIGGRAPH 2016 next week.
For use in virtual reality, foveated rendering relies on eye-tracking so that the scene can be continuously updated to render sharply in the very center of the user’s vision, with the foveated rendering only kicking in on the peripheral. Nvidia partnered with SMI which supplied the researchers with a VR headset with inbuilt eye-tracking tech capable of accurately tracking the eye’s gaze direction 250 times per second.
“You can’t outrun [the eye tracking] with your eyes. It’s fast enough and accurate enough to give us a foundation to build [foveated rendering technologies] on,” said Lefohn.
So we won’t be seeing foveated rendering in consumer virtual reality until VR headsets adopt eye-tracking technology. Japan-based FOVE raised $11 million following a successful Kickstarter to develop a consumer VR headset with built-in eye tracking. Major headsets like the Rift, PSVR, and Vive have yet to ship such technology, but many suspect it may be in the works for future generations of the headsets.