Tis the season of 5G, the next-gen mobile data technology which promises to boost bandwidth and reduce latency. With 5G phones and networks slowly starting to become a reality, carriers would have you believe that the tech is going to radically change the world, and then some. But what does 5G actually mean for VR & AR?
As the cellular industry’s hottest new technology (and hopeful revenue driver), every stakeholder from cell phone makers to chip manufacturers is playing up 5G as the start of a revolutionary shift—and what better way to do so than to associate it with buzz-worthy technologies?
You can Google “5G” along with any other hot technology in the same query and you can find someone telling you that somehow and at some point 5G is going to revolutionize that thing. I’m not even joking. Let’s take a quick survey:
The list goes on and on, and it of course includes 5G + VR and 5G + AR, with terrible articles like this one that fundamentally butcher the relationship between the technologies with sentences like these:
AR and VR depend on decreased latency to run efficiently. The user experience will be amazing with a reduced lag time. With a better UX comes more opportunity. Telemedicine, virtual training environments for business and more will be the result.
In an effort to combat this hand-waving nonsense, let’s talk about what 5G could actually mean for VR and AR. The easy way to do that is to first understand what 5G actually brings to the table.
What Does 5G Mean For You
Quite simply: 5G promises greater bandwidth and lower latency than current mobile data connections offer. It also has the potential to make such high performance networks more accessible to more people.
Exactly how fast and what latency 5G will provide to individual end users won’t be clear until the technology is actually rolled out at scale, but mobile industry marketing would have you believe that 1Gbps download bandwidth and 1ms latency is possible.
To quickly put that into perspective, the average mobile and home connection speed in the US was 33Mbps and 112Mbps, respectively, in January 2019, according to internet metrics company Ookla. As for latency, 35ms for mobile and 5-15ms for home connections are common. Roughly speaking then, 5G could offer 30x and 9x the bandwidth of mobile and home connections, respectively, with 35x and 10x faster latency.
But what in the realm of VR and AR is actually enabled or enhanced with more bandwidth and less latency?
Immersive Video Streaming
When it comes to streaming non-interactive 360 or 180 video content to VR and AR devices, one major misconception is that 5G’s low latency will be a huge benefit. In fact, the opposite is much more likely to be true—it’s 5G’s big bandwidth which opens the door to significantly improved quality which would not only improve the immersive video experience of today, but also offer a pathway for further growth as VR and AR displays become higher resolution.
The reason latency hardly matters for non-interactive streaming video content is that headtracking on VR and AR headsets is not coupled to the streaming latency. In most incarnations of 180 and 360 immersive streaming technology, the visual adjustments from the movement of the user’s head are processed locally on the device at a high rate, rather than sending the motion to some remote server and waiting to receiving an updated frame which corrects for head motion.
In fact, on today’s VR headsets you could be in the middle of watching a 360 video and then completely lose connection and the only impact would be that the video would freeze—moving your head would still respond just fine because the scene’s movement with regard to your head is processed right on the device.
There’s some nuance to this: some more sophisticated VR and AR streaming uses ‘view-dependent’ technology which optimizes the quality of the footage specifically in the area that you’re looking (in an effort to minimize the bandwidth bottleneck). High latency in this case could mean that if you turn your head quickly to a new part of the scene, you might notice that the video isn’t as sharp until the higher quality section of the video pops into view. In this case lower latency will make those transitions smoother, but it’s more of bonus than a radical shift in the experience.
When it comes to bandwidth however, 5G could significantly improve 180 and 360 video streaming quality. The most immersive footage—360 3D or full volumetric video—is also the most bandwidth intensive and is used sparingly today in a large part because most users don’t have sufficient connections to stream this kind of content in high quality.
Cloud-rendered VR and AR Gaming
When it comes to interactive VR content, 5G could make cloud-rendered VR and AR gaming possible, but this use-case is still largely hypothetical.
The pitch goes: users will own a simple inexpensive headset that streams high-end PC VR or AR graphics right from the cloud with no PC needed! It sounds amazing, but you aren’t going to see this happen until well after normal non-VR game streaming matures because VR demands significantly higher resolutions and lower latency than non-VR game streaming (which is still struggling to meet the performance needs of competitive gamers).
While latency isn’t really that important in the case of non-interactive video streaming, here it is crucial because what’s being rendered needs to change in real-time based on the actions of the user. So if 5G can truly offer up, let’s say realistically, 4ms or less latency, it really could unlock the cloud-rendered use-case of VR and AR.
But that’s a big if. Not because sub-4ms latency isn’t possible, but because you need that level of latency not just to your nearest connection in the network, but specifically to the location where your frames are being rendered.
If you run a network speed test right now, the test will very likely connect to a server that’s very close to you. Your latency is the round-trip time that it takes to get information to that server and back. If you test again and select a server that’s a few states over, you’ll find something very different. I just did this exercise and saw 10ms of latency to a server in my city and 35ms to a server that’s three states over.
This is to demonstrate that network latency increases as a function of your physical distance to the servers you are pinging. If you want to render VR and AR content in the cloud at low latency, the user needs to be physically close to the location where the frames are being rendered. This is much more difficult to achieve at scale because the frames of an interactive application have to be rendered on the fly.
When it comes to non-interactive video streaming, it’s relatively trivial to distribute the pre-rendered video files to servers across the globe ahead of time, and then serve them over the network as needed. Real-time rendering of VR content requires high-end GPU hardware in each data center from which cloud rendering would happen; each properly equipped data center can only deliver those frames with low enough latency to users who are within a certain physical distance.
That is to say that a single data center in the middle of the continental US, for instance, would have too much latency by the time it reaches the country’s coasts for a viable cloud-rendered experience. A distributed cluster of capable data centers (also known as ‘edge computing’) is the key to making this viable; 5G helps by expanding the ‘coverage’ of capable data centers by reducing latency, while high bandwidth makes greater quality imagery possible in this use-case (which will be increasingly important as the resolution of VR and AR devices increase).
Though it may take both 5G and edge computing to make this all possible, the good news is that cloud-rendered VR and AR services can piggyback off the same infrastructure that’s being deployed for non-VR game streaming services like Google’s Stadia and Microsoft’s xCloud.
