nvidia-cloudxr — Image courtesy Nvidia

NVIDIA CloudXR 3.0 Update Brings Bidirectional Audio Support for Remote Collaboration

By

-

Jun 28, 2021

NVIDIA’s CloudXR technology is designed to provide real-time XR cloud streaming, which aims to remove the need for local high-end PC rendering hardware. At Mobile World Congress 2021 today, the company announced that its CloudXR 3.0 release will include bidirectional audio support—a fundamental piece of the equation when it comes to collaboration and social interaction.

Rendering immersive XR experiences in the cloud, compressing the video and audio streams, and serving that up to VR and AR devices for interactive experiences over 5G and WiFi networks—all at a low enough latency to make it feel like you’re tethered to powerful PC—is no small feat. That’s XR cloud streaming in a nutshell.

Although it’s been two years since the company first publicly demoed the technology, Nvidia appears to be making good headway. It’s already brought CloudXR client applications to PC, HoloLens 2, Android VR devices (including Oculus Quest), Android AR devices, and iOS. It offers support for OpenXR, bringing SteamVR content to those devices. It has an SDK for the service and has also brought CloudXR to the three leading cloud computing platforms: AWS, Microsoft Azure, and Google Cloud.

Valve Releases Hand-tracking Passthrough for Steam Link, Making Quest an Even Better PC VR Headset

Now, the company announced it’s added bidirectional audio so developers can bring a host of social-based apps to the cloud rendering platform which were previously impossible. Besides opening the door for early testers like Accenture and Autodesk to integrate their own VR and AR collaboration-based productivity apps, the company says its CloudXR 3.0 release features:

iOS client improvements including world origin updates
Asynchronous connection support, so clients are kept responsive during connection establishment
In the Wave™ VR client, improving color and gamma handling
Updated Windows SDK from 8.1 to 10
Multiple API changes — see CloudXR SDK API for additional details
Various bug fixes and optimizations

Ronnie Vasishta, senior vice president of telecommunications at NVIDIA, is set to speak more about the 3.0 release at Mobile World Congress this week in a talk called ‘AI-on-5G: Bringing Connected Intelligence to Every Industry’. We’ll be tuning in here on July 1st at 8:00 AM CET (local time here).

There, Nvidia says Vasishta will “share how the fusion of enterprise AI-on-5G at the edge opens the doors to multi-trillion dollar industry transformation and new capabilities for smart cities, security systems, retail intelligence, industrial automation and optimization of network capacity utilization.”

kontis

This type of apps and services (VR streamed from a server) are not allowed on Oculus Store or even AppLab.

I assume this is meant only for Enterprise and only those who use the more expensive commercial version of Quest with no real path for anything based on this tech for consumers…

Funny thing is Nvidia and Amazon brag about playing HL: Alyx and Boneworks on Quest with this. So they clearly want mass consumer adoption of this tech.

EDIT:
I just found this statement from Nvidia employee:

Correct, NVIDIA CloudXR at present is targeted at Enterprise and the OFB distribution is the target. The “no streaming apps” stance in the Oculus store is consistent from early on. The fact that Oculus has allowed VR Desktop into the store is actually a step in a positive direction.
- Christian Schildwaechter
  
  At one point Facebook will have to give up their “no streaming apps” policy, simply because streaming will be the only way to get content at very high graphical fidelity onto any mobile AIO HMD, and VD and AirLink both have proven that this is a viable option. CloudXR actually improves on this by allowing some rendering on the device, e.g. the UI, while only the performance heavy parts are streamed.
  
  Facebook doesn’t like Streaming, because it is the most serious threat to their dominance in consumer VR. They kind of had to offer link via USB, so they could drop the Rift and get everybody onto Quest, but made it somewhat unattractive by charging USD 80 for the cable. People used it anyway with much cheaper USB 3 cables, still making it as inconvenient as tethered PCVR.
  
  Then VD enabled wireless streaming, taking away one of the main advantages of native Quest apps, and Facebook promptly forbid it, citing quality issues. People found a way again by patching VD via Sidequest, so Facebook had to introduce AirLink. AirLink is limited to local streaming, so you still have to have a powerful gaming PC with you, making it inconvenient.
  
  But again VD allowed streaming from a remote host, if you have the competence to set up your network appropriately, proving that Cloud VR is actually viable. The last hurdle is now that you usually have to own the PC. If this last hurdle falls, everybody with a somewhat decent internet connection can work around the Oculus store and the whole Facebook platform.
  
  It will actually be worse, because the software and hardware requirements for a HMD that does streaming only are much lower than for an AIO that has to render itself. Maybe there could be an Nvidia Shield VR, a slim HMD with a simple SOC that only does video decoding and tracking with a minimal amount of RAM and flash? They could produce it at a much lower cost than the Quest 2 and subsidize it by having it require a USD 10 Geforce Now subscription, which already allows you to stream many of the games you own on Steam.
  
  Sony can keep people from switching to PC or Xbox due to their strong exclusives. Facebook doesn’t have a strong software catalogue, they keep people from switching by subsidizing hardware so heavily that nobody else can compete. Take away that advantage, and Facebook’s platform plans are screwed, so they want to prevent cloud streaming. But at some point someone will offer something like a shield VR, and Facebook will be forced to compete, at least by offering their own streaming service, which in turn will make it hard to lock out others like CloudXR.
  - kontis
    
    Of course they will allow it one day when they have their own solution.
    
    And for 3rd party solutions they will probably do the same thing Apple did – every streamed game as a separate product listed in the store with the 30% cut going to FB/Apple.
    
    Sony can keep people from switching to PC or Xbox due to their strong exclusives
    
    FB isn’t scared by PSVR because they don’t consider gaming their main target and that focus is only temporary because Quest is a brick toy and more like a Switch than the iPhone killer glasses they hope to introduce in the “near” future. They want billion casuals and working people, not just millions of geeks and nerds. Sony also doesn’t have any big VR titles. Beat Saber is owned by Facebook and I wouldn’t be surprised if they don’t properly support PSVR2 with it (and will never release it on any Apple XR device, which is the main threat for them).
    - Christian Schildwaechter
      
      I’m pretty sure regulators will go and rain into Apple’s strange argument that you can stream Netflix movies for free, but that they have to check every single game you stream and therefore can demand their 30% fee. Which is quite ridiculous. So I doubt Facebook will be able to enforce that either.
      
      And I agree about the “Facebook wants social XR” and games just being a stepping stone. AR AIOs will become a much hotter battleground, but we are still some years away from that, and anything that leads to more platform diversity until then should be welcome.
  - RSIlluminator
    
    The “no streaming apps” policy will definitely go away at some point, but how does something like CloudXR realistically scale to thousands of users? The last time I looked, all of those nodes on AWS cost hundreds of dollars per week per user, because you’re essentially commandeering a high end machine with a server grade gpu for one person.
    
    While this technology is promising, at the moment it seems that any sort of pixel streaming is offsetting the cost from the user to the business offering the VR experience. I’ve seen 3d streaming prices range from about a dollar per hr/ per user on a dedicated non-scaling machine, to 3.6 dollars per hr/ per user on a scalable set up. In order to recoup that cost, I can only see this being used for something like b2b at the moment.
    - Christian Schildwaechter
      
      Usually data centers run non standard hardware, e.g. high core count CPUs with multiple GPUs virtualized into several machines, but let’s simplify and do some math:
      
      Assuming that the machine you rent is a USD 1500 PC that is replaced every three years, it has to earn USD 500 per year or about USD 42 per month to brake even. If it runs 24/7 @ 300W, it consumes about 220kWh per month. Data centers get their energy cheaper, let’s say USD 0.2 per kWh, adding USD 44 to a total cost of USD 86 per month. Geforce Now costs USD 10 per month, so nine users sharing one cloud PC are enough to break even, which means 2:40h daily playtime per user.
      
      In reality there are added costs for administration, but the energy costs are actually lower due to a) the machines not always running at full power and b) much lower prices (e.g. for data centers build near hydroelectric power plants). Most users will not even have the time to play for 18:40h every week, but play time will not be distributed evenly.
      
      With a sufficiently large number of users, you can distribute cost and calculate how much you have to charge for a standard product. You can adapt the terms of service, e.g. allowing for a reduction in performance at peak times. Unused time can either be given to free tier users (Geforce Now does this), be sold/auctioned off at lower prices (AWS) or the machines can be shut down to conserve energy.
      
      Geforce Now further reduces costs by not allowing you to install your own games, instead they have a large list of games from Steam/Epic/Uplay preinstalled. If you own one of them on these platforms, you can play them, which reduces the user specific storage requirements to a few MB of game save data. And you have to pay the monthly fee even if you don’t use it.
      
      All this makes it a lot cheaper than renting a dedicated machine on AWS for just a few hours. And even the higher priced AWS instances are often cheaper than buying a dedicated machine, if you actually calculate usage time, administration cost and energy prices for consumers. In the end it is all about accounting.
- Darlene Moore
  
  I’m able to collect $88 every hour to do easy tasks staying at home..~or304~ I have not at all imagined that it’d achievable but my close friend has obtained $27k only within a month by doing this super opportunity as well as she had convinced me to try…~or304~Check extra information on visiting following web page. >>> https://c2d.in/recognize70
  .
- TonyVT SkarredGhost
  
  Actually I have downloaded and tested CloudXR completely for free
TonyVT SkarredGhost

Having tried it, I can confirm it is a great product. It’s cool to see that it is evolving so fast