htc vive pro review (7) — Photo by Road to VR

Vive Pro Gets AR Capabilities Thanks to HTC’s New Tools for Front-facing Cameras

By

-

Apr 16, 2018

HTC has announced a new set of tools allowing developers to build applications which take advantage of the Vive Pro’s stereo front-facing cameras, effectively turning the device into an AR headset dev kit. The new tools allow the cameras to capture depth, spatial mapping data, hand input, and seamlessly shift between VR and AR worlds.

While the original Vive launched with a front-facing camera, it went sorely unused. This time around, with the Vive Pro, the company is offering the VIVE SRWorks SDK. Announced last week, HTC says that the SDK includes three modules: a depth module, see-through module, and a 3D reconstruction module—effectively a foundational set of tools enabling the headset to sense the world through its front-facing cameras, and allowing developers to use that data for creating interesting experiences that can be pure AR or VR, or a combination of both.

HTC Vive Pro Headset Review: Welcomed Improvements Overshadowed By a Steep Price

This example video, using the Vive Pro and the SRWorks SDK shows how the tools can be used to seamlessly link VR and AR worlds, creating new and interesting gameplay and application possibilities:

The company says that the SRWorks SDK includes support for native development with plugins for Unity and Unreal, and that the modules can enable the following:

Depth

Spatial Mapping (static and dynamic meshes)

Placing virtual objects in the foreground or background

Live interactions with virtual objects and simple hand interactions

This example shows how the Spatial Mapping module can create a model of the room’s geometry for use in applications:

The Vive SRWorks SDK is available in beta through the company’s developer portal.

Like the ZED Mini depth camera add-on, thanks to the new tools, the Vive Pro’s front-facing stereo cameras effectively emulate the sort of experience that AR glasses will ideally achieve in the future—an immersive, wide field of view with precision tracking and environment mapping, potentially turning it into a great dev kit for developers building toward XR hardware of the future.

VR Geek

I have read several times now that AR via a pass through camera does not feel right as your eyes suddenly are shifted forward to where the cameras are. I would be curious if HTC solved this issue, but I doubt that very much. This announcement is more like a desparate plee to entice suckers in their overly priced HMD.
- benz145
  
  It’s true that the perspective will be incorrect, though the problem is identical for any pass-through AR headset. Camera feed latency is also a problem. However, passthrough AR on a VR headset like the Vive Pro has other advantages (like high resolution, great tracking, and a wide field of view) which today’s opaque AR headsets haven’t achieved.
- dk
  
  u can have a software adjustment of the video correcting for that to some extent but it’s not perfect….it’s basically a built in zed mini ….but it’s mainly for developers to experiment with ar …..not really a great working consumer ar product…… also I’m hearing the res and latency is not great
- Lucidfeuer
  
  Yup, that was the problem from the beginning, adjusting the perspective is quite (but not so difficult) at least for far away elements, it gets complicated for closer ones like your hands for adjusting the perspective. But as most things, solution already exists thanks to software research, it’s just that they’re not integrated and implemented by corporations…
- Graham J ⭐️
  
  Just a thought but if their SDK is capable of realtime reconstruction of the environment then you should be able to move the virtual camera location back a bit.
- NooYawker
  
  When I look through the Vive camera everything looks enormous.
Harris

Curious regarding specifications of the box used to drive the spatial scan demo.
dk

it will be cool to check out the specs of the cameras ….res fov fps…added latency
one dude was saying that from the sdk the specs don’t look good….vga res and 200ms latency
dk

http://community.viveport.com/t5/Developer-Blog/Announcing-VIVE-SDKs-for-the-Vive-Pro-front-facing-cameras-and/ba-p/17573
Badelhas

What about the normal Vive? Will the camera ever be be used?
- Laurence Nairne
  
  Original Vive only has one camera and I don’t think they intended for it to be used in an AR context. It can do passthrough and chaperone type tasks, but nothing to the degree that this video shows on the Pro.
Jean-Sebastien Perron

Yet the Vive Pro (Problems) cannot even proper headphone and microphone.
- NooYawker
  
  We get it. The vive killed your puppy. You’re incessant whining about how much you hate the Vive is pathetic.
  - Flamerate1
    
    After snooping around his profile like the person with good internet etiquette would do, I have confirmed that he is indeed a vive hater.
    
    Quick question for Jean: Do you have a vr headset and if yes, then which one is it? This is of course to understand your bias if you have one.
    - Jean-Sebastien Perron
      
      I have a PSVR, Rift and now a Asus MR. My friend has an HTC Vive. I have nothing against Vive itself, but the way the company treat it’s customers is illegal and fraudulent. Vive and Vive pro are the most expensive VR, yet they are at the bottom of the quality in term of graphics and sound quality. Their controller is almost as bad as PSVR.
      - Flamerate1
        
        Ok, I see. I don’t necessarily see them to be fraudulent, but the price (as well as the initial coverage) is why I bought the oculus.
      - Jean-Sebastien Perron
        
        It is fraudulent and illegal, they changed the warranty to not cover sweat damage after the product was bought. They ask you to send it for free for repair. They they kidnap your Vive and won’t return it until you pay a 45euros ransom (not repaired). Breaking after 2 months because of sweat is a design flaw and should be covered by warranty because it is normal use of the product. https://youtu.be/ymhQxBalzCE
      - jj
        
        umm theyre certainly not at the bottom of quality in terms of graphics and sound quality…. Thats just bs and you know it. jean is a troll he uses multiple accounts to bash things he dousnt like and references his friends youtube video as click bait. dont fall for it
      - Jean-Sebastien Perron
        
        I only have one account and this is my real name, i am not a troll like you. Stop attacking the messenger and try to bring some arguments about the subject.
      - jj
        
        seriously jean? anybody can look at your post history and realize you’re full of shit. They can also see that you only follow/are followed by one other account who suspiciously has similar posts to your narrative, almost word for word.
        
        Its funny you try denying being a troll, look at every post you’ve made on RtVR before this one and you’re clearly a troll.
        
        Don’t come inhere with that bs that you’re not a troll and were just attacking you because you have brought the more harmful vocabulary and posts than anybody on here and there’s over a dozen people reading this right now that will agree with me.
        
        Some of my posts arent nice but none of them are troll posts like that majority of yours clearly are. Plus you cannot defend any statements you’re outcome is always posting that one video as click bait. I post more on here about how annoying you are than about vr because of how actively you troll.
      - Jistuce
        
        ” in Canada.”
        Pffft. As if anyone cares about Canada.
      - chris
        
        thats rubbish the controllers on vive are brilliant what is crap about vive is the none user replacement battery scheme they have reminds me of apple and it´s very frustrating to be treated like an idiot and overcharged for a battery! especially when i want 6 batteries and i´m quite capable of changing them myself
Dave

Does anybody care about the Vive Pro, I mean really?
- Jistuce
  
  Yes. It is where I get a good percentage of my laughs lately.
- jj
  
  now i do
D3Pixel

That looks quite good if i’m honest. Certainly good if you want to start understanding and developing AR apps for the future,
- Graham J ⭐️
  
  I think that’s the key here – not that it’s necessarily a great AR experience but that it gives you enough to get started developing for an AR future. There aren’t a lot of solutions for that right now.
NooYawker

AR through a camera isn’t the same as AR glasses. But it’s a nice addition. If you have a Vive Pro that’s is.
- Sponge Bob
  
  if latency is minimal it can be practically the same – with higher res cameras
  
  this is not new btw
  
  See Vrvana Totem headset sold to apple a while ago
  - jj
    
    maybe in a few years but as of now nothing supports the bandwidth required for these
    - Sponge Bob
      
      Bandwidth?
      This is all tethered to pc
      - JJ
        
        yeah bandwidth of usb cables
      - Sponge Bob
        
        yeah
        
        I guess that’s why lo-res VGA cameras on Vive Pro
        
        …and no mixed reality for Microsoft’s “Mixed Reality”
Zach Mauch

This is exactly what I’ve been waiting on!
- Graham J ⭐️
  
  Here’s a video where they’re using it:
  
  youtube . com/watch?v=-gt1vBJqhco
  
  Quality doesn’t look great there, hopefully it doesn’t do it justice.
  - Jerald Doerr
    
    Ug!!! I can tell if they’re using crapy video cards or the cams in the vive are slow but that looks like crap…
    - Graham J ⭐️
      
      It’s the cameras. Saw some other more raw vids posted on /r/vive and it’s all low res.
    - TonyVT SkarredGhost
      
      Vive Pro’s cameras are not high res at all
      - Jerald Doerr
        
        Interesting to see what the max res is for the cameras. I would have put no less than the resolution of the internal screens myself. Question is the max frame per resolution as I’m guessing that would affect the smoothness of the AR.. Ug 30 or 60 fps might look slow with AR.. Something I just don’t know enuff about.
Stefan Küppers

I wish they would go beyond unity and unreal to open this up for webvr/xr.
- Lucidfeuer
  
  Aren’t Unity and Unreal already compatible with webvr/xr?
  - jj
    
    yeah they are but not their vr aspects and plugins.
dexkz

The real question, will it support inside-out tracking?
- Laurence Nairne
  
  The real answer: no.
  - Johannes Bühler
    
    Are you sure about that? Given it allows for spacial mapping it most certainly could be used to significantly improve playspace setup.
    
    But I know nothing about the latency nor about how persistent that SLAM data is so you’re most probably right that it won’t be possible.
    - dk
      
      technically it can be done and u can have hand tracking ….so no need for the controllers and light houses …but they haven’t mentioned anything about a software doing that ……and besides it’s just meant for devs to experiment with ar it’s not really a great consumer ar device
    - Laurence Nairne
      
      As dk says below, it’d be technically possible, but HTC won’t support it because they’ve already got a robust tracking solution and there’s little of commercial value to give an inferior (in terms of tracking accuracy) solution.
      
      Also, the feed latency seems far too high for use with tracking. I saw someone mention 200ms below!
- Sponge Bob
  
  inside-out is inferior to Valve’s laser based tracking, especially in its second generation
  for pro headset tracking has to be 100%
- Hacker4748
  
  Well, technically the Vive only has inside-out tracking.
RJH

I’m assuming that this should also be coming for Windows Mixed Reality headsets given what the name implies!
- Lucidfeuer
  
  This is indeed what the name means, not just implies…
  - Sponge Bob
    
    you mean you can actually see through those cameras on Windows MR headsets ?
    I thought they were just for inside-out tracking, no ?
    - JJ
      
      spone is right they are comepletely different types of cameras. The windows current cameras cannot give you the visable data that normal cameras provide.
      - Sponge Bob
        
        In other words, Microsoft appropriated (trademarked) the name “mixed reality” without actually providing anything close to it
        
        Me thinks they should be sued and the name “mixed reality” should be taken away from them (unless they can demonstrate one immediately)
        
        And they were complaining about “patent trolls”
        
        how about “trademark trolls” ?
      - JJ
        
        yes i would love someone to go after Microsoft for that. I feel like the entire first part of thier sales were do to misconceptions and beliefs they they purposefully allowed to happen
Lucidfeuer

So finally they did some of their job somewhat. I will consider the Vive Pro depending on what other upcoming headsets are proposing of the sort.
- jj
  
  yeah this really changes things but still needs some time to develop.
Nathanael Mooth

That might almost make the price of the pro worth it.
- Peter Hansen
  
  No. The fact that this device also is an AR dev kit increases the price of the product being a consumer headset. The decisions of HTC don’t get any better with this.
guest

If you’ve used the Leap Motion or Hololens, this is not that. The outward cams have a tiny fov. They are slow and nauseating. They provide none of the hand rec, edge detect, or any recognition middleware.. so you have about 50 dev years of catching up to do on that front. We’ve never seen the spatial mapping do anything but crash the unit.

For commercial applications, we continue to see devs tape better cameras overtop of these cameras because they have no value. HTC has been told this many times and this will confused people on the state of AR which has moved past this. The Pro is great for a lot of things… but this isn’t AR and it can’t be used for playspace set up.
- JJ
  
  this is exactly what the Zed mini is like! Tiny fov, slow, nauseating, no tracking and more bad decisions.
  - Keelan_Stuart
    
    The issue with optical pass-through MR (and in spite of any caveats I’ll mention, it IS the future) is current camera / connectivity / bandwidth limitations. High speed, high resolution cameras require a lot of bandwidth for even a single frame from just one camera… think 2048 * 2048 bytes per frame at 90Hz (that’s 377MB / second!)… so, USB 3.1 data rates are necessary – and camera manufacturers only guarantee performance with specific chipset manufacturers on the PC-side of the connection… and you need multiple controllers, one for each camera. Plus, there are things most people don’t even consider… like the fact that the native pixel format of the sensors isn’t RGB, it’s bayer mosaicked (look it up if you’re curious). Software that relies on the driver to do a bayer-to-RGB conversion will never be fast enough to sync with the 90Hz refresh rate on the headset and will leave you with frame jitter due to mismatched framerates… so, the decoding has to be in a pixel shader. Which leads us to the graphics software itself… PC graphics APIs want the entire texture in memory before presenting the scene to the video display device, meaning you have to wait for a whole frame of video from each camera before rendering – this, opposed to progressively reading and combining pixels as scan lines are captured and output, respectively.
    The point is: once every PC has multi-channel USB 3.1+ connectivity out of the box, this technology can be fully realized and enjoyed by the mainstream… but probably not until then.
    - Keelan_Stuart
      
      Sigh – I meant to say “The issue with **digital** pass-through…”
      - Sponge Bob
        
        So its either digital camera pass through with all the issues you mention
        OR
        transparent AR displays like those used by Meta
        
        for standalone it’s most certainly transparent display technology
        although it wont allow all the tricks afforded by fully digital
        like placing real objects in foreground or background
    - jj
      
      thats very insightful thank you!
    - Redouane Kachach
      
      I totally agree. Pass-thru AR is the future. All the problems you describe are technical (sooner or later will get solved with new, more powerful hardware — in terms of bw, processing, etc).
      
      In addition it has a very good property that Hololens (or similar transparent AR devices) cannot do. Basically It stops the real light from getting to your eyes. What you see is a capture of the reality, processed and presented to you eyes. Thus, you can add/blend new elements to do classical AR, in addition, you can REMOVE things from the reality. In summary, you can modify the reality and represent it in a different fashion. Using AR devices such as Hololens you can’t stop seeing the physical world (light) around you.
      
      This is very important for example to develop realistic AR apps where you want to place items in the world keeping physical properties such as depth/occlusions. Using Zed-mini spatial mapping for example you can put a 3D object in the world and if it happens to be in front or behind a physical object you will see these relations in the generated AR. However, you can’t do that using AR glasses such as hololens or similar because you can’t stop the physical light emitted by the object.
Jerald Doerr

Saw this coming a long time ago… I see they basically took the safe route with the vive.. Seems like they could combined camera tracking with the lighthouses tracking to improve tracking even more. Guess will have to wait for that but I’m disappointed to hear the cams are basically crap.
- Sponge Bob
  
  lighthouse tracking can’t be improved by adding inferior inside-out tracking
  - Jerald Doerr
    
    Only because you say so??? I’m 100% sure you can combine deferent methods of tracking to assist when one fails. They already do this in defern’t existing applications.
    - Sponge Bob
      
      adding additional base stations for Valve tracking (Generation 2) will give much better results than adding inferior inside-out tracking in case of possible occlusions or for extending the range
      - Jerald Doerr
        
        Will Gen 2 let you do that without slowing down the rate like people are suggesting gen 1 would!?
      - Sponge Bob
        
        the IR flash is removed from Gen2 so I assume if multiple base stations can be synchronized to each other (e.g. via sync cables) you can have many of them in the same area without slowing down update rate
        it’s a cumbersome installation indeed but much superior to inside-out tracking (which relies on environment too – try it in a big hall with white ceiling and walls – it simply will not work)
Cat of Many Faces

looks promising!
sfmike

Love the special scan!
Peter Hansen

So basically, the Vive Pro is not a finished product, but a developer kit. That is why it is neither targeted at consumers nor the business sector. Now it makes sense.
TonyVT SkarredGhost

Amazing
dk

https://www.youtube.com/watch?v=UEPX2C1n5Js