Oculus Chief Scientist Dives Deep Into the Near Future of AR & VR

By

-

Oct 16, 2018

Computer Vision

Michael Abrash speaks on stage at Oculus Connect 5. | Image courtesy Oculus

Abrash went on to describe the commonality in computer vision technologies that VR and AR devices will enjoy in the future, as both types of hardware will be capable of rendering convincing mixed reality, where real-world objects and environments are shared with virtual ones. The approaches to this challenge are different; VR headsets of the future will be able to scan and render the real world in the virtual environment in real time, Abrash says (he believes this will be workable within four years), whereas AR glasses ‘overlay’ a rendered layer to the real world seen through a transparent display.

While he thinks that VR will be the “best [form of] mixed reality for a long time,” he also expects the two technologies to eventually share the same underlying interface, and share the same developer environment and tools, with apps and environments working seamlessly across both types of headset.

Virtual Humans

One of these is real and the other is a computer recreation. Can you tell which? | Image courtesy Oculus

Finally, he spoke about convincing virtual humans, and the uncanny valley that, until recently, seemed almost impossible to overcome. Abrash highlighted a machine learning-based approach called Codec Avatars, running real footage of a talking head next to a remarkably accurate reconstruction. While still in its early stages, he believes that this could revolutionise virtual communication and collaboration. “I’m not betting on having convincingly human avatars within four years,” he said, “but I’m no longer betting against it either.”

Researchers Showcase Impressive New Bar for Real-time Digital Human Rendering in VR

“Four years from now, VR is going to jump to the next level,” he surmised. “And that’s just the start. Every area will continue to improve, and virtual humans and likely haptic hands will be along before too long. In short, as far as I can see—and I can see pretty far—the future of VR couldn’t be brighter.”

While Abrash spent his time talking about underlying technologies and capabilities, we recently explored 10 specific projects that have us excited about the future of AR and VR.

Thomas

Man, that’s some awesome stuff!
It’s mean, that you don’t tell which one of the last images is the real photo ;-)
I’d say the left one is real, because the image is a bit sharper and there is a reflection on the tooth, that’s missing in the right image :-D
- Foreign Devil
  
  I’m sure they are both photographs of a real person. . just one has the photo textures overlaid on a 3D mesh.
paratay

Michael Abrash is a tool, ALL his predictions are obvious and he gets a big head about it. I can give you much more accurate prediction and I am prepared to bet him if he is man enough. He is wrong in most
- Bob
  
  “For one thing how can an HMD like the Oculus Quest which is going backwards in Hardware performance be capable of the CG quality he is talking about.”
  
  You truly are clueless aren’t you?
  - paratay
    
    What are you 8, dick head. yeah I clueee lessss. No you are correct, the piece of shit hardware on the Quest is a miracle. Sorry you win
    - saintkamus
      
      Don’t forget to breathe.
    - BananaBreadBoy
      
      Abrash’s talk was on *PCVR*, moron. Not Mobile headsets like Quest.
      - Kev
        
        Stuff that will not release for years, if ever. They have done nothing in the PC VR arena except create walled gardens within a budding industry that certainly doesn’t need massive infighting. It needs innovation (which they have done nothing) and hardware competition (also nothing). Best they can do for PC is release new versions of their software over and over and over.
    - Andrew Jakobs
      
      Maybe you should read what Abrash is talking about.. he’s not talking about the Quest..
    - IsaiahGilliland
      
      Better to remain silent and thought a fool than open your mouth and remove all doubt.
  - Engineer_92
    
    Not clueless, just an outright idiot
- saintkamus
  
  You’re dumb.
- Xron
  
  Well, maybe he is, maybe he is not, but for the most part, people care about what oculus can offer for them as 2nd gen pc vr (atleast me) and if they can offer something similar to Abrash’s predictions by 2020 summer, I would be really happy to buy it :)
  Commenters below, we’re all interested in Vr, don’t let us go down to flame war with each other like in some reddit forums.
- MosBen
  
  The Quest won’t be as powerful as a 2080ti, but then, it doesn’t need to be. Future generations of the Quest will undoubtedly have more robust hardware, but it won’t ever match what a GPU in a big PC case is capable of doing. But again, it doesn’t need to be.
- IsaiahGilliland
  
  He’s not talking to you idiot, he’s talking to the investors in the audience who barely know how to use email.
- impurekind
  
  You know what would really help: You paying more attention to what is actually being said and shown and the context it was said and shown in. Just saying.
- FloridaOJ
  
  Weak bait 2/10. The people who seriously responded to you.. they’re new to the internet and don’t count. Try harder next time.
Maciej Soinski

The future is bright, but the deeper we go into development of this technology we are more aware that VR/AR needs not years but decades to mature. For consumers these are not good old days.
- impurekind
  
  Most of the essentials for a genuinely great VR experience with little compromise will be nailed long before “decades”. It will just be real hardcore stuff that most consumers could essentially live without, like having full body tracking with haptic feedback and that kind of thing, that will take a lot longer. But I’d guess that withing a generation or two we’ll have affordable VR headsets with resolutions that are high enough that any further improvements will be more welcome than essential, a field of view that is pretty on-spec with the actual human field of view and therefor doesn’t need to be any higher, a refresh rate that is also high enough that anything beyond will similarly be welcome but not essential, fully 3D sound that is so good that anything else beyond it is barely even perceptible at that point, and so on. So, yeah, the [near] future of VR is looking very bright indeed.
Andrew Jakobs

Problem with AR headsets is that they tend to be a screen in front of your vison of the real world (and that’s a good thing ofcourse), but VR IMHO requires a headset that will block all outside light/vision. So the demand of both headsets are quite different.

And as always, the biggest problem at the moment isn’t really the headset itself, but the GPU’s, they just aren’t powerfull enough for mainstream adoption with AAA expected visuals. Even the GTX1060/6GB is still 250+ euro, and that’s about the lowend GPU you need for even a bit of VR with the current Rift/Vive, for higher resolutions you’ll need much better GPU’s, even the 2080ti isn’t capable of driving a 4k per eye display at 90+ fps with everything on expected visuals. And foveated rendering is nice and all, but as long as decend eye/gaze tracking isn’t available, it’s still a long way off..
For the real enthousiast with enough money or Businesses it might be affordable to go for the highend VR, but for VR to be mainstream affordable, which means less then $400 ALL-IN with visuals we expect these days from AAA games, we’re still at least 4 years away..
- MosBen
  
  I agree that $400 all-in is the sweet spot for mainstream VR, and I think that Oculus agrees with us, hence the Quest. Sure, in ten years the Quest’s hardware will be quaint, but for the purposes of next year, I think that it’s worth non-VR nerds getting excited about.
- impurekind
  
  Well this is kinda the entire point of his presentation: He’s predicting where he thinks this stuff will be in the [near] future. And, literally, basically all his predictions were for about four years in the future.
  - daveinpublic
    
    Glad to know there are companies out there that are still investing everything they can into it.
Kev

Oculus sure talks alot about things they can’t even demo. I bet they are the last to market with every one of the innovations he discussed.
- BananaBreadBoy
  
  Nobody cares about what comes first, only what does it right. No one cared about the IBM Simon or LG Prada even though they came before the iPhone ¯_(ツ)_/¯
  - Kev
    
    They’ve been actually going backwards. Instead of showing evolving VR instead they went for battery operated smartphone devices with even lower specifications. Next year we can look forward to a three year old snapdragon 835 device from Oculus!! It will have the same 6DOF capability that the DK1 had in 2013! Sweet.
- impurekind
  
  The stuff they talk about is often the same stuff they demo the next Connect and then tend to release the next one or two after that. That’s not too shabby at all considering they are both predicting and then then actually building the future of VR. And, on multiple occasions they’ve been the first to market with VR headsets that are beyond the rest of the competition. The Quest being a perfect example of this, being the most powerful fully wireless/untethered VR headset out there that also has full 6DoF on both the headset and controllers. So it’s both inaccurate and unfair to act like Oculus isn’t delivering the goods when it actually is–regularly.
  - Kev
    
    We had 6DOF in the DK1 in 2013. I would also point out the Quest can only track your controllers right in front of you. You cannot swing behind your head or as low as your beltline. It’s far more limited than even the existing touch controllers.
    - impurekind
      
      Not on an all-in-one completely wireless untethered headset.
      
      And even with some limitations, the Quest is still the most advanced all round affordable fully wireless/untethered all-in-one VR solution on the market.
      
      You honestly seem to be missing the point here.
impurekind

I always enjoy watching Michael Abrash talk about VR.
jimbo5Z

Abrash, as always, impressive and reassuring.
Lucidfeuer

Can’t wait to see the Oculus Quest’s numbers…though the kool-aid will never run out, this cycle of VR product and interest might. Well at least we’re still getting lots of branding VR projects…
oompah

Waveguide goggles is what will sustain momentum of
virtual reality , or else it may perish being too uncomfortable.
Waveguides the magical futuristic tech. that takes away that
box off ur face & will give u natural feeling of being in that computed world.

But I wonder , imagine it becomes so refined that govts. start to
embed it into ur retina at birth & show u what the govt wants u to see
or believe
Kenji Fujimori

Oculus is getting crappier.. John Carmack on his tweets even hinted at soon leaving, so that is a sign of whats going on internal
- Mrfox Babbit
  
  OO! oooo! Maybe he’ll jump over to Valve to begin on the followup to the index.
  - Kenji Fujimori
    
    Actually I jumped into your mother’s hole
    - Mrfox Babbit
      
      Look at you go! You typed that out all by yourself! Good for you.
      The old “Mom” hacks, the mark of a slim-witted child. Keep trying though champ.
      - Kenji Fujimori
        
        Yep keyboard warrior Anglo prick, if we met face to face, you would face a different fate.
      - Mrfox Babbit
        
        Ah yes, your irrefutable manly boast shows the mark of a true intellectual genius, with a mythical c*ck so massive the swagger is off the charts.
        
        Boy oh boy you sure are tough and intimidating there champ, nobody F’s with you… pfft what an abject loser poseur.
        
        Truth of the matter is that undoubtedly you still demand your grilled cheese is browned just right and cut into triangles with crusts removed otherwise a meltdown tantrum ensues.
      - Kenji Fujimori
        
        Yep, your mother liked it that much, she came back for round three of a creampie
      - Mrfox Babbit
        
        Uninspired boring rhetoric of a 12 year old.
      - Kenji Fujimori
        
        Anglo bitch
      - Mrfox Babbit
        
        You might be somebody one day.
      - Kenji Fujimori
        
        You too, once you get off unemployment
  - BananaBreadBoy
    
    Why would he leave for the Index? Carmack’s all about mobile VR, to the point that his joining condition for Oculus was that he be allowed to work on mobile headsets. He’s right where he wants to be at Oculus.
Kenji Fujimori

Same bullshit was also dictated by Magic Leap
Jim from Texas

Why create flat display panels? If Facebook is going to have to develop the technology for next generation display, create a display that is curved in just the right way to fix the issues with optics/lenses.
Nerd Man

Does this mean that we aren’t going to see a new RIFT for 4 to 5 years? (The most important thing that he said is that VR manufacturers are going to have to create the next generation of displays themselves instead of using technology from smartphones. I am not sure that the business case exist for this kind of investment)
Darren

Fixing the multi-focus problem via software to blur the part of the image that weren’t in focus was really interesting. The “Half-Dome” prototype fixed this problem via mechanically moving the display closer or further away. Assuming that “Half-Dome” was a prototype for Rift 2, this seems like a dramatic change in the hardware for the Rift 2.
Kenji Fujimori

Looks Crapulus
Mrfox Babbit

NEAT!
dz11

And then they release the Rift S. Overwhelming.