Facebook announced today that an upcoming update to the Quest development SDK will include experimental support for a Passthrough API which will allow Unity developers to build AR experiences and features into apps on Quest 2.

Although Quest wasn’t launched as an AR headset initially, its impressive passthrough camera capability showed that it could handle AR functions surprisingly well. In fact, if you use the passthrough view as your default background, the Quest boundary and main menu floats against the backdrop of your playspace—effectively making it an AR experience that’s been available on the headset for some time.

Until now Facebook kept the tools for building AR apps on the headset to itself, but today it announced it’s unlocking the same capabilities for third-party developers too.

Initially rolling out as an “experimental” feature, the so called ‘Passthrough API’ will become available in the v31 update to the Quest development SDK. The Passthrough API is only available on Quest 2 for now; it isn’t clear if Facebook plans to extent it to the original Quest.

Facebook expects the new capabilities will enable a brand new class of applications on Quest 2 that take advantage of AR in interesting ways. Here’s some examples and explanations of the features developers can use:

Composition: You can composite Passthrough layers with other VR layers via existing blending techniques like hole punching and alpha blending.

Styling: You’ll be able to apply styles and tint to layers from a predefined list, including applying a color overlay to the feed, rendering edges, customizing opacity, and posterizing.

Custom Geometry: You can render Passthrough images to a custom mesh instead of relying on the default style mesh—for example, to project Passthrough on a planar surface.

Although transparent AR headsets—like HoloLens and Magic Leap—give a much clearer (and color) view of the real-world, passthrough AR headsets like Varjo XR-3 (and now Quest 2), tend to offer a much more immersive field-of-view and more convincing virtual imagery thanks to complete opacity control and the potential for perfect latency between  real world imagery and virtual imagery. Though Quest 2’s AR view is still fairly low resolution and black & white, it’s expected that future headsets from Facebook will focus on improving the passthrough AR view.

While developers will be able to start experimenting with AR on Quest 2 with the v31 SDK via Unity, support for other engines—and the ability to actually publish AR apps to users—is expected later this year.

SEE ALSO
Shiftall Opens Pre-orders for 'MeganeX superlight' Ultra High-Resolution OLED PC VR Headset

With a device like Quest 2 loaded full of cameras, it’s definitely worth thinking about privacy, especially now that third-party apps can make use of AR capabilities. On that front, Facebook says that “apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors,” and that “raw images from device sensors are processed on-device,” meaning that the images the camera sees don’t get sent to Facebook or to third-party developers.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben Lang
Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Holdup

    Give us color, black n white AR sucks

    • xyzs

      If you code a program that transforms black and white camera input into a colored output, apply at Facebook then.

      • Arnaud Delmotte

        it already exists and it’s a well researched field in AI :
        https://scholar.google.com/scholar?q=infrared+image+colorization

        The color are chosen by the network based on what it is most likely to be based on training data.
        Not sure if the quality and speed is good enough to be used instead of grayscale.

        • Christian Schildwaechter

          I skimmed some of the papers. Even though it doesn’t work perfectly, the results are very impressive. It is probably still easier/cheaper to integrate one wide-angle RGB camera to grab the additional color signal for the pass-through. The XR2 can handle up to seven concurrent cameras, so adding a fifth to a future Quest shouldn’t be an issue.

          But deriving color information from a near-infrared image is pretty cool and another glimpse into how important ML driven image enhancement could become to drive visual quality in XR.

          • Arnaud Delmotte

            Yes, adding one color camera is definitely better. But for the current hardware, this may be a solution.
            This will of course not work in many cases : if an object has not been seen during training, the network can not magically guess the correct color. Same problem if an object exists in multiple colors.
            Another problem that may occur is the temporal stability of the solution. Chosen colors may change from frame to frame if there is no mechanism to prevent it in the network (and it’s generally not easy to do).

    • BCC Dude

      Speaking of which I always wondered why passthrough has to be black and white?

      Is this just a software limitation that can be altered, or that the tracking cameras really can only see black and white?

      • The cameras are b/w IR ones, so they can only provide grayscale images

        • BCC Dude

          I see.
          So are all IR cameras b/w, or Qst2 just happens to use b/w to save cost or something?

          Also is it really possible like what xyzs suggested, to colour the b/w passthrough via some sort of post processing algorithm (whether by Oculus or 3rd party)?

          • Blaexe

            Future headsets will simply have dedicated cameras for passthrough.

          • Christian Schildwaechter

            Infrared is a frequency of light, just as red, green and blue with wavelengths from 0.4 to 0.7 micrometers, so there is no “color infrared”. Regular cameras have to have three types of sensors in a grid for each of the three red, green and blue frequencies, and this only works because our eyes also react to these frequencies and the brain interpolates all other colors from overlapping reactions of these sensors. There are additional receptors in the eye for brightness, which are more sensitive than the RGB receptors, which is why everything looks gray in low light conditions.

            You can build a camera that detects near-infrared just as well as RGB, and in fact many webcams have extra filters to remove near-infrared, as it distorts the image. The Quest cameras are actually recording near-infrared (0.7µm – 1.3µm), you need a thermal camera to record infrared (1.3µm – 15µm).

            The image that you see in the Quest 2 pass-through isn’t actually the near-infrared image recorded, simply because your eyes cannot see infrared, only RGB/brightness. Instead the Quest takes the recorded nIR image and “translates” it to light visible to the human eye, which is done by creating a gray scale RGB image. If you were looking only at the actual recorded near-infrared image, all you would see is black.

            xyzs’ answer was a sarcastic reaction to someone asking for RGB output from nIR input, which is technically impossible.

          • Have you experimented with index cameras?

            Been pleasantly surprised with HTC hand tracking software running on my Index, whilst the different camera vision modes (2D and 3D) are great fun in a well lit room.

          • Cost has a lot to do with it, yes.

        • People have yet to grasp the tech that’s involved
          in a $299 virtual reality device is an absolute steal.

      • Holdup

        Probably the camera oof

      • Adrian Meredith

        This is so it can see the tracking led s on the controller’s and because they are way more sensitive in low light (just like your eyes).

      • Ad

        The index passthrough is color.

        • TimmyP

          The index is a real headset.

          • Andrew Jakobs

            Just like the Quest 2 is, and if you claim it isn’t, well then the Index also isn’t a real headset.

          • TimmyP

            The Quest 2 is a single LCD, phone style VR headset evolution. It is gated heavily from PCVR (where 99.99999r percent of the good stuff is) by ONLY streaming compressed video wirelessly, or with the link cable. Native SteamVR interfacing, which is possible with USBC, was conveniently cancelled.

            The FoV is box style, with a severely limited vertical FoV which takes away from the subversion of reality that these things are so good at. There are sets like the Index and the Odyssey+ that have “infinite” vertical FoV directly down the center of the lenses, because the lenses themselves cover more than your eyesight does vertically. Do you know how much better this looks? *The horizontal fov on both aforementioned sets is also higher, or at least equal to the FBQ2 (IPD hardware dials change FoV slightly, for better or worse, due to the much higher range they have vs software IPD).

            There is the obvious Facebook requirement. The ToS all their users blindly signed allows Facebook access to all camera and mic data. This includes everything that nice big IR camera sees and scans. Go ahead and reread that ToS if you think Im kidding. Its only gonna get worse in 2022 as well, with regard to account restrictions.

            I understand you might be new to VR, but as an experienced vet who has tried every headset under the sun (virtually harhar). Native, tethered exclusive PCVR headsets look FAR superior, even at lower resolutions with regard to PCVR; 99.999999r percent of everything that matters.

          • Ad

            Valve has done fuck all with it and it really seems like facebook offers more access (when they offer any).

          • TimmyP

            You have access to PCVR with all sets. They all have the “same” access. You can play most of the Oculus “exclusives” (VR tech setbacks) with ReVive.

          • Ad

            Access to turning on the cameras, and camera data.

      • benz145

        Pasting from another comment:

        As far as I understand, going to black and white (IR) means being able to get higher resolution out of the same sensor, and helps with low-light sensitivity. Other headsets have used color cameras for inside-out tracking.

    • Elite-Force_Cinema

      Then what do you want black and white AR passthrough to do in order for them to not suck, you clown?

      • Holdup

        If it’s the camera limitation then that’s unfortunate and we’ll just have to wait till future headsets but if it’s just a software fix that Facebook is holding back then that’s something they need to fix. Once you try color AR it will blow your mind, for me black n white is like meh

        • TimmyP

          Its not a camera limitation. They have to be black and white to determine contrast for tracking the controllers.

          *called inside-out tracking

          • benz145

            It doesn’t “have to be.” As far as I understand, going to black and white (IR) means being able to get higher resolution out of the same sensor, and helps with low-light sensitivity. Other headsets have used color cameras for inside-out tracking.

          • Jonathan Winters III

            Ben, would be nice if you could moderate these posts. People arguing and calling eachother names is a waste of out time in reading the comments.

        • AR is AR: monochrome OR color, it STINKS.

        • Elite-Force_Cinema

          I said what do you want black and white AR passthrough to do in order for them to not suck? Do you want them to die as a whole for the rest of your life or what?

          • Holdup

            When I picture AR I picture a projection of the digital world on top of the real world, the real world isn’t black n white.

          • Elite-Force_Cinema

            Answer my question as to what you want the black and white AR passthrough to do in order for them to not suck you clown! Your response is garbage and should not exist!! FOREVER!!!!!

          • Holdup

            You’re the clown, for it not to suck they need to make it color true software, or if it’s a hardware limitation they should work towards making the next gen headsets have color AR, if not then they should change the title from AR, don’t call it AR, this isn’t AR to me. Another thing they need for true AR is environmental recognition, make the digital objects recognize objects in the real world, like be able to place a digital screen on the real wall, or have your digital character recognize a chair and walk around it instead of disappearing or just walking through it like it doesn’t exist.

          • Holdup

            I can say low quality bootleg movies suck too, your emotions doesn’t change the fact that it does.

          • Holdup

            And if it’s a hardware limitation that can’t ever be fix in the future then black n white passthrough AR just sucks and it’s not the way to go, it’s neat but it sucks.

  • Miguel

    They’re doing this, because they know apple is about to rease their vr kraken

  • I can’t wait to experiment with it!

  • Adil H

    I would like to play Eleven Table Tennis with a passthrough background from cam.

    • Holdup

      Chess club would be neat, or poker star if we could get color passthrough working

  • TimmyP

    You have to be out of your *** **** mind to allow FACEBOOK to have this kind of access to your home. Holy cow, please get ANY other headset. Read that ToS you signed, they are gonna begin scanning your own home.

    • Ooh look, everyone! A “Facebook is Big Brother” postorz !!11!
      How Spring 2021 …. Find a new bandwagon to jump on, champ.

      • TimmyP

        Yeah okay shill.

        Let fisheye IR cameras that sees more than your eyes do, that is produced by FACEBOOK, into your home.

        Ignore the ToS, obviously you would given your avatar. Its not like it directly says they can access camera and mic data. /s

        And then there is the below average performance of the device itself, specifically regarding PCVR. If you do some homework, like you tried to do judging from your avatar, you can find BETTER Oled headsets for cheaper. The only thing you have to not mind is a tether. Literally everything else sans tracking that matters is superior on real HMDs.

        *Who do you think you are? Ive never seen someone look like such a douchbag solely from text formatting and the way you try and overpower my words. Good luck with that!

  • Ad

    I hope Valve has AR functionality in their next hardware. There does seem to be a lot of potential. I also hope they actually make it usable for devs and don’t drop the ball like they did with the promised code for ML on the index cameras.

  • A caveat: from this other post I understand that this will be supported only on Unity 2020+ as part of the new OpenXR Plugin. Older Unity versions will not have access to the passthrough API as they’ll keep using the old OVRPlugin. But it’s not very clear, so not completely sure

  • sfmike

    I’m sure it will be overpriced.

  • Deborah Crum

    I am able to receive 85 dollars/h to work part time on the internet…~or109~I certainly not realized like it would achievable however my confidant friend has produced $25,000 only within three weeks just completing this super work & also she had convinced me to join…~or109~Know better updates on visiting following web page >>> https://plu.sh/financial

  • starchaser28

    The B/W passthrough actually makes sense, it will help viewers more easily distinguish the virtual /digital content from real by adding that contrast.

  • Lulu Vi Britannia

    Finally, the real first step to Augmented Reality is here. The article is spot on when it’s about the FOV: AR through VR headset will be more convincing than small angle AR glasses… once the Passthrough gets better. Of course, for now, it sucks. But it’s a first version.

    AR is very interesting, but AR displays still don’t really exist. AR through VR is an interesting compromise for the time being. I can’t wait for devs to show off what AR can do! Especially if we can “record” virtual objects position in a Guardian. For example, you could attach a TV screen on one of your walls. If it can be recorded, that TV screen would stay here as long as you keep that Guardian on.

  • oomph2

    I thought it was already there

    • Nancy Chambers

      I’m receiving $85 every hour to complete easy tasks online…~nr1130~I have not at all believed like it’d even possible however one of my confidant friend making $26,000 only within three weeks by working this top offer furthermore she has satisfied me to try…~nr1130~Read better details through this web-page >>> https://link.gy/Money10
      .

  • Binja Fuller

    I’m glad to hear that Facebook won’t use my cameras and other imaging hardware to generate indoor maps and a comprehensive guide to all my belongings, habits, and preferences. It’s nice of them to throw away the trillions of dollars that info would provide. Their generosity knows no bounds.