Initially announced in 2021, Meta CEO Mark Zuckerberg shared an update on the company’s research partnership with BMW, which focuses on integrating AR and VR into vehicles to make people more productive, social, and entertained while traveling.

The ultimate aim in the BMW/Meta partnership is to accurately anchor virtual objects relative to the car’s motion by hooking into the tracking system of both the car and a Meta headset, which researchers say includes the Meta Quest Pro standalone mixed reality headset and the company’s in-development AR headset, Project Aria.

Without such a system in place, the headset’s rotational tracking would noticeably drift as the car makes turns and other adjustments, making it essentially unusable for anything but perfectly straight sections of road.

Check out the video detailing the research below:

Still considered a proof-of-concept prototype, Meta says the partnership has already overcome some key technical challenges, such as fusing the headset and car’s sensors to understand their relative position. That said, the companies don’t think it’s ready for the public just yet.

“It is too early to tell exactly how or when this technology will make it into customers’ hands, but we envision a number of potential use cases for XR devices in vehicles—from assisting the driver in locating their car in a crowded parking lot to alerting them to hazards on the road and surfacing important information about the vehicle’s condition,” said Claus Dorrer, Head of BMW Group Technology Office in the US. “The implications of future AR glasses and VR devices—for passengers as well as drivers—are promising. The research partnership with Meta will allow us to discover what immersive, in-vehicle XR experiences could look like in the future and spearhead the seamless integration of such devices into cars.”

SEE ALSO
Orion & Quest 3S Signal a New Era for Meta, Here's What it Means for the Industry at Large

AR and VR integration in cars isn’t an entirely new area of research. It’s been the sole focus of Audi-backed startup Holoride, which recently partnered with HTC to deliver in-car VR entertainment via HTC Vive Flow. Still, Holoride has been mostly grabbed headlines as a tradeshow mainstay; it hasn’t seen mass adoption yet despite only requiring a $200 retrofit pack, which enables Vive Flow owners to play VR in cars.

In the end, it seems car companies are now seeing the writing on the wall that riders will maybe very soon—but not right now—want to bring their own XR devices and actually use them in the car, just like you might a smartphone, albeit with more utility than any infotainment screen on offer.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Scott Hayden
Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 4,000 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • Christian Schildwaechter

    TL;DR: WTF Meta? At least pretend that improving actual user needs is somewhere on your priority list.

    In his 2021 Connect keynote, John Carmack criticized that Meta was focusing too much on abstract concepts and not enough on actual, achievable progress. He repeated his opinion that “the metaverse is a honeypot trap for architecture astronauts“, basically a playground for people who prefer to look at things from the highest level, and avoid solving the million of small problems on the way towards this goal. Obviously he didn’t agree with that approach, instead suggesting that Meta should focus on improving the many usability problems of their headsets and software first, and thereby slowly approach a level of usefulness that one day may turn it into the metaverse. Or, in his words: “And I just want to tear my hair out at that, because that’s so not actually the things that are actually important when you are building something.”

    By now Carmack has finally left Meta, and looking at recent Meta projects doesn’t give much hope about them prioritizing actual user needs. We recently had:

    – The metaverse, of course.
    – VR conferencing in Horizon Workrooms, which was supposed to be Meta’s way to break out of the gaming niche and into the VR business world, alongside great concepts like using cloud-based Office 365 inside a low PPD headset. Carmack mentioned in the same keynote that “sometimes” Horizon Workrooms is actually better than Zoom calls, which alongside Meta employees actively avoiding to use Workrooms doesn’t exactly inspires trust.
    – A Quest headset with Pro moniker who’s main feature seemed to be eye and face tracking for the VR conferencing everybody is supposed to spend lots of time in.
    – Mixed Reality, really just VR with passthrough. Useful, but so far Meta has failed to demonstrate why this is so impressive that it deserves its own name or being a main focus on the Quest 3, esp. since it doesn’t do any of the useful things an AR HMD could do. And they basically said that proving that the tech is as great and essential as Meta claims is actually the job of users and developers.
    – They recently removed the option for user to create their own events in Horizon Worlds, stating that the function wasn’t up to their quality standards, but failed to replace it with anything that would provide this functionality essential for social spaces. This is their nth attempt to create a flourishing VR social community, and every time users weren’t excited enough, they just scrapped the current iteration, released a new one, often removing the very few functions people actually liked about them.

    And now we get VR/AR usable in cars for entertainment. No doubt the best place for VR experiences, where you are not only seated, but also wearing a seat belt for maximum safety and minimum room scale. Plus a double dose of motion sickness even in virtual cinema, when in a reverse to the usual causes you sit still in VR, but your body physically turns around. The video works around that by only mentioning AR and integrating the car movement, but Meta doesn’t actually offer any AR HMDs yet, and convincing the brain that the cinema with you in it just moved around the corner seems challenging. Maybe this is the magic that Mixed Reality will bring: to reduce the chance of someone throwing up in your car while wearing a headset for reasons.

    Now if there was a poll what VR users want the most, there would be the usual suspects like AAA titles, larger FoV, better graphics, less weight and improved ergonomics and more. I am pretty sure that none of the great Meta concepts listed above would have made the top 10 or anywhere close. And most people wouldn’t even have considered VR usage in moving vehicles a sane idea.

    They recently said they realized that they needed to become more efficient and focus on making the actual experience better for users, and what we get is 10,000 people fired and VR/AR in cars. While I appreciate forward thinking research and innovation, this mostly triggers a “WTF Meta?”

    • Jonathan Winters III

      Very agreed, except your point on mixed reality, which will have many jaw dropping titles/uses on the Quest 3 onwards.