A new photo of Magic Leap 2 appears to show the device’s controller equipped with cameras for inside-out tracking which would be the first time we’ve seen the approach employed in a commercial XR headset.
Though we learned plenty of interesting details about the forthcoming Magic Leap 2 AR headset back in January, it looks like there’s still some secrets left to uncover.
A recent photo of Magic Leap 2 posted by Peter H. Diamandis is, as far as we know, the first time we’ve gotten a clear look at the front of the Magic Leap 2 controller. The photo clearly shows what appear to be two camera sensors on the controller, indicating a high likelihood it will have on-board inside-out tracking.
The original Magic Leap 1 also had a motion controller, but it used magnetic tracking. This was the reason for the curious square sticking out of the headset’s right side (it contained a receiver that sensed the magnetic field emitted from the controller).
Magic Leap 2 has ditched the square and appears to be moving to entirely on-board inside-out tracking for its controller. To our knowledge this will be the first time a commercial XR headset makes use of the approach; that is, assuming Magic Leap 2 beats Meta’s Project Cambria to market (the latter is also expected to use on-board inside-out tracking for its controllers, based on some leaked details, though it hasn’t been confirmed yet).
Other standalone XR headsets, like Quest 2, typically use headset-based inside-out tracking to track their controllers (or the user’s hands). That is: cameras on the headset look for the controllers (typically arrayed with IR LEDs) and use their location to map their movement relative to the headset.
While this approach has proven effective, it only works when the controllers are in view of the headset’s cameras. That means it’s possible for the headset to lose track of the controllers if they spend too much time outside of the camera’s view (like if you held them too low, too high, or behind your back). Putting cameras on the controllers themselves would enable inside-out tracking that, in theory, has coverage no matter where the user holds it.
Beyond ‘anywhere’ tracking coverage, putting inside-out tracking directly on the controller means the headset wouldn’t need to be covered in so many cameras (Quest 2 uses four cameras while Rift S tops that at five), and it means the elimination of the ‘tracking rings’ (seen on most standalone headset controllers) making them sleeker and perhaps less prone to breakage.
There’s more potential benefits too. Giving controllers their own inside-out tracking could make it easier to use a combination of input methods, like a controller in one hand and controllerless hand-tracking in the other. Currently that’s a challenge because the cameras on the headsets tend to use different exposure settings for tracking controllers compared to tracking hands. Furthermore, decoupling controller tracking from the headset has the potential to reduce controller tracking latency.
But, as they say, there is no free lunch. Giving controllers on-board inside out tracking likely means putting a dedicated processor inside with enough power to crunch the incoming images and compute a position, which we’d expect to result in higher costs and more battery drain compared to having simple IR LEDs on-board.
There’s also the issue of motion blur for the on-board cameras. While inside-out tracking has been shown to work well for headsets, controllers can move considerably faster, especially during intense games—so fast, in fact, that it prompted Valve to update its SteamVR Tracking tech to account for the speed of top Beat Saber players.
As you can imagine, swinging a camera around that fast is likely to induce motion blur across the image sensor, especially in darker scenes where the exposure time needs to be increased to gather more light.
Of course, Magic Leap 2 isn’t intended for gamers, so it’s possible that this wouldn’t be an issue for the headset’s enterprise-focused use-cases. Or maybe the image sensors on the controllers are fast enough to avoid motion blur even during quick movements?
Perhaps most likely the sensors aren’t above average, but the controller will simply lean primarily on IMU-based tracking until the controller slows down enough to get a proper position (this is the same approach that existing standalone headsets use to deal with controllers that occasionally leave the headset’s field-of-view).
In any case, we’ll know more as Magic Leap 2 approaches commercial availability. The headset is expected to launch this year but a specific release date hasn’t been announced yet. In the tweet which included the controller photo, Peter H. Diamandis said he would be hosting Magic Leap CEO Peggy Johnson at his Abundance 360 event at the end of April which could signal the next time the company will divulge official details on the headset.
Editor’s note: Before someone in the comments says ‘SteamVR Tracking tracking controllers have been doing inside-out tracking for years!’ the industry vernacular does not consider systems using artificial markers as ‘inside-out tracking’.