Vision Pro doesn’t have controllers, but it’s clear Apple will need to support immersive hardware input somehow in the future, if only for the greater precision it might bring to industry professionals. Sony has a pretty interesting take on how to serve up ‘pro’ style controllers in its recently revealed enterprise-focused standalone, which could point the way forward for Vision Pro.

Vision Pro is a mixed reality headset with a bit of an identity crisis. While it can technically play fully-immersive VR content, without motion controllers many of the VR games you’ll find on Quest simply won’t make the jump to the $3,500 device, which is instead being pitched as a productivity machine capable of running the standard fleet of iOS apps in addition to visionOS-native apps.

Ok, maybe not an “identity crisis” as such, but Apple made some clear marketing decisions with Vision Pro by omitting controllers, instead focusing on eye and hand-tracking input. Granted, that’s not to say we’ll never see motion controllers on Vision Pro, but probably not the all-purpose Touch-style controllers many manufacturers have converged upon following Meta’s dominance in VR. At least not in the near future.

At CES 2024 in January, Sony unveiled its Snapdragon-based standalone MR headset (still unnamed) which promises a ton of enterprise-friendly features, like “4K” OLED microdisplays, passthrough video, a flip-up visor, and a unique pair of controllers that aim to appeal to the sort of professionals Apple undoubtedly wants to capture at some point too.

When it ships sometime “later in 2024,” Sony is including both a pointer-like controller that’s operated with the index finger, and a small ring-like device that’s controlled with the thumb. Yes, they’re funky, but they have some pretty specific use-cases in mind.

The ring lets you select items and anchor them firmly to your hand, while the so-called ‘pointing controller’ enables what Sony calls “stable and accurate pointing in virtual spaces, with optimized shape and button layouts for efficient and precise operation.” It’s basically a 3D stylus Sony hopes will appeal to creators building and viewing 3D objects in the headset itself, but in a natural and precise way, making for an ostensibly pared down version of Wacom’s VR Pen, which was announced back in 2020.

SEE ALSO
Snapchat Reveals Latest Spectacles AR Glasses with Attractive Pricing for Developers

Wait. Why are controllers so important in the first place? Vision Pro’s hand-tracking is some of the best—and will likely evolve over the headset’s life cycle—but all of the same caveats apply to Vision Pro’s optical hand-tracking as they do in any modern XR headset. That’s to say, there is only so much Apple (or anyone) can do without controllers.

Hand-tracking doesn’t provide haptic feedback, which isn’t great for establishing physical relationships to digital objects. Most of all though, using an empty hand to ‘pinch and drag’ doesn’t provide the sort of precision grip you can get from tracked controllers, which necessarily include buttons that don’t need to ‘guess’ when you’re using them, since buttons have precise activation thresholds that you can intuitively learn to manipulate and anticipate. The button ‘knows’ when you’re just resting your finger on it, when you’re depressing it half-way, and when it’s all the way down—all of which can be integrated into a more rich and accurate way of spatially computing beyond just browsing through UI menus and doing some lighter gaming stuff.

Image courtesy Sony

Not to mention, going for something similar to Sony’s ring/pointer combo would allow Apple to retain that pro-level marketing bend, and also offer a sleek (and very likely expensive) accessory we all know Apple loves to sell—but that’s besides the point.

It’s not like any of this is a big surprise to Apple though, or the developers making visionOS-native apps that use of the headset’s 3D spatial abilities. For now, Apple wants Vision Pro to be more ‘face-computer’ than gaming console, but also more ‘bottom-tier iPad’ than Mac Pro, at least in terms of hardcore industry apps it can support. There’s plenty of room to grow however, as right now Vision Pro has a few apps for viewing 3D assets in immersive space, like Polycam and the enterprise-focused app JigSpace, and casual sculpting and 2D art apps—but still none of the industry standard tools like Blender or Autodesk, both of which actually support VR headsets.

Still, Apple is very much a ‘wait and see’ sort of company, so its competitors may need to collectively be “the first guy through the wall” (who always gets bloody) before the Fruit Company follows suit. Given a second iteration of the headset has been rumored to arrive sometime in the next two years, Apple may be waiting to bump that particular feature set later rather than sooner, if only to have that really useful One More Thing™ to announce next time Tim Cook takes the stage.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Scott Hayden
Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 4,000 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • psuedonymous

    As with the ‘Apple pencil’ for the iPad – where Apple will maintain that stylus inputs for tablets are doubleplusungood and nobody wants them and they’re stupid, right until they announce they have invented the amazing new concept of a stylus input for tablets – Apple will claim controllers for VR devices are worthless right up until they announce the invention of amazing and revolutionary new idea of a controller for VR devices.

  • wheeler

    Having demoed the Vision Pro, it convinced me–an otherwise VR Gamer–that full-on motion controllers are honestly just a bad fit for non-gaming/-immersive applications. This is not to say that there shouldn’t be physical input devices for non-gaming VR stuff but rather that they shouldn’t be typical gaming motion controllers. So I can entirely see why Apple isn’t using them. They add unnecessary friction, they’re “in way way” when you need to do other things with your hands (for example, use a keyboard or engage with things outside of VR), and in most cases they offer little value (again, to these other non-gaming contexts).

    Now obviously this isn’t true for art/design (which Apple hasn’t targeted–yet), but still it would seem like there are better input/output devices for this context, specifically ones that are not as cumbersome as full on motion controllers. So yes, I entirely agree that Sony’s concepts are the general path forward.

    With the ring, you get physical inputs/feedback while largely maintaining the freedom of handtracking / avoiding the burden of motion controllers. And then obviously there’s the 6DOF tracked pencil which is much more natural and better suited to design/art than a full on motion controller (and certainly better than the hack of using the butt of the motion controller).

    I’m sure there will be those that claim these devices are *technically* motion controllers as well, but their use-cases are entirely different. I wouldn’t want to use a 6DOF tracked pencil for VR games any more than I’d want to use yet another boring oculus touch clone for productivity applications. There is a major distinction and little overlap with the VR gaming devices.

    It’ll be interesting to see where all of this goes. Up until now, VR has been decidedly focused on gaming applications. And despite many billions having been burned away, it is stubbornly low retention and sees infrequent use. AAA VR gaming titles (like AC Nexus, where everything has gone right) still see lackluster engagement and sales. Social VR was perhaps the most interesting alternative use-case because despite being niche the demand for it is quite natural and proportioned. Many like to celebrate how far the VR gaming medium has come, but given the resources involved it’s been like using an aircraft carrier to tow a tugboat (and after 8 years that tugboat is still entirely dependent on the aircraft carrier to maintain its speed and scope)

  • XRC

    Sony are on the right track with their new system, the hand input control is very impressive.

  • Christian Schildwaechter

    In January “Patently Apple” reported a patent for an motion tracking Apple Pencil for AVP, showing several concepts, the simplest being a pencil shaped controller emitting light tracked by the HMD. Much of the article is about self-tracking with cameras similar to Quest Pro controllers, but with a number of special uses for these cameras. The pencil controller serves as an extension of the field of view of AVP, letting the user look behind or inside of objects otherwise hidden, allowing to e.g. virtually airbrush a real statue while seeing the color on the side and back shine through it in AVP, or record the insides of a machine. The pencil would serve as both a controller and a 3D scanner.

    Apple releasing a pencil for AVP makes sense, with a pencil also available for iPad and used for lots of art apps. The iPad pencils already offer limited tracking, both the (near) contact of the tip and the rotation of the pen in space is measured. Combined with the thin glass glued onto the digitizer this allows to draw exactly at the tip, while most generic pens only work precisely when pointing straight down, otherwise users need to offset a positional shift while holding the pen at an angle. Input isn’t limited to buttons, a touch surface would allow fine grained analog input for e.g. sub-millimeter finger control on a dual-action airbrush. They mention extra ultrasonic/electromagnetic sensors that might be placed inside the pencil controller, though no particular application is named.

  • Ender772

    and mr stands for what?