Quest 3 Brings a Big Change to Controller Tracking Coverage

You might be right. I can rememebr specific situation where you don’t, like monkey bar traversing. But the are not that comon. Might be worth the tradeof.

Of course you had to attack one of the most popular, and highest grossing VR games as Meta not caring about perfect enough tracking for it. It’s Beat Saber dude.

I’m not interested in the issues you are experiencing, especially when you counter with with “but dont with lighthouse”. If I were to say I constantly had issues with lighthouse tracking, you just wouldn’t believe it. Besides, imagine playing mobile game with “high end” lighthouse tracking.

You edited your comment.
Hands behind your back? Yeah, are you scratching your butt? Or play tens of hours a month the only type of game where it’s actually usable – pool games that haven’t seen any updates in years?
People put their controllers in front of eachother all the time – no issues here man.

I have no experience with Quest 3 controllers.
I did use Quest Pro controllers and I liked the build quality, it’s notable step-up, just like the Pro controllers for Xbox or whatever they are called.

I did. And i played them just fine using Quest. It doesn’t magically stops tracking the moment cameras can’t see controllers.

VALVE CUCKHARD

TL;DR: IMUs are more than just gyros and accelerometers, SlimeVR calculates 3D position from skeletal data and rotation data, because our limbs are of a fixed length, which makes this pretty precise, not because accelerator data is only usable for fractions of a second, Meta very likely tested if tracking without a top camera would still work well enough, and an improved IMU tracking on Quest 3 may actually reduce some Quest 2 artifacts. The rest is very long and only for those interested in a lot of technical details about how IMUs and tracking based on them work.

IMU stands for Inertial Measurement Movement, because the main sensors measure acceleration along or around different axis. Simple ones may only be 3DoF with a gyroscope, reporting rotation around XYZ, most of those used in HMDs/controllers are at least 6DoF that also add acceleration along XYZ. So called 9DoF usually add a 3DoF digital compass, and 10DoF a 1DoF gravity sensor pointing down or a pressure based 1DoF altitude sensor, which helps when use in drones.

Data from these sensors is often combined/fused, because using just one can introduce errors, as there is no reference to check the values against. But whenever you rotate the IMU, it will also be accelerated linearly in one direction, if the IMU isn’t sitting exactly at the center of the rotation. So a controller takes both the rotation and translation data and combines/compares them to improve data quality. The magnetometer in 9DoF IMUs could in theory be used to deliver absolute rotational data, and therefore completely correct for the gyroscope drift, but in reality the magnetic field data is extremely jittery and getting close to a metal object seriously distorts it, so any detected field line change is cross checked with gyro and accelerometers. If these don’t show movement, the data is thrown out.

This plus powerful micro-controllers/digital motion processors (DMP) capable of doing a lot of signal processing is how IMUs today achieve much higher precision and repeatability, which is also why all the IMUs recommended for SlimeVR are 6DoF or 9DoF, not just 3DoF gyros. SlimeVR uses the accelerators to correct the data from the gyroscopes, so they absolutely use accelerometers to measure their motion through 3D space, though only for fusion, and usually letting the IMU’s DMP do that. They just don’t use it to later determine the trackers position in 3D space, because SlimeVR uses a skeletal model to determine where neighboring trackers can be. For someone with a 40cm long lower leg, the foot ankle will always be 40cm from the knee joint, its position can be determined with 3DoF data alone, so in this situation it makes no sense to try to calculate the distance from the movement measured by the accelerators, as the result would always be worse. Arm/elbow tracking with SlimeVR is possible, using the same skeletal calculation as with the legs, and it support up to 16 tracked positions, so you could also track the hands. But hand tracking with SlimeVR would be pretty pointless if you are already holding a controller with absolute 6DoF tracking in your hands, so nobody does that. One could track the wrists, but their position could just be derived from elbow position and controller position plus rotation.

And all that without ever having to rely on positional data from the trackers thanks to all our joints. But that doesn’t mean you can’t try, SlimeVR just doesn’t need to. The Oculus Go did not only have rotation tracking, you could lean to the side and forward and the head model would follow. Anatomy and balance limit how far we can lean, so this guessing the 3D position from relative rotation and translation sensor data is again pretty safe.

You could query the SDK for the acceleration on HMD and controller with OVRDisplay.acceleration and OVRInput.GetLocalControllerAcceleration. So the Go controller could not only be rotated in place, but moved through 3D like e.g. a golf club, with some software trying to “guess” the movement from a combination of rotation, translation and assumptions about the arm length. The problem with this approach is always that all the translation data is relative, so you never get an absolute position, only a change from the last estimated position that you have to add. Swinging your arm like hitting with a tennis rack means thousands of direction changes for the accelerators, which quickly leads to the accumulated errors I mention a lot, so it is not a good method for reliably tracking arm movement behind your back on a Quest.

But the Quest 3 can of course do exactly what SlimeVR does: it knows that the controller is attached to the hand, the hand to the arm, the arm to the shoulders, and the shoulders to the head. This plus our shoulder and elbow joints severely limits where arm, hand and the controller could be, so if the user moved the controller over his/her head with a virtual sword, Meta can first follow that movement and correct the data with absolute values from the camera tracking, and then closely project the “flight path” along the same trajectory by checking acceleration and rotation, as our anatomy will most likely make it follow a curve. If the user instead just pushes straight up, there will be no X or Z data from the accelerators. If the user than starts to move the controller to straight to the back, there will be Z data. But if s/he just holds the sword over the head, all sensors will report basically no movement, and the position will be very stable based on the initial trajectory projection for a long time. Again, moving around will quickly ruin the precision.

Its far from perfect and not usable for tracking complex movement, but that is exactly why I in my first comment limited the usefulness to situations where there isn’t a lot of movement, and then went on to describe why this is most likely sufficient for tracking over the head due to “typical” behavior limiting complex movement to where we can see it. We just don’t do complex things over our head without looking up. And that is also why Meta wrote that “because the headset can effectively estimate the position of the controllers when outside of the tracking area for short periods.”

You are trying to make an argument why solely IMU based positional tracking is too imprecise, but nobody is trying to do this. We are talking about a very specific situation where the controller is in an area not tracked by the camera, with the possible positions limited by anatomy, the initial trajectory until it left the tracking space known, and very limited movement expected that could lead to accumulated errors.

Your whole argument why it isn’t possible is basically “accelerometers are not accurate enough to work for more than a fraction of a second before drifting, badly.“, without providing any evidence at all, and a somewhat simplified idea of how IMUs and IMU tracking work. I would fully agree that IMU just based on adding up a series of non-linear acceleration values will quickly become unusable, but that’s just not what Meta is doing here, they are merging a whole lot of well known data and only use the IMU to track the movement within very specific limits.

I don’t know if they ever implemented the trajectory predictions in the same way on the Quest 2, as they could still rely on the tracking cameras facing up, so the artifacts you describe may be simply due to a less smart implementation, and could actually be gone on the Quest 3, if they now have to rely more on IMU tracking and skeletal models. Though I’d still assume that the “locking in space” is mostly a precaution, so they could do better. They just don’t bother, because they want you to move the controllers back to the front, where the cameras can see them reliably.

Excellent posts (as usual).

Ironically enough, you can really see the internal deprioritization of accelerometer tracking vs SLAM with the Touch Pros. Put them down when watching a movie in bed and within a few minutes they’ll either start drifting away or pop up a tracking lost notification.

This doesn’t happen nearly as often with the Touch v2 and I’d bet anything is the result of some algorithm saying “IMU positioning isn’t good enough, what do the cameras say?”. On the Touch v2, the HMD cameras can pick up stationary controllers as long as they’re within field of view but the Touch Pros use their own cameras and have a hard time properly positioning relative to a bed sheet or desk surface in extreme proximity.