‘Modbox’ Developer Shows Off Knuckles Controllers in Action with SteamVR Skeletal Input

By
Scott Hayden
-
29

Alien Trap’s Lee Vermeulen, one of the developers behind VR creation tool Modbox, received a brand new pair of Knuckles controllers recently. One of his first missions: show off Valve’s Skeletal Input system in detail, which was released in beta to developers late last month.

In a follow-up to his earlier video, which showed off how Valve’s SteamVR Skeletal Input works with a standard HTC Vive controller, Vermeulen has put together a new video that zeros in on the high level of hand presence Valve’s upcoming motion controller can afford a user.

Using a HTC Vive for AR passthrough and a Zed Mini stereoscopic camera to capture the scene outside the headset, Vermeulen puts Knuckles through its paces, giving us a good look of a 1:1 comparison of physical hand movements and how the match up to the virtual counterparts.

Image courtesy Cloudhead Games

As seen in the video, Valve’s SteamVR Skeletal Input essentially does the job of estimating where a user’s fingers are at any given moment, and reproducing an in-game hand model to match. Data is gathered from Knuckles’ integrated capactive sensors, which are on many parts of the controller including its touchpads, control sticks, buttons, and capacitive ‘hot spots’, which line the grip of the controllers to capture individual finger movements. The Knuckles dev kit seen in the video is the company’s latest public iteration—a critical redesign of the previous, control stick-less design.

SEE ALSO
Meta Opens Project Aria to Researchers Tackling All-day AR Challenges

In the video, Vermeulen demonstrates how Skeletal Input behaves both with and without a controller rendered in his hand. Featuring an ‘open palm’ design, he can even pet his cat while keeping the controllers strapped on.

There’s still no word on consumer release of Knuckles, but the fact that Valve has already seeded big studios and indie developers alike with the motion controller dev kit means that hand presence is definitely the next big area of development for SteamVR hardware ecosystem and the apps therein. We’re certainly hoping for SteamVR base station-capable headsets, such as HTC Vive, to finally get access to greater hand presence—something Oculus Rift owners have had (albeit to a lesser extent than Knuckles) via the system’s 2016-era dedicated motion controllers, Oculus Touch.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


Scott Hayden
Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 4,000 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • Luke

    fingers sensor tracking is not dynamic right? it reads or open or close but can not follow all the steps in between?

    • Fulby

      It attempts to – it has capacitive sensors which can track how close the finger is too the controller, and maybe how far round each finger is wrapped.

      • Luke

        thx for reply, I wish to see a video that show that, in this one seems to do it pressing the triggers.

        anyway it cant track “opposable thumbs” natural movement right?

    • HybridEnergy

      I don’t know what you mean, it’s pretty obvious (just look at 27 seconds into the vid about) where the fingers gently close and open just as he does in real life.

      • Luke

        imho from 27 sec he is pressing the triggers (the animation may be related also to the physical buttons), anyway this video is not explicative and I’m very skeptical about this controller. I’ll wait for next videos or a review.
        the animations seems to be prerendered and seems to be not possible to track the real movements of the fingers, this would translate in having less immersion. for next gen I expect full motion tracking of the hands fingers plus physical controller, but this one seems to come at a middle way. it looks amazing but a little bit in late for my hype expectations.

        • HybridEnergy

          Well more power to you, but even from what the devs behind The Gallery games said it seems to track the fingers more than just on and off. I feel, even if it was just that though people are going to jump on this controller because the VIVE community just can’t wait to get their hands on an analog stick and get rid of the wands it seems.

    • Zerofool

      Check this video out, especially the portion starting at 1:36

      https://youtu.be/JLjs_PtVyKM?t=96

      • Luke

        thx this is very cool!

  • Ombra Alberto

    If I had a Vive it would certainly be my purchase.

    I hope Oculus projects something similar.

    I wish great fun to all Vive users.

    • Lucidfeuer

      Well that’s a neutral statement. Can’t disagree with it.

  • HybridEnergy

    Looks awesome, can’t wait to get them. I hope the touch pad is still more or less functional. Some games were designed for it.

  • Lucidfeuer

    I still don’t understand what causes this video capture tracking flickering. Also pretty similarly, it’d be better if they took the LeapMo ambiguation approach ie. instead of tracking finger movement 1:1 in accuracy it could visually make up for fluid articulation in between steps of finger tracking.

    • Jerald Doerr

      If your talking about the jitter like the shaking of the hands I think the bone setup is wrong… the pivot point for the hole hand model is setup from the grip of the controller instead of the wrist and the model does not have a wrist or forearm with IK so it makes hand movement look weird… as for the fingers I think most of that can be fixed in software. All in all I’ll take a set unless they are 400+

  • Molmir

    If it does not have 1:1 tracking, they might aswell stop trying to reinvent the wheel and implement leap motion for the finger tracking while keeping the rest of the controller as is, but placing the thumbstick in the middle. Trackpad ish thing should not have the best comfort position. Sometimes i wonder if they intentionally put the thumbstick in a bad location so that they can show “progress” for future iterations…

    • Sandy Wich

      I agree 100%

      Forcing that stupid ass trackpad is ridiculous. But it’s also clearly in the, “comfort zone”, of where the thumbstick should be.

      Like, I haven’t used this thing yet, but it seems to me like in order to use the thumbstick version i’ll have to stretch my thumb out further than I should have to.

      I hope I’m wrong.

      • Caven

        I think you might be looking at the controllers backward. Based on layout, it looks like the thumbstick is actually the closest control to the thumb, followed by the trackpad, and then the buttons farthest out. From what I can see, if one were to hold a Knuckles controller vertically, the thumb would point straight up when resting on the thumbstick, and the thumb would tilt progressively to the side to reach the touchpad or buttons.

  • Sandy Wich

    Now that is fkn cool. Just imagine how realistic it would be to grab your waifu’s boob- B-BOOKS! Books. BOOKS. From her ches-SHELF.

    Fuck.

    • Trenix

      Or you can just get a girlfriend?

      • HybridEnergy

        Those are even more expensive and require far more than a 1080 ti to run. Plus you can’t turn them off.

    • Vegeta785

      Yeah

  • Sandy Wich

    I think this is one of if not the most important things being developed for VR right now.

    Think about it. Everyone’s struggling to find a way to move in VR. Tredmills, shoes, discs, jumpsuits etc. And although I think that’s very important as well to figure out, it’s obviously far enough away that it’s not something we’re going to have anytime soon. And what else is there? Wires? Yes. Very important to remove. Higher FOV/Resolution? Yep, same. But do any of these really matter in comparison to having you’re only interaction with this experience being that of someone with nerve damage? Like you can barely pick up objects in games without them flying around. Most of the time it’s just a snap-attach, not genuine gripping an object. This new Knuckles controller is almost like giving a totally blind/half-deaf man perfect hearing again so he can enjoy music to it’s absolute fullest, even if he can’t see. Like how we can’t move well in VR.

    • Trenix

      I got to admit, after using a rift controller, it’s hard to use any other. The controller is extremely important and so is making people move naturally rather than all jumpy up like what we experience in VR currently. It’s not immersive when it looks like people’s hands look like they just came out of a woodcutter.

  • Andrew Jakobs

    I wonder how they do the fingertracking, because if it’s using a magnetic field, then I wonder how healthy it is in the long run..

    • Caven

      We spend every second of our lives enveloped in a planet-sized magnetic field. I don’t see how a magnetic field generated by a wireless game controller could be any more hazardous to health than all the magnetic fields we already expose ourselves to from permanent magnets, electric motors, and the Earth itself.

      • Andrew Jakobs

        It all depends on how strong the field is. Spending a lot of time in a CT scanner will disrupt your body.

        • Stuart Marshall

          Have you seen the size of a CT scanner? I doubt that exists in these controllers. Plus the tracking is done by estimating the position of the fingers in relation to the touch sensors positioned on the controller, did you read the article?

        • Caven

          That’s because a CT scanner uses a moving x-ray beam, so your body is constantly getting bombarded by ionizing radiation. That’s the same problem good old fashioned x-ray machines have always had.

          An MRI scanner is the one that uses magnets, The only reported health risk directly related to the magnet field is getting injured or killed by metal objects that are attracted to the magnet, due to the incredible power of the very large, superconducting electromagnets used. For instance, a child’s head was crushed by an oxygen tank that was pulled across the room by an MRI scanner’s magnetic field. I’m pretty sure a magnetic field generated by an electromagnet powered by a single AA battery or small Li-Ion battery won’t be anywhere near that powerful.

          By the way, don’t use Oculus Touch because they have magnets in them. The neodymium magnet used to keep the battery door in place on each controller has a magnetic field density comparable to that of MRI scanners. I’d also recommend staying away from headphones. Speakers inherently require magnets in order to function, and sticking a person’s brain right between two of them can’t be good. After all, the magnets have to be powerful to allow a speaker membrane to oscillate several thousand times per second. That of course means removing the built-in Oculus headphones, or never upgrading to the Vive Deluxe Audio Headstrap.

          But seriously, magnets are everywhere, and a lot of them are quite powerful. They’re in your phone (even old fashioned wired phones), they’re in your home appliances. There are even more of them in a laptop than you’d expect, since not only do the speakers and fan motors have magnets, but the bezels usually have magnets that are used so the laptop can tell when the lid is closed. Worrying about the mere possibility of magnetic fields from the Knuckles controllers is like worrying about drowning in one particular drop of ocean water.

  • Cool, but I hoped for a little better accuracy

  • Martijn Valk

    I have to say this looks a lot better than the demo I saw about a half a year ago. Here’s hoping they won’t be too expensive..