Vive Tracker Used to Create Pressure Sensitive Photoshop Stylus for VR

By
Dominic Brennan
-
23

Developer Jimmy Gray recently posted a short video (see above), showing the latest version of a DIY motion-tracked VR stylus prototype. Using an HTC Vive Tracker mounted above a stylus-shaped unit, a VR stylus can be tracked and rendered, usable in Photoshop and capable of pressure sensitivity.

Update (7/8/17): Gray has released a new video showing a more polished version of his prototype which he now calls the Versa. The video shows how the device can be held like a traditional controller (with the Tracker facing upward), utilizing a thumbpad for typical interactions.

When flipped upside down with the tip pointing outward the Versa is in Stylus Mode, which gives the user a pencil-grip for better control with drawing tasks like sketching in Tilt Brush. Gray calls it Tablet Mode when the device is used against a flat surface, allowing for precise control for 2D interactions.

This latest Versa prototype appears to be fully wireless now compared to the prior version which appeared tethered to a computer. Though bulky today, it’s an interesting concept for a different type of ‘tool’ that might become a staple of VR input in the future. It isn’t clear how far Gray plans to take the project, which seems like it could be on a trajectory to a crowdfunding campaign; we’ve reached out to learn more about Versa’s future.

Original Article (4/12/17): According to Gray’s tweets, the prototype emulates an HID device using ‘Teensy‘, a very compact USB development board. For now, this part of the device appears to connect directly to the PC via a USB cable, but it could potentially be made wireless, like the Vive Tracker itself, in future iterations.

It’s easy to see where the motivation to start this project came from, as Gray is also an artist, with a few examples of his 3D sketch work in Tilt Brush on his YouTube channel. Like many of these motion-tracked accessory projects, the stylus started life using a Vive motion controller to perform the tracking, and this earlier footage shows the flex of a stylus tip being connected to an input on the Vive controller’s trackpad, acting as the pressure sensitivity gauge.

Tracking a stylus in this way is yet another example of the precision of Valve’s Lighthouse technology, something that was highlighted by Valve themselves in a similar way with the marker board in The Lab. However, the system has its limits, and is not going to be as accurate as using a conventional stylus, but there may be applications where a VR stylus is favourable, especially thanks to the real feedback it can provide.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


Dominic Brennan
The trial version of Microsoft’s Monster Truck Madness probably had something to do with it. And certainly the original Super Mario Kart and Gran Turismo. A car nut from an early age, Dominic was always drawn to racing games above all other genres. Now a seasoned driving simulation enthusiast, and former editor of Sim Racer magazine, Dominic has followed virtual reality developments with keen interest, as cockpit-based simulation is a perfect match for the technology. Conditions could hardly be more ideal, a scientist once said. Writing about simulators lead him to Road to VR, whose broad coverage of the industry revealed the bigger picture and limitless potential of the medium. Passionate about technology and a lifelong PC gamer, Dominic suffers from the ‘tweak for days’ PC gaming condition, where he plays the same section over and over at every possible combination of visual settings to find the right balance between fidelity and performance. Based within The Fens of Lincolnshire (it’s very flat), Dominic can sometimes be found marvelling at the real world’s ‘draw distance’, wishing virtual technologies would catch up.
  • Me

    This is like building a blender for elephants. It might work, and even involve nice science behind, but what is the damn point of that thing ? I can’t see a single valid use case.

    • NooYawker

      I think the last paragraph explains it. It’s all experimental. Like the guy who got the tracker working on his rift. But he had to use lighthouses. So, it’s just people seeing if they could.

      • jlschmugge

        I think we get that, and kudos to Mr. Gray for getting this to work. I wont deny that it is clever, but I can’t see how this is practical. I really tried to think where this could be used instead of a pressure sensitive tablet or stylus-enabled touchscreen, so I’m curious where Mr. Gray is going with this.

        How much are Wacom tablets now? I’m lucky to have a Surface 4 if I need to draw, but granted its a smaller screen. This is a long-shot to save a buck with an $800 headset.

        And to Road to VR. Really please let people know that you are posting an article about them. I think this is the second time I saw someone caught off guard. You are a popular enough VR publication that I think people will give you the time for a phone interview.

        • James Gray

          I totally get the skepticism. Especially remembering the pixel resolution of current HMDs, tracking reliability, etc. Yeah, this video was more demonstrating the concept (that I could simulate using a pen display with VR), however other individuals have indicated that they would find this useful. I do plan on showing a more polished video at some point demonstrating some different use cases with a more marketed approach. I don’t really know where it stands as a potentially successful product idea or not, and I don’t want to come off as thinking this is some groundbreaking thing, I just feel it could be useful for some occasions.

          But to pique your curiosity a bit, and to give you more clarification as to where I think it *could* go:

          Since that video recording, I implemented a haptic motor driver in it for providing haptic feedback when drawing on a virtual canvas in mid-air. The haptic vibration would simulate the feeling of friction generated from drawing on a surface.

          When using it in mid-air, there’s no need to be tied to the specific dimensions of a real-world surface area — this would allow someone to draw in a program like photoshop with the canvas being virtually as big or as small as they want. Whereas actual consumer hardware devices like Cintiqs and whatnot are static in terms of surface area and only get so big.

          I could also see a “sub-vr” use case where people use it mainly for the Lighthouse tracking, without an HMD — so they could draw on their computer screens (or on some plain surface) directly (though not with a rough tip of course which would be damaging to the screen, I will plan to reveal how I plan on enabling this possibly in a future update).

          Edit: Also, imagine multiplayer virtual art classes/sessions where participants all draw a model (human, vehicle, etc) in the center of the virtual room. Or 2D illustrators drawing in VR while getting to walk around a 3D scene, or just drawing from 3D reference in VR with all the virtual space that could be provided to re-position stuff, and not be limited to reference images eating up monitor real-estate.

          • Dave Myers

            James, Are you going to be at VRLA?

          • James Gray

            Unfortunately not, but I plan on getting a booth at the Maker Faire coming up in Detroit soon. I will definitely be looking to attend future events in the west coast, just wasn’t ready for VRLA.

    • James Gray

      Hey man, it simply started from the idea that I’d love to be able to use a pen display to do some concept art and illustration work. Good pen displays (like Wacom Cintiqs) aren’t too cheap and I already had a Vive, so I had the idea to make basically a virtual pen display where I could draw directly on screen, except the screen is virtual and the stylus positioning is captured using Lighthouse. I was tinkering with just using the Vivemotes as the input, but as an artist who wants to use an expressive tool like a Wacom, I needed to emulate a stylus, which is why I used hardware to accomplish that. It’s totally unrefined and this one little video doesn’t market the concept well at all. I didn’t even know this article was going to be published! But I did get some positive feedback from people that they’d want to use something like this, so I plan on at least making a better video with a producer friend of mine, to demonstrate what a more polished experience would look like.

      • Paul

        Dude, pay not attention to this guy. Bravo for doing development work. Congratulations. The medium is new and evolving and I think ideas like this are fantastic. The technology and cost will shrink and this is a great step!

        • Bryan Ischo

          Skepticism is a healthy and necessary part of the VR ecosystem as well.

      • Jeff

        I think this is a great idea! With one or two extra tips it seems like this could be an awesome way to draw our physical spaces and objects for use in VR. It’s cool as a pen too, but have you guys tried that at all? I would love to draw my play space, add whatever I want, and hang out in it…

      • Hogo

        please keep up the work on this! it’s a really cool idea, and as the resolution of the vive and other VR headsets increases, it will become more and more interesting. It’s really easy for someone to jump on a comment section and spout useless negative crap like “Me” did. A lot harder to try new things and create new experiences. bravo.

      • Sponge Bob

        Nice work, dude
        The problem is not your lack of industrial design skills (there are plenty of those around but they cost $$$ and require VC funding – there is no way around it)
        The problem is the whole setup – 2 lighthouses making it stationary and bulky (expensive too)
        I have always said that future VR mouse or stylus must be tracked relative to headset , not to 2 outside basestations
        Nvertheless , yo uare thinkin in the right direction
        The existing VR controllers like Vive or Touch are an ugly pos not even remotely suitable for precision productivity applications in VR
        Something much smaller and ergonomic is needed (in addition to getting rid of 2 external basestations)
        think about it

    • CazCore

      good thing the world isn’t limited by YOUR imagination limits. :)

  • Bryan Ischo

    How many multiples of a normal stylus does this thing weight? 10x? 100x? How will anyone use this for any significant length of time given that? How will artists invoke subtle movements with the pen when they have to balance all that weight?

    It’s fine for a gee-whiz tech demo, but I doubt anyone would seriously use this.

    • James Gray

      I’m sort of limited by the Vive Tracker and my lack of industrial design skills at the moment, which is why it’s so bulky. My near-term goal is to get a new design without using the Tracker so that I can make an entirely new tracked object, but this will take quite a bit of work. I thought it would be okay for it to weigh a bit more than a typical stylus if it could also function as a generic VR controller. But I agree that it definitely needs to be capable for serious work, I would not want this to be a product if it wasn’t capable.

      • Gus Bisbal

        James, I have a small Australian prototyping business. I can help you with this. I am producing haptics for VR. If you want I can give you a hand on how to make your own stylus from scratch

        • James Gray

          Thanks Gus, I’m interested. Would you like to DM me on Twitter? I’m @jimmygray_ on Twitter.

          • Gus Bisbal

            I sent a replied Tweet. At the least I will let you know the process and the how to avoid costs that don’t need to be there.

      • Bryan Ischo

        You seem very aware of the limitations of the demo, and it sounds like your making some contacts with people who can help with the next steps, so best of luck to you. I think the concept of a tracked stylus is interesting and I hope your finalized version is a success.

      • Sponge Bob

        you can use those chips from Triad Semiconductor to build your own tracked object but its not gonna be any smaller or better
        those vive trackers have their weird ugly shape for a good reason

  • Mark Vincent

    Hey James,

    This looks very promising! As a VR developer I’m especially interested in the live link between Unity and Photoshop. Could you point me in the right direction on how to achieve this?

    Thanks!

  • Rogier de Leeuw

    what about having a tracker on your wrist and let it track a stylus in your hands?

    • James Gray II

      Yeah, I had a prototype where the tracker is mounted on the back of the wrist. Might be going forward with that. Thanks for the feedback. That first one you linked to reminds me of the Novint Falcon.