Last time, we detailed our initial explorations of single-handed shortcuts systems. After some experimentation, we converged on a palm-up pinch to open a four-way rail system. Today we’re excited to share the second half of our design exploration along with a downloadable demo on the Leap Motion Gallery. Guest Article by Barrett Fox & Martin Schubert Barrett is the Lead VR Interactive Engineer for Leap Motion. Through a mix of prototyping, tools and workflow building with a user driven feedback loop, Barrett has been pushing, prodding, lunging, and poking at the boundaries of computer interaction. Martin is Lead Virtual Reality Designer and Evangelist for Leap Motion. He has created multiple experiences such as Weightless, Geometric, and Mirrors, and is currently exploring how to make the virtual feel more tangible. Barrett and Martin are part of the elite Leap Motion team presenting substantive work in VR/AR UX in innovative and engaging ways. https://youtu.be/LFRKEmzrzP8 We found the shortcuts system comfortable, reliable, and fast to use. It also felt embodied and spatial since the system didn’t require users to look at it to use it. Next it was time to put it to the test in a real-world setting. How would it hold up when we were actually trying to do something else with our hands? We discussed a few types of potential use cases: #1. Direct abstract commands. In this scenario, the system could be used to directly trigger abstract commands. For example, in a drawing application either hand could summon the shortcut system – left to undo, right to redo, forward to zoom in, or backwards to zoom out. #2. Direct contextual commands. What if one hand could choose an action to take upon an object being held by the other hand? For example, picking up an object with your left hand and using your right hand to summon the shortcut system – forward to duplicate the object in place, backward to delete it, or left/right to change its material. #3. Tool adjustments. The system could also be used to adjust various parameters of a currently active tool or ability. For example, in the same drawing application your dominant hand might have the ability to pinch to draw in space. The same hand could summon the shortcut system and translate left/right to decrease/increase brush size. #4. Mode switching. Finally, the system could be used to switch between different modes or tools. Again in a drawing application, each hand could use the shortcut system to switch between free hand direct manipulation, a brush tool, an eraser tool, etc. Moreover, by independently tool-switching with each hand, we could quickly equip interesting combinations of tools. Of these options, we felt that mode switching would test our system the most thoroughly. By designing a set of modes or abilities that required diverse hand movements, we could validate that the shortcuts system wouldn’t get in the way while still being quickly and easily accessible. Mode Switching and Pinch Interactions In thinking about possible abilities we’d like to be able to switch between, we kept returning to pinch-based interactions. Pinching, as we discussed in our last blog post, is a very powerful bare handed interaction for a few reasons: It’s a gesture that most people are familiar with and can do with minimal ambiguity, making it simple to successfully execute for new users. It’s a low-effort action, requiring only movement of your thumb and index fingers. As a result, it’s suitable for high-frequency interactions. Its success is very well-defined for the user who gets self-haptic feedback when their finger and thumb make contact. However, having an ability triggered by pinching does have drawbacks, as false triggers are common. For this reason, having a quick and easy system to enable, disable, and switch between pinch abilities turned out to be very valuable. This led us to design a set of pinch powers to test our shortcut system. Pinch Powers! We designed three pinch powers, leaving one shortcut direction free as an option to disable all pinch abilities and use free hands for regular direct manipulation. Each pinch power would encourage a different type of hand movement to test whether the shortcut system would still function as intended. We wanted to create powers that were interesting to use individually but could also be combined to create interesting pairs, taking advantage of each hand’s ability to switch modes independently. The Plane Hand For our first power, we used pinching to drive a very common action: throwing. Looking to the physical world for inspiration, we found that paper plane throwing was a very expressive action with an almost identical base motion. By pinching and holding to spawn a new paper plane, then moving your hand and releasing, we could calculate the average velocity of your pinched fingers over a certain number of frames prior to release and feed that into the plane as a launch velocity. Using this first ability together with the shortcuts system revealed a few conflicts. A common way to hold your hand while pinching a paper plane is with your palm facing up and slightly inwards with your pinky furthest away from you. This fell into the gray area between the palm direction angles defined as ‘facing away from the user’ and ‘facing toward the user’. To avoid false positives, we adjusted the thresholds slightly until the system was not triggered accidentally. To recreate the aerodynamics of a paper plane, we used two different forces. The first added force is upwards, relative to the plane, determined by the magnitude of the plane’s current velocity. This means a faster throw produces a stronger lifting force. The other force is a little less realistic but helps make for more seamless throws. It takes the current velocity of a plane and adds torque to bring its forward direction, or nose, inline with that velocity. This means a plane thrown sideways will correct its forward heading to match its movement direction. With these aerodynamic forces in play, even small variations in throwing angle and direction resulted in a wide variety of plane trajectories. Planes would curve and arc in surprising ways, encouraging users to try overhanded, underhanded, and side-angled throws. In testing, we found that during these expressive throws, users often rotated their palms into poses which would unintentionally trigger the shortcut system. To solve this we simply disabled the ability to open the shortcut system while pinching. Besides these fixes for palm direction conflicts, we also wanted to test a few solutions to minimize accidental pinches. We experimented with putting an object in a user’s pinch point whenever they had a pinch power enabled. The intention was to signal to the user that the pinch power was ‘always on.’ When combined with glowing fingertips and audio feedback driven by pinch strength, this seemed successful in reducing the likelihood of accidental pinches. We also added a short scaling animation to planes as they spawned. If a user released their pinch before the plane was fully scaled up the plane would scale back down and disappear. This meant that short unintentional pinches wouldn’t spawn unwanted planes, further reducing the accidental pinch issue. The Bow Hand For our second ability we looked at the movement of pinching, pulling back, and releasing. This movement was used most famously on touchscreens as the central mechanic of Angry Birds and more recently adapted to three dimensions in Valve’s The Lab: Slingshot. Virtual slingshots have a great sense of physicality. Pulling back on a sling and seeing it lengthen while hearing the elastic creak gives a visceral sense of the potential energy of the projectile, satisfyingly realized when launched. For our purposes, since we could pinch anywhere in space and pull back, we decided to use something a little more lightweight than a slingshot: a tiny retractable bow. Pinching expands the bow and attaches the bowstring to your pinched fingers. Pulling away from the original pinch position in any direction stretches the bowstring and notches an arrow. The longer the stretch, the greater the launch velocity on release. Again we found that users rotated their hands while using the bow into poses where their palm direction would accidentally trigger the shortcut system. Once again, we simply disabled the ability to open the shortcut system, this time while the bow was expanded. To minimize accidental arrows spawning from unintentional pinches, we again employed a slight delay after pinching before notching a new arrow. However, rather than being time-based like the plane spawning animation, this time we defined a minimum distance from the original pinch. Once reached, this spawns and notches a new arrow. The Time Hand For our last ability, we initially looked at the movement of pinching and rotating as a means of controlling time. The idea was to pinch to spawn a clock and then rotate the pinch to turn a clock hand, dialing the time scale down or back up. In testing, however, we found that this kind of pinch rotation actually only had a small range of motion before becoming uncomfortable. Since there wasn’t much value in having a very small range of time-scale adjustment, we decided to simply make it a toggle instead. For this ability, we replaced the pinch egg with a clock that sits in the user’s pinch point. At normal speed the clock ticks along quite quickly, with the longer hand completing a full rotation each second. Upon pinching, the clock time is slowed to one-third normal speed, the clock changes color, and the longer hand slows to complete a full rotation in one minute. Pinching the clock again restores time to normal speed. Continued on Page 2: Mixing & Matching Mixing & Matching Since each hand can change modes independently, quickly mixing and matching hands can create interesting combinations of abilities. Double Plane Hands Equipping the plane ability on both hands lets you create new planes rapidly, throwing them in interesting overlapping arcs and curves. Free Hand and Time Hand You can use one hand to slow time while using the other free hand to gently tap or grab planes. Bow Hand and Time Hand One hand slows time and the flight of planes, while the other picks them off with arrows from the bow hand. Double Time Hand Using time abilities on both hands lets you stack their effects, slowing time to one-ninth normal speed. Double Bow Hands Equipping the bow ability on both hands initially let you use two individual pinch bows. This was fun but we thought we could do a little better. Two-handed bows are a staple of VR gaming today, taking full advantage of physical movement with 6 DoF input. So, when both hands have the bow ability enabled, pinching with the left hand will expand a larger slightly offset bow, while pinching with the right hand will spawn and notch flaming arrows! Holding both pinches lets a user aim their arrows using two pivot points, allowing much greater control and accuracy. In using the two-handed bow we initially required the user to pinch near the bowstring with their right hand to notch a new flaming arrow, then pull back and release to launch it. However, we found that letting users pinch anywhere to notch a new arrow, while a little less realistic, reduced arm fatigue. It also let them fire flaming arrows at an incredibly high rate by rapidly tapping their index fingers and thumbs together. Interestingly, some users reported being able to do this action quicker than they could click a mouse or press a mechanical button. Continued in Page 3: UX Refinements UX Refinements While user testing the shortcut system with all these pinch powers, we quickly noticed a couple of issues many users had right at the start. First, if the system and all the powers were enabled when a user put on the headset, they would be overwhelmed by all the sounds and animations from simply moving their hands around. To ease users into the system we added a little introduction where users place their hands (with empty PowerCube holders on the back) into hand outlines on a podium. This lets them receive the PowerCubes. Then, starting in the unpowered Free Hand mode, they flip their hand over and pinch the PowerCube. At first, they’re only able to switch into the Plane Hand. After successfully switching to and using the Plane Hand, the Bow Hand option becomes available, followed by the Time Hand option. The other issue was one we’d predicted – users would try to tap, push, or grab the PowerCube on the back of their hand with their other hand. Despite our various attempts to un-afford other hand interaction with the PowerCube, many users still tried to interact with it. Rather than fighting this behavior we decided to embrace it and use it to include a new feature. We added the ability to disable the PowerCube entirely by tapping it with the other hand. This turned the entire shortcut system off as well as any pinch powers, allowing totally free movement of the hands. Tapping the PowerCube again reactivates the system, resetting the hand to the last enabled mode. All Together Now! In practice, these three abilities and switching between them using the shortcut system come together as a sandbox experience about throwing paper planes and hitting them with tiny arrows to transform them into colorful flocks. Hitting a white plane with a single regular arrow converts it into a bright ribbon-trailing version of itself and send it to join its own flock. Hitting a white plane with a flaming arrow sets it on fire, burning it up and destroying it. However, hitting a transformed colorful plane with a flaming arrow makes it burn with a plane-specific flame color while glowing brightly for a short time. If you have a Leap Motion Controller mounted to an HTC Vive or Oculus Rift, you can download the executable and try it yourself. We’re very interested to hear how you find the Shortcuts system in practice, especially after a few minutes of use. Other Guest Articles by Barrett Fox & Martin Schubert Leap Motion Explores Ways to Make Controller-free Input More Intuitive and Immersive Summoning & Superpowers – Designing VR Interactions at a Distance Scaffolding in VR – Interaction Design for Easy & Intuitive Building Designing Single-handed Shortcuts for VR & AR