For new computing technologies to realize their full potential they need new user interfaces. The most essential interactions in virtual spaces are grounded in direct physical manipulations like pinching and grabbing, as these are universally accessible. However, the team at Leap Motion has also investigated more exotic and exciting interface paradigms from arm HUDs and digital wearables, to deployable widgets containing buttons, sliders, and even 3D trackballs and color pickers. Guest Article by Barrett Fox & Martin Schubert Barrett is the Lead VR Interactive Engineer for Leap Motion. Through a mix of prototyping, tools and workflow building with a user driven feedback loop, Barrett has been pushing, prodding, lunging, and poking at the boundaries of computer interaction. Martin is Lead Virtual Reality Designer and Evangelist for Leap Motion. He has created multiple experiences such as Weightless, Geometric, and Mirrors, and is currently exploring how to make the virtual feel more tangible. Barrett and Martin are part of the elite Leap Motion team presenting substantive work in VR/AR UX in innovative and engaging ways. As we move from casual VR applications to deeper and longer sessions, design priorities naturally shift toward productivity and ergonomics. One of the most critical areas of interaction design that comes up is mode switching and shortcuts. Today we use keyboard shortcuts so often that it’s difficult to imagine using a computer without them. Ctrl+Z, Ctrl+C, and Ctrl+V are foundational to the efficiency of keyboard and mouse input. Most of you reading this have committed these to muscle memory. In VR we’ve seen controller inputs adopt this shortcut paradigm relatively easily by remapping commands to buttons, triggers, trackpads, and analog sticks. To increase or decrease the brush size in Tilt Brush you swipe right or left on the trackpad of your brush hand. But what happens when we think about one-handed rapid selections for bare-handed input? This requires a different kind of thinking, as we don’t have buttons or other mechanical inputs to lean on. In our previous work, we’ve mapped these kinds of commands to either world-space user interfaces (e.g. control panels) or wearable interfaces that use the palette paradigm, where one hand acts as a collection of options while the other acts as a picker. But if we could mode switch or modify a currently active tool with just one hand instead of two we would see gains in speed, focus, and comfort that would add up over time. We could even design an embodied and spatial shortcut system without the need to look at our hands, freeing our gaze and increasing productivity further. Direct Manipulation vs. Abstract Gestures One way to activate a shortcut with a single hand would be to define an abstract gesture as a trigger. Essentially this would be a hand pose or a movement of a hand over time. This is an exception to a general rule at Leap Motion, where we typically favor direct physical manipulation of virtual objects as an interaction paradigm over using abstract gestures. There are a few reasons for this: Abstract gestures are often ambiguous. How do we define an abstract gesture like ‘swipe up’ in three-dimensional space? When and where does a swipe begin or end? How quickly must it be completed? How many fingers must be involved? Less abstract interactions reduce the learning curve for users. Everyone can tap into into a lifetime of experience with directly manipulating physical objects in the real world. Trying to teach a user specific movements so they can perform commands reliably is a significant challenge. Shortcuts need to be quickly and easily accessible but hard to trigger accidentally. These design goals seem at odds! Ease of accessibility means expanding the range of valid poses/movements, but this makes us more likely to trigger the shortcut unintentionally. To move beyond this issue, we decided that instead of using single gesture to trigger a shortcut, we would gate the action into two sequential stages. The First Gateway: Palm Up Our interaction design philosophy always looks to build on existing conventions and metaphors. One major precedent that we’ve set over time in our digital wearables explorations is that hand-mounted menus are triggered by rotating the palm to face the user. This works well in segmenting interactions based on which direction your hands are facing. Palms turned away from yourself and toward the rest of the scene imply interaction with the external world. Palms turned toward yourself imply interactions in the near field with internal user interfaces. Palm direction seemed like a suitable first condition, acting as a gate between normal hand movement and a user’s intention to activate a shortcut. The Second Gateway: Pinch Now that your palm is facing yourself, we looked for a second action which would be easily triggered, well defined and deliberate. A pinch checks all these boxes: It’s low-effort. Just move your index finger and thumb! It’s well defined. You get self-haptic feedback when your fingers make contact, and the action can be defined and represented by the tracking system as reaching a minimum distance between tracked index and thumb tips. It’s deliberate. You’re not likely to absent-mindedly pinch your fingers with your palm up. Performing both of these actions, one after another, is both quick and easy, yet difficult to do unintentionally. This sequence seemed like a solid foundation for our single-handed shortcuts exploration. The next challenge was how we would afford the movement, or in other words, how someone would know that this is what they needed to do. Thinking back on the benefits of direct manipulation versus abstract gestures we wondered if we could blend the two paradigms. By using a virtual object to guide a user through the interaction, could we make them feel like they were directly manipulating something while in fact performing an action closer to an abstract gesture? The Powerball Our solution was to create an object attached to the back of your hand which acts as a visual indicator of your progress through the interaction as well as a target for pinching. If your palm faces away, the object stays locked to the back of your hand. As your palm rotates toward yourself the object animates up off your hand towards a transform offset that is above but still relative to your hand. Once your palm fully faces toward yourself and the object has animated to its end position, pinching the object – a direct manipulation – will trigger the shortcut. We dubbed this object the Powerball. After some experimentation, we had it animate into the pinch point (a constantly updating position defined as the midpoint between the index finger and thumb tips). This blend of graphic affordance, pseudo-direct manipulation, gestural movement, and embodied action proved easy to learn and ripe with potential for extension. Now it was time to look at what kinds of shortcut interface systems would be ergonomic and reliably tracked from this palm-up-pinched-fingers position. Continued on Page 2: Spatial Interface Selection » Spatial Interface Selection We quickly realized that the simplest system would be to pinch and then release the Powerball. This would let you execute a single shortcut command, switch between two modes, or even cycle between more than two modes with successive pinches. However, the real utility of a shortcut system comes from having quick and direct access to a range of options. So we explored systems which displayed more than one shortcut and involved moving the pinched fingers into predetermined positions to choose one. Our first step was to investigate comfortable ranges of movement. We found a few movement patterns which felt easily repeatable without strain. The first was rotating our pinched fingers down and inwards by rotating our wrist and forearms. Another was tilting our pinched fingers forward or backwards, hinging at the wrist. And a third was translating our pinched fingers in any direction by hinging at our elbows and shoulders. From these movement patterns, we sketched out a few spatial selection systems which would let a user move their pinch point to choose a shortcut option, then release their pinch to confirm their selection. One was based on trigger volumes where the Powerball would move 1:1 with the user’s pinched fingers. When released within a volume, it would confirm a selection. [gallery size="large" columns="2" ids="78167,78159"] The arc volumes and translation volumes concepts. The other was based on a rail system where the Powerball would attempt to move with the user’s pinched fingers but remain constrained along the rail, snapping between dots along the rail’s length. [gallery size="large" columns="2" ids="78157,78158"] The arc rail and translation rail concepts. We then prototyped three of these sketches; the arc volumes, the arc rail, and the translation rail. In testing we again discovered benefits and issues with each system which weren’t apparent from the sketches alone. Variant #1: Arc Volume In the arc volume system, our first prototype, determining how to anchor the volumes proved difficult. With the Powerball pinched to open the shortcut system, we first tried spawning the volumes in world space, locking them in place wherever we had originally pinched. We’d rotate our pinch through the volumes into the one we wanted to select, and then release. One drawback with this method was that it required that you avoid translating your hand much. To ensure that users could move comfortably, we anchored the volumes to your hand so they would move with it. Now the volumes were oriented along three axes when they spawned – one pointing forward from the hand, another pointing inwards, and a third pointed toward the camera. This system was now relatively fast and comfortable to use. However, the volumes themselves were quite busy and made it difficult to discern the location of the Powerball, and whether it had moved from one volume into the next. Assigning colors and animating the currently selected volume out as an extrusion helped, but it still felt like a lot of overlapping information. Variant #2: Arc Rail Next we prototyped the arc rail system. It solved some of the previous system’s issues right away. By constraining the Powerball to dots along a rail, users were free to rotate their pinch inward and down along whatever curve they felt was comfortable. The Powerball remained locked to the rail and the snapping showed its progression more clearly than freeform movement through volumes. The rail system also removed a lot of the visual clutter of the volumes. But we still needed a way to clearly convey the currently selected shortcut option – especially when the system was in the user’s peripheral vision. We decided to infuse this information into the Powerball itself through an extruding blendshape animation driven by pinch strength. We also started experimenting with where the Powerball would live on the back of a user’s hand as well as its shape, looking to afford pinching more clearly. This led to the Poweregg. Much better! Now you could pinch, rotate inward, and down in whatever way was most comfortable. You could also clearly see which shortcut you were selecting, and release your pinch to confirm. The only remaining functional issue with this system was that to reliably select the middle shortcut option, you needed to move quite deliberately so as not to overshoot into the last option. You also needed to look at your pinched fingers to do so. Choosing the first option or the third at the end of the rail didn’t require this, and as a result were much quicker and more reliable. Taking this into account, we set about prototyping our third system, the translation rail. Variant #3: Translation Rail In this system, since we were translating our pinched fingers and not rotating them, we decided to lock the translation rails in world space upon the initial pinch. Unlike with the arc rail system – where all three shortcut options were accessible along the same direction of movement – this system designated a different direction for each shortcut. By doing this, we ensured that a user could choose a direction and corresponding shortcut without the need to be deliberate about how far they translated their pinch. This selection action was fast, reliable, and didn’t require precise movements, or even that the user look at their pinched fingers. We also modeled and animated the Powercube – a squishy shapeshifting cube which morphed into a tiny pinchable accordion and back again to give more visual feedback at each stage. This system felt embodied and spatial in exactly the way we had imagined a hands-based shortcut system in VR should feel. Using the system repeatedly built up muscle memory of how the options were organized, a critical component of a successful shortcut system. It could easily handle four options or could be expanded with rails going up or down for a total of six options. The spherical volume around your initial pinch position could theoretically be sliced up in even more ways to accommodate as many options as needed. That wraps up the first section of our single-handed shortcut sprint – but our work here wasn’t done yet. In order to properly validate our translation rail shortcut system, we needed to test it in a few more areas. We needed to ask how learnable the system was for new users. Would they try to press the Powerball while it was on the back of their hand and if so how could we either avoid affording that action or embrace and use it? What kinds of affordances did we need to guide users through each stage of the system? How could we load information into their visual and audio channels to create an even greater sense of tactility and draw their attention to where it was needed? Finally, how well did the system hold up while actually using our hands to do something else? Did the system enable easy mode switching while in the middle of actual tasks? Would the system trigger accidentally and get in the way when we didn’t intend it to? Watch for our next blog post where we attempt to answer these questions – and others we didn’t even know we needed to ask.