On free-hand TV control: experimental results on user-elicited gestures with Leap Motion

Abstract

We present insights from a gesture elicitation study conducted for TV control, during which 18 participants contributed gesture commands and rated the execution difficulty and recall likeliness of free-hand gestures for 21 television control tasks. Our study complements previous work on gesture interaction design for the TV set with the first exploration of fine-grained resolution 3-D finger movements and hand gestures. We report lower agreement rates than previous gesture studies (${\mathcal {AR}}=.158$) with 72.8 % recall rate and 15.8 % false positives, results that are explained by the complexity and variability of unconstrained finger and hand gestures. However, our observations also confirm previous findings, such as people preferring related gestures for dichotomous tasks and more disagreement occurring for abstract tasks, such as “open browser” or “show the list of channels” for our specific TV scenario. To reach a better understanding of our participants’ preferences for articulating finger and hand gestures, we defined five measures for Leap Motion gestures, such as gesture volume and finger-to-palm distance, which we employed to evaluate gestures performed by our participants. We also contribute a set of guidelines for practitioners interested in designing free-hand gestures for interactive TV scenarios involving similar gesture acquisition technology. We release our dataset consisting in 378 Leap Motion gestures described by fingertips position, direction, and velocity coordinates to foster further studies in the community. This first exploration of viewers’ preferences for fine-grained resolution free-hand gestures for TV control represents one more step toward designing low-effort gesture interfaces for lean-back interaction with the TV set.

Appendix: Complete set of gestures collected during the elicitation experiment

No.	Referent	Participants’ gesture proposals
1	Open	Open palm, wave hand, move hand up, move fist up followed by wrist rotation, move hand down, move palm away from the body, thumbs-up
2	Close	Close palm, wave hand, close hand into pinch, move hand down, fist moves up and down twice, move fist up followed by wrist rotation, hand performs the “go away” cultural gesture, draw “X”, perform click in mid-air with the index finger
3	Next	Move hand from right to left and Move hand from left to right, thumbs-up while move hand to the right, “OK” sign with the thumb, moving the index finger from left to right and then click in mid-air. Please note the two options in terms of movement direction (left-to-right and right-to-left) that correspond to two distinct metaphors: moving the viewing window (as it happens with scrolling actions and traditional GUI) and moving the items themselves [37]
4	Previous	Move hand from left to right and Move hand from right to left, thumbs-up while move hand to the left, perform double click in mid-air with the index finger, draw circle clockwise. Please note the two options in terms of movement direction (left-to-right and right-to-left) that correspond to moving the viewing window (as it happens with the scrolling action and traditional GUI) or the items themselves [37]
5	Volume up	Move hand up, hand in pinch pose expanding fingers, thumbs-up with hand moving to the right, thumbs-up with hand moving up twice, rotate imaginary button to the right, draw the “+” symbol in mid-air, open hand from thumb-index pinch, draw triangle, open palm, draw circle clockwise, hand performs the “go away” cultural gesture
6	Volume down	Move hand down, hand changes pose from open palm to index-thumb pinch, thumbs-up with hand moving to the left, thumbs-up with hand moving down twice, rotate imaginary button to the left, move hand from left to right, draw triangle, hand closing fingers into pinch, draw circle counterclockwise, hand performs the “come here” cultural gesture
7	Volume mute	Close fingers into pinch, fist followed by extending the little finger, open palm, thumbs-down to thumbs-up, draw letter “X”, close fist, open palm to index-thumb pinch, pinch with the thumb and little finger, open palm facing down while hand moves from left to right, draw crossed zero, draw circle counterclockwise, move hand down, move hand from left to right, move hand from right to left, move palm away from the body
8	Open menu	Draw letter “M”, draw small “o”, small “m”, rotate wrist, move hand up, open fingers from the thumb-index pinch, perform click in mid-air with the index finger, hand in pinch pose opens fingers, wave hand, move hand down
9	Hide menu	Hand wave, closed fist opens index and little finger, close fist, open hand rotates, hand changes from open palm to pinch with the thumb and index finger, move hand up, hand performs the “go away” cultural gesture, rotate palm from left to right, hand performs the “come here” cultural gesture
10	Help	Draw symbol “?” or letter “H” in mid-air, hand performs the “come here” cultural gesture, hand performs the “peace” sign, wave fingers, finger snapping, move hand toward the body, wave hand, pinch and wave
11	Yes	Thumbs-up, draw the “check” sign, three fingers down, draw letter “Y” in mid-air, thumbs-up rotated 90 degrees to the right, perform a click in mid-air with the index finger, open palm, pinch followed by thumbs-up
12	No	Hand wave, close fist, three fingers down, move hand up, thumbs-up rotate from right to up, draw letter “X” in mid-air, move hand down
13	Go to favorite channel	Show index finger, thumbs-up, perform the “come here” cultural gesture, pinch between the thumb and little finger, move index finger down and up, draw the “check” sign, open palm, fingers snapping, draw star, thumbs-up rotated to right, move hand down and back then and forward and down
14	Go to 2nd favorite channel	Show two fingers (index and middle), thumbs-up rotated 90 degrees to the right, pinch with the thumb and the ring finger, two fingers up, “peace” sign left to right, fingers snapping twice, draw digit “2” in mid-air, two fingers up followed by thumbs-up, move hand forward two times
15	Go to random channel	Palm facing down rotates to palm up, draw circle clockwise, draw circle anticlockwise, draw circle anticlockwise three times, close fingers, open palm rotates down to up, wave with fingers, perform a click in mid-air with the index finger, open palm, draw letter “R”, wave hand, draw the universal quantifier symbol “$\forall$”
16	Go to channel #7	Draw digit “7” in mid-air, various finger configurations to indicate number “7”
17	Go to channel #27	Draw number “27” in mid-air, various finger configurations to indicate number “27”
18	TV Guide	Draw letter “G” in mid-air, move hand up, thumbs-up, draw letter “M”, move palm in front of the body, move hand down, move hand from left to right, show the index finger, pinch followed by wave, open palm
19	Show channels list	Draw square in mid-air, four fingers move down, open palm, draw letters “C” and “L”, move hand down, hand performs the “come here” cultural gesture, wave hand, move hand up, pinch followed by wave, perform double click in mid-air with the index finger, move hand toward the body, draw letter “L” in mid-ar, draw circle anticlockwise
20	Last channel	Draw circle anti-clockwise, index and middle fingers move to the left, fingers snapping, two fingers up, thumbs-up moving back, hand rotates an imaginary button to the left, wave hand
21	Open browser	Draw symbol “@”, move palm down twice, close fist, draw letter “W” three times as in “WWW”, hand performs the “go away” cultural gesture, pinch twice, pinch followed by thumbs-up, move index finger from left to right, move hand from right to left, move hand down, move palm in front of the body, show three fingers