Designer Led Computational Approach to Generate Mappings for Devices with Low Gestural Resolution

  • Roberto Montano-MurilloEmail author
  • Teng HanEmail author
  • Pourang IraniEmail author
  • Diego Martinez-PlasenciaEmail author
  • Sriram SubramanianEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11746)


We present an approach for the semi-automatic generation of gesture mappings for devices with low gestural resolution such as the Myo Armband, an off-the-shelf EMG capture device. As an exemplar interactive task, we use text-entry: a pervasive and highly complex interaction. We quantify data related to interaction combining systematic studies (i.e., error, speed, accuracy) and semi-structured workshops with experts (e.g., cognitive load, heuristics). We then formalize these factors in a mathematical model and use optimization algorithms (i.e. simulated annealing) to find an optimum gesture mapping. We demonstrated our method in a text-entry application (i.e., complex interactive dialogue) comparing our approach with other computationally determined mappings using naive cost functions. Our results showed that the designers mapping (with all factors weighted by designers) presented a good balance on performance in all factors involved (speed, accuracy, comfort, memorability, etc.), consistently performing better than purely computational mappings. The results indicate that our hybrid approach can yield better results than either pure user-driven methodologies or pure data-driven approaches, for our application context featuring a large solution space and complex high-level factors.


Gestural interaction Semi-automated interaction design Optimization Computational approaches 

Supplementary material

Supplementary material 1 (MOV 113219 kb)


  1. 1.
    Bi, X., Smith, B.A., Zhai, S.: Quasi-qwerty soft keyboard optimization. In: Proceedings of the SIGCHI Conference. ACM (2010)Google Scholar
  2. 2.
    Bi, X., Zhai, S.: IJQwerty: what difference does one key change make? gesture typing keyboard optimization bounded by one key position change from qwerty. In: Proceedings of the 2016 CHI Conference. ACM (2016)Google Scholar
  3. 3.
    Borg, G.: Psychophysical scaling with applications in physical work and the perception of exertion. Scand. J. Work Environ. Health 1, 55–58 (1990)CrossRefGoogle Scholar
  4. 4.
    Cadoz, C.: Les réalités virtuelles (1994)Google Scholar
  5. 5.
    Davey, B., Parker, K.R.: Requirements elicitation problems: a literature analysis. Issues Inform. Sci. Inf. Technol. 12, 71–82 (2015)CrossRefGoogle Scholar
  6. 6.
    Dutta, T.: Evaluation of the Kinect™ sensor for 3-D kinematic measurement in the workplace. Appl. Ergon. 43(4), 645–649 (2012)CrossRefGoogle Scholar
  7. 7.
    Fabiani, L., Burdea, G.C., Langrana, N.A., Gomez, D.: Human interface using the Rutgers Master II force feedback interface. In: VRAIS (1996)Google Scholar
  8. 8.
    Feit, A.M., Oulasvirta, A.: Pianotext: redesigning the piano keyboard for text entry. In: Proceedings of the 2014 Conference DIS. ACM (2014)Google Scholar
  9. 9.
    Gabbard, J.L., Hix, D., Swan, J.E.: User-centered design and evaluation of virtual environments. IEEE Comput. Graph. Appl. 19(6), 51–59 (1999)CrossRefGoogle Scholar
  10. 10.
    Gelain, M., Pini, M.S., Rossi, F., Venable, K.B., Walsh, T.: Elicitation strategies for soft constraint problems with missing preferences: properties, algorithms and experimental studies. Artif. Intell. 174(3), 270–294 (2010)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Gong, J., Haggerty, B., Tarasewich, P.: An enhanced multitap text entry method with predictive next-letter highlighting. In: CHI 2005. ACM (2005)Google Scholar
  12. 12.
    Grossman, T., Wigdor, D., Balakrishnan, R.: Multi-finger gestural interaction with 3D volumetric displays. In: Proceedings of the 17th Annual ACM UIST. ACM (2004)Google Scholar
  13. 13.
    Haldar, R., Mukhopadhyay, D.: Levenshtein distance technique in dictionary lookup methods: an improved approach. arXiv e-print (arXiv:1101.1232) (2011)
  14. 14.
    Kammer, D., Wojdziak, J., Keck, M., Groh, R., Taranko, S.: Towards a formalization of multi-touch gestures. In: ACM ISS, pp. 49–58. ACM (2010)Google Scholar
  15. 15.
    Kerber, F., Lessel, P., Krüger, A.: Same-side hand interactions with arm-placed devices using EMG. In: Proceedings of the ACM CHI, pp. 1367–1372. ACM, Seoul (2015)Google Scholar
  16. 16.
    Kessler, G.D., Hodges, L.F., Walker, N.: Evaluation of the CyberGlove as a whole-hand input device. ACM TOCHI 2(4), 263–283 (1995)CrossRefGoogle Scholar
  17. 17.
    Kim, D., et al.: Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In: Proceedings of ACM UIST. ACM (2012)Google Scholar
  18. 18.
    Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Klimt, B., Yang, Y.: The enron corpus: a new dataset for email classification research. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 217–226. Springer, Heidelberg (2004). Scholar
  20. 20.
    Knibbe, J., et al.: Extending interaction for smart watches: enabling bimanual around device control. In: CHI 2014. ACM (2014)Google Scholar
  21. 21.
    Lyons, J.: A branch of both mathematics and computer science, cryptography is the study and practice of obscuring information.
  22. 22.
    McIntosh, J., et al.: EMPress: practical hand gesture classification with wrist-mounted EMG and pressure sensing. In: Proceedings of the 2016 CHI Conference. ACM (2016)Google Scholar
  23. 23.
    McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press, Chicago (1992)Google Scholar
  24. 24.
    Pavlovych, A., Stuerzlinger, W.: Model for non-expert text entry speed on 12-button phone keypads. In: Proceedings of the SIGCHI Conference. ACM (2004)Google Scholar
  25. 25.
    Rekimoto, J.: SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In: Proceedings of the SIGCHI Conference. ACM (2002)Google Scholar
  26. 26.
    Rimé, B.: The elimination of visible behaviour from social interactions: effects on verbal, nonverbal and interpersonal variables. Eur. J. Soc. Psychol. 12(2), 113–129 (1982)CrossRefGoogle Scholar
  27. 27.
    Rimé, B., Schiaratura, L.: Gesture and Speech (1991)Google Scholar
  28. 28.
    Robertson, R.J., et al.: Concurrent validation of the OMNI perceived exertion scale for resistance exercise. Med. Sci. Sports Exerc. 35(2), 333–341 (2003)CrossRefGoogle Scholar
  29. 29.
    Sridhar, S., Feit, A.M., Theobalt, C., Oulasvirta, A.: Investigating the dexterity of multi-finger input for mid-air text entry. In: Proceedings of ACM CHI Conference (2015)Google Scholar
  30. 30.
    Sturman, D.J., Zeltzer, D.: A design method for “whole-hand” human-computer interaction. ACM Trans. Inf. Syst. (TOIS). 11(3), 219–238 (1993)CrossRefGoogle Scholar
  31. 31.
    Sturman, D.J., Zeltzer, D., Pieper, S.: Hands-on interaction with virtual environments. In: Proceedings of the 2nd Annual ACM SIGGRAPH UIST. ACM (1989)Google Scholar
  32. 32.
  33. 33.
    Weichert, F., Bachmann, D., Rudak, B., Fisseler, D.: Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5), 6380–6393 (2013)CrossRefGoogle Scholar
  34. 34.
    Weissmann, J., Salomon, R.: Gesture recognition for virtual reality applications using data gloves and neural networks. In: International Joint Conference on Neural Networks, 1999. IJCNN 1999. IEEE (1999)Google Scholar
  35. 35.
    Wu, M., Balakrishnan, R.: Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In: Proceedings of ACM UIST. ACM (2003)Google Scholar
  36. 36.
    Zhang, Y., Zhou, J., Laput, G., Harrison, C.: SkinTrack: Using the body as an electrical waveguide for continuous finger tracking on the skin. In: Proceedings of the 2016 CHI Conference. ACM (2016)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  1. 1.Department of Informatics/Interact LabUniversity of SussexBrightonUK
  2. 2.Department of Computer ScienceUniversity of ManitobaWinnipegCanada

Personalised recommendations