Advertisement

Universal Access in the Information Society

, Volume 18, Issue 1, pp 107–126 | Cite as

Toward the gestural interface: comparative analysis between touch user interfaces versus gesture-based user interfaces on mobile devices

  • Gonzalo Pomboza-Junez
  • Juan A. Holgado-TerrizaEmail author
  • Nuria Medina-Medina
Long Paper
  • 246 Downloads

Abstract

Human–computer interaction is transformed as well as the technology used to achieve it. This paper discusses the ease of use of a classic touch screen user interface (TUI) in contrast to a gesture-based user interface (GBUI) applied to a mobile device. Two basic positions are used for comparison: “normal” when individual is standing and “special” when individual is lying down. The first represents a common position for healthy individuals, while the second is used to represent sick, disabled or comfortable individuals. Two software applications have been developed in order to set an interaction test for TUI and GBUI, which was performed by 25 users. The interaction tests show that GBUI has a very reasonable average accuracy of 88.5% in special position and 89.8% in normal position, while TUI presents an average accuracy of 96.5% in special position and 97.4% in normal position. Therefore, GBUI could be a complementary form of interaction to TUI that can be useful to any user, especially for cases of illness or simple comfort of the user. As study case, we have developed a software solution called SICLLE (emergency call control interactive system) that manages phone calls in a smartphone and applies the concept of gestural control tree to set a gestural navigation through a gestural language from a gestural vocabulary. An architecture for mobile devices supporting GBUI has been proposed and used in the implementation of SICLLE. SICLLE allows the personalization of the gestures, which optimizes the gestural dictionary and achieves an easier learning and use. In addition, SICLLE incorporates a hearing guide which replaces the typical visual control, reaching total acceptance in the experiment conducted with real users.

Keywords

Natural user interface Touch user interface Gesture interface Hand gesture recognition Gesture control tree Mobile devices 

Notes

Acknowledgements

This work was supported by the Secretariat of Higher Education, Science, Technology and Innovation SENESCYT of Ecuador, and the Spanish Ministry of Science and Innovation, as part of the DISPERSA Project (TIN2015-67149-C3-3-R), and by the Andalusia Research Program under the project P11-TIC-7486 co-financed by FEDER (European Regional Development Fund ERDF).

References

  1. 1.
    Abid, M.R., Shi, F., Petriu, E.M.: Dynamic hand gesture recognition from bag-of-features and local part model. In: Proceedings of 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE 2012), IEEE, pp. 78–82 (2012). doi: 10.1109/HAVE.2012.6374443
  2. 2.
    AdnanIbraheem, N., Zaman Khan, R.: Survey on various gesture recognition technologies and techniques. Int. J. Comput. Appl. 50(7), 38–44 (2012). doi: 10.5120/7786-0883, http://research.ijcaonline.org/volume50/number7/pxc3880883.pdf
  3. 3.
    Al-Timemy, A.H., Bugmann, G., Escudero, J., Outram, N.: Classification of finger movements for the dexterous hand prosthesis control with surface electromyography. IEEE J. Biomed. Health Inform. 17(3), 608–618 (2013). doi: 10.1109/JBHI.2013.2249590 Google Scholar
  4. 4.
    Blazquez Cano, M., Perry, P., Ashman, R., Waite, K.: The influence of image interactivity upon user engagement when using mobile touch screens. Comput. Hum. Behav. 1–7 (2017). doi: 10.1016/j.chb.2017.03.042, http://linkinghub.elsevier.com/retrieve/pii/S0747563217301991
  5. 5.
    Buchner, H., Petersen, E., Eger, M., Rostalski, P.: Convolutive blind source separation on surface EMG signals for respiratory diagnostics and medical ventilation control. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp. 3626–3629 (2016). doi: 10.1109/EMBC.2016.7591513, http://ieeexplore.ieee.org/document/7591513/
  6. 6.
    Card, S.K., Newell, A., Moran, T.P.: The Psychology of Human–Computer Interaction. Lawrence Erlbaum Associates Inc., Hillsdale, NJ (1983)Google Scholar
  7. 7.
    Chen, C., Jafari, R., Kehtarnavaz, N.: Improving human action recognition using fusion of depth camera and inertial sensors. IEEE Trans. Hum. Mach. Syst. 45(1), 51–61 (2015). doi: 10.1109/THMS.2014.2362520, http://ieeexplore.ieee.org/document/6934998/
  8. 8.
    Chen, X., Wang, Z.J.: Pattern recognition of number gestures based on a wireless surface EMG system. Biomed. Signal Process. Control 8(2), 184–192 (2013). doi: 10.1016/j.bspc.2012.08.005 Google Scholar
  9. 9.
    Chittaro, L.: Visualizing information on mobile devices. Computer 39(3), 40–45 (2006). doi: 10.1109/MC.2006.109, http://ieeexplore.ieee.org/document/1607948/
  10. 10.
    de Waard, D., Westerhuis, F., Joling, D., Weiland, S., Stadtbäumer, R., Kaltofen, L.: Visual map and instruction-based bicycle navigation: a comparison of effects on behaviour. Ergonomics 0139(April), 1–14 (2017). doi: 10.1080/00140139.2017.1282628 Google Scholar
  11. 11.
    Dinh, D.L., Kim, J.T., Kim, T.S.: Hand gesture recognition and interface via a depth imaging sensor for smart home appliances. Energy Procedia 62(62), 576–582 (2014). doi: 10.1016/j.egypro.2014.12.419 Google Scholar
  12. 12.
    Erden, F., Çetin, A.E.: Hand gesture based remote control system using infrared sensors and a camera. IEEE Trans. Consum. Electron. 60(4), 675–680 (2014). doi: 10.1109/TCE.2014.7027342 Google Scholar
  13. 13.
    Fernandez, R.A.S., Sanchez-Lopez, J.L., Sampedro, C., Bavle, H., Molina, M., Campoy, P.: Natural user interfaces for human-drone multi-modal interaction. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), IEEE, pp. 1013–1022 (2016). doi: 10.1109/ICUAS.2016.7502665
  14. 14.
    Fitts, P.M.: The amplitude of movement. J. Exp. Psychol. 47, 381–391 (1954)Google Scholar
  15. 15.
    Flores-Abad, A., Ma, O., Pham, K., Ulrich, S.: A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci. 68, 1–26 (2014). doi: 10.1016/j.paerosci.2014.03.002, http://linkinghub.elsevier.com/retrieve/pii/S0376042114000347
  16. 16.
    Flowers, K.A., Robertson, C.: The effect of Parkinson’s disease on the ability to maintain a mental set. J. Neurol. Neurosurg. Psychiatry 48(6), 517–529 (1985). doi: 10.1136/jnnp.48.6.517 Google Scholar
  17. 17.
    Francese, R., Passero, I., Tortora, G.: Wiimote and Kinect. In: Proceedings of the International Working Conference on Advanced Visual Interfaces—AVI ’12, ACM Press, New York, New York, USA, pp 116–123 (2012). doi: 10.1145/2254556.2254580
  18. 18.
    Garawi, S., Istepanian, R., Abu-Rgheff, M.: 3G wireless communications for mobile robotic tele-ultrasonography systems. IEEE Commun. Mag. 44(4), 91–96 (2006). doi: 10.1109/MCOM.2006.1632654 Google Scholar
  19. 19.
    Gokgoz, E., Subasi, A.: Effect of multiscale PCA de-noising on EMG signal classification for diagnosis of neuromuscular disorders. J. Med. Syst. 38(4), 31–38 (2014). doi: 10.1007/s10916-014-0031-3 Google Scholar
  20. 20.
    Guerrero, F.N., Spinelli, E.M., Haberman, M.A.: Analysis and simple circuit design of double differential EMG active electrode. IEEE Trans. Biomed. Circuits Syst. 10(3), 787–795 (2016). doi: 10.1109/TBCAS.2015.2492944 Google Scholar
  21. 21.
    Hargrove, L., Englehart, K., Hudgins, B.: A comparison of surface and intramuscular myoelectric signal classification. IEEE Trans. Biomed. Eng. 54(5), 847–853 (2007). doi: 10.1109/TBME.2006.889192, http://ieeexplore.ieee.org/document/4154997/
  22. 22.
    Hasan, H., Abdul-Kareem, S.: Static hand gesture recognition using neural networks. Artif. Intell. Rev. 41(2), 147–181 (2014). doi: 10.1007/s10462-011-9303-1 Google Scholar
  23. 23.
    Heo, H., Lee, E.C., Park, K.R., Kim, C.J., Whang, M.: A realistic game system using multi-modal user interfaces. IEEE Trans. Consum. Electron. 56(3), 1364–1372 (2010). doi: 10.1109/TCE.2010.5606271 Google Scholar
  24. 24.
    Hooper, C.J., Dix, A.: Web science and human–computer interaction. Interactions 20(3), 52–57 (2013). doi: 10.1145/2451856.2451868 Google Scholar
  25. 25.
    Hug, F.: Can muscle coordination be precisely studied by surface electromyography? J. Electromyogr. Kinesiol. 21(1), 1–12 (2011). doi: 10.1016/j.jelekin.2010.08.009 Google Scholar
  26. 26.
    Ismirle, J., OBara, I., Swierenga, S.J., Jackson, J.E.: Touchscreen voting interface design for persons with disabilities: insights from usability evaluation of mobile voting prototype. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1), 780–784 (2016). doi: 10.1177/1541931213601179
  27. 27.
    Jaimes, A., Sebe, N.: Multimodal humancomputer interaction: a survey. Comput. Vis. Image Underst. 108(1–2), 116–134 (2007). doi: 10.1016/j.cviu.2006.10.019 Google Scholar
  28. 28.
    Jin, Z.X, Plocher, T., Kiff, L.: Touch screen user interfaces for older adults: button size and spacing. In: Stephanidis, C. (ed.) Universal Access in Human Computer Interaction. Coping with Diversity, Lecture Notes in Computer Science, vol. 4554, Springer, Berlin, pp. 933–941, (2007). doi: 10.1007/978-3-540-73279-2_104, http://www.springerlink.com/index/10.1007/978-3-540-73279-2, http://link.springer.com/10.1007/978-3-540-73279-2
  29. 29.
    Jung, Y.: What a smartphone is to me: understanding user values in using smartphones. Inf. Syst. J. 24(4), 299–321 (2014). doi: 10.1111/isj.12031
  30. 30.
    Kane, S.K, Jayant, C., Wobbrock, J.O, Ladner, R.E.: Freedom to roam: a study of mobile device adoption and accessibility for people with visual and motor disabilities. In: Proceeding of the Eleventh International ACM SIGACCESS Conference on Computers and Accessibility—ASSETS ’09, ACM Press, New York, NY, USA, pp. 115–122 (2009). doi: 10.1145/1639642.1639663
  31. 31.
    Kaushik, D.M., Jain, R.: Gesture based interaction NUI: an overview. Int. J. Eng. Trends Technol. 9(12), 633–636 (2014). doi: 10.14445/22315381/IJETT-V9P319, https://arxiv.org/abs/1404.2364
  32. 32.
    Kim, G.J.: Human-Computer Interaction: Fundamentals and Practice. CRC Press, Boca Raton (2015)Google Scholar
  33. 33.
    Klboz, N.Ç., Güdükbay, U.: A hand gesture recognition technique for human–computer interaction. J. Vis. Commun. Image Represent. 28, 97–104 (2015). doi: 10.1016/j.jvcir.2015.01.015, http://linkinghub.elsevier.com/retrieve/pii/S10473203
  34. 34.
    Kühnel, C., Westermann, T., Hemmert, F., Kratz, S., Müller, A., Möller, S.: I’m home: defining and evaluating a gesture set for smart-home control. Int. J. Hum. Comput. Stud. 69(11), 693–704 (2011). doi: 10.1016/j.ijhcs.2011.04.005, http://linkinghub.elsevier.com/retrieve/pii/S1071581911000668
  35. 35.
    Lara, O.D., Labrador, M.A.: A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 15(3), 1192–1209 (2013). doi: 10.1109/SURV.2012.110112.00192 Google Scholar
  36. 36.
    LaViola, J.J.: 3D gestural interaction: the state of the field. ISRN Artif. Intell. 2013(2), 1–18 (2013). doi: 10.1155/2013/514641 Google Scholar
  37. 37.
    Lee, M., Hong, Y., Lee, S., Won, J., Yang, J., Park, S., Chang, K.T., Hong, Y.: The effects of smartphone use on upper extremity muscle activity and pain threshold. J. Phys. Ther. Sci. 27(6), 1743–1745 (2015). doi: 10.1589/jpts.27.1743, https://www.jstage.jst.go.jp/article/jpts/27/6/27_jpts-2015-015/_article
  38. 38.
    Leitão, R.A.: Creating mobile gesture-based interaction design patterns for older adults: a study of tap and swipe gestures with Portuguese seniors. Ph.D. thesis, Universidade do Porto (2012) http://repositorio-aberto.up.pt/handle/10216/68413
  39. 39.
    Li, G., Li, Y., Zhang, Z., Geng, Y., Zhou, R.: Selection of sampling rate for EMG pattern recognition based prosthesis control. In: 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology, IEEE, vol. 2010, pp. 5058–5061 (2010). doi: 10.1109/IEMBS.2010.5626224
  40. 40.
    Li, C., Ma, H., Yang, C., Fu, M. Teleoperation of a virtual iCub robot under framework of parallel system via hand gesture recognition. In: 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), IEEE, pp 1469–1474 (2014). doi: 10.1109/FUZZ-IEEE.2014.6891887, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6891887
  41. 41.
    Lv, Z., Feng, S., Feng, L., Li, H.: Extending touch-less interaction on vision based wearable device. In: 2015 IEEE Virtual Reality (VR), IEEE, pp 231–232 (2015). doi: 10.1109/VR.2015.7223380, http://ieeexplore.ieee.org/document/7223380/, https://arxiv.org/abs/1504.01025
  42. 42.
    MacKenzie, I.S.: Fitts’ law as a research and design tool in human–computer interaction. Hum. Comput. Interact. 7(1):91–139 (1992). doi: 10.1207/s15327051hci07013, http://portal.acm.org/citation.cfm?id=1461857
  43. 43.
    Mehler, A., Lücking, A., Abrami, G.: WikiNect: image schemata as a basis of gestural writing for kinetic museum wikis. Univers. Access Inf. Soc. 14(3), 333–349 (2015). doi: 10.1007/s10209-014-0386-8
  44. 44.
    Merletti, R., Botter, A., Troiano, A., Merlo, E., Minetto, M.A.: Technology and instrumentation for detection and conditioning of the surface electromyographic signal: state of the art. Clin. Biomech. 24(2), 122–134 (2009). doi: 10.1016/j.clinbiomech.2008.08.006
  45. 45.
    Min, B.W: Improvement of mobile U-health services system. Commun. Comput. Inf. Sci. 262 CCIS(PART 1), 44–51 (2011). doi: 10.1007/978-3-642-27204-2_6
  46. 46.
    Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 37(3), 311–324 (2007). doi: 10.1109/TSMCC.2007.893280, http://ieeexplore.ieee.org/document/4154947/
  47. 47.
    Nguyen, Q., Premaratne, P.: Consumer electronics control system based on hand gesture moment invariants. IET Comput. Vis. 1(1), 35–41 (2007). doi: 10.1049/iet-cvi:20060198, http://ieeexplore.ieee.org/document/4159604/?arnumber=4159604
  48. 48.
    Nielsen, J.: Noncommand user interfaces. Commun. ACM 36(4), 83–99 (1993). doi: 10.1145/255950.153582 MathSciNetGoogle Scholar
  49. 49.
    Nielsen, M., Störring, M., Moeslund, T.B., Granum, E.: A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. In: Gesture-Based Communication in Human–Computer Interaction, pp. 409–420 (2004). doi: 10.1007/978-3-540-24598-8 38
  50. 50.
    Ohn-Bar, E., Trivedi, M.M.: Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans. Intell. Transp. Syst. 15(6), 2368–2377 (2014). doi: 10.1109/TITS.2014.2337331, http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6883176
  51. 51.
    Page, T.: Touchscreen and perceived usability: a comparison of attitudes between older and youger mobile device. I-Manag. J. Mob. Appl. Techonol. 3(1), 1–16 (2016). https://search.proquest.com/docview/1853319087?accountid=14542, http://hdl.handle.net/11025/1847
  52. 52.
    Peters, A., Morrison, J.H.: Cerebral Cortex: Neurodegenerative and Age-Related Changes in Structure and Function of Cerebral Cortex, vol. 14. Springer, Berlin (2012)Google Scholar
  53. 53.
    Phinyomark, A., Phukpattaranont, P., Limsakul, C.: Feature reduction and selection for EMG signal classification. Expert Syst. Appl. 39(8), 7420–7431 (2012). doi: 10.1016/j.eswa.2012.01.102 Google Scholar
  54. 54.
    Pomboza-Junez, G., Holgado-Terriza, J.A.: Control of home devices based on hand gestures. In: 2015 IEEE 5th International Conference on Consumer Electronics—Berlin (ICCE-Berlin), IEEE, pp 510-514 (2015). DOI: 10.1109/ICCE-Berlin.2015.7391325
  55. 55.
    Pomboza-Junez, G., Holgado-Terriza, J.A.: Hand gesture recognition based on sEMG signals using Support Vector Machines. In: 2016 IEEE 6th International Conference on Consumer Electronics—Berlin (ICCE-Berlin), IEEE, pp. 174–178 (2016). doi: 10.1109/ICCE-Berlin.2016.7684748
  56. 56.
    Premaratne, P., Ajaz, S., Premaratne, M.: Hand gesture tracking and recognition system using Lucas-Kanade algorithms for control of consumer electronics. Neurocomputing 116, 242–249 (2013). doi: 10.1016/j.neucom.2011.11.039 Google Scholar
  57. 57.
    Prieto De Lope, R.P., Medina Medina, N.: A comprehensive taxonomy for serious games. J. Educ. Comput. Res. (2016). doi: 10.1177/0735633116681301 Google Scholar
  58. 58.
    Quigley, M., Goodrich, M., Beard, R.: Semi-autonomous human–UAV interfaces for fixed-wing mini-UAVs. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566), IEEE, vol. 3, pp. 2457–2462 (2004). doi: 10.1109/IROS.2004.1389777
  59. 59.
    Raez, M.B.I., Hussain, M.S., Mohd-Yasin, F., Reaz, M., Hussain, M.S., Mohd-Yasin, F.: Techniques of EMG signal analysis: detection, processing, classification and applications. Biol. Proced. Online 8(1), 11–35 (2006). doi: 10.1251/bpo115 Google Scholar
  60. 60.
    Rainoldi, A., Melchiorri, G., Caruso, I.: A method for positioning electrodes during surface EMG recordings in lower limb muscles. J. Neurosci. Methods 134(1), 37–43 (2004). doi: 10.1016/j.jneumeth.2003.10.014 Google Scholar
  61. 61.
    Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015). doi: 10.1007/s10462-012-9356-9 Google Scholar
  62. 62.
    Rempel, D., Camilleri, M.J., Lee, D.L.: The design of hand gestures for human-computer interaction: Lessons from sign language interpreters. Int. J. Hum. Comput. Stud. 72(10–11), 728–735 (2014). doi: 10.1016/j.ijhcs.2014.05.003 Google Scholar
  63. 63.
    Rezende, A., Schneider, G., Prakash, P.: Method and apparatus for customizing a display screen of a user interface. https://www.google.com/patents/US9304668, US Patent 9,304,668 (2016)
  64. 64.
    Robertson, C., Flowers, K.A.: Motor set in Parkinson’s disease. J. Neurol. Neurosurg. Psychiatry 53(7), 583–592 (1990). doi: 10.1136/jnnp.53.7.583 Google Scholar
  65. 65.
    Sanchez-Nielsen, E., Hernandez-Tejera, M., Anton-Canalís, L. (2004) Hand gesture recognition for human–machine interaction. J. WSCG 12(1-3), 395–402, http://hdl.handle.net/11025/1847
  66. 66.
    Sanna, A., Lamberti, F., Paravati, G., Manuri, F.: A Kinect-based natural interface for quadrotor control. Entertain. Comput. 4(3), 179–186 (2013). doi: 10.1016/j.entcom.2013.01.001 Google Scholar
  67. 67.
    Schaller, R.: Moore’s law: past, present and future. IEEE Spectr. 34(6), 52–59 (1997). doi: 10.1109/6.591665, http://ieeexplore.ieee.org/document/591665/
  68. 68.
    Stergiopoulou, E., Papamarkos, N.: Hand gesture recognition using a neural network shape fitting technique. Eng. Appl. Artif. Intell. 22(8), 1141–1158 (2009). doi: 10.1016/j.engappai.2009.03.008, http://linkinghub.elsevier.com/retrieve/pii/S0952197609000694
  69. 69.
    Turk, M.: Multimodal human–computer interaction. In: Real-Time Vision for Human–Computer Interaction, Springer-Verlag, New York, pp. 269–283 (2005). doi: 10.1007/0-387-27890-7
  70. 70.
    Turk, M.: Multimodal interaction: a review. Pattern Recognit. Lett. 36(1), 189–195 (2014). doi: 10.1016/j.patrec.2013.07.003 MathSciNetGoogle Scholar
  71. 71.
    Von Neumann, J., Kurzweil, R.: The Computer and the Brain. Yale University Press, New Haven (2012)Google Scholar
  72. 72.
    Walker, F.O.: Huntington’s disease. The Lancet 369(9557), 218–228 (2007). doi: 10.1016/S0140-6736(07)60111-1. 1111.6189v1Google Scholar
  73. 73.
    Wang, D., Xiang, Z., Fesenmaier, D.R.: Smartphone use in everyday life and travel. J. Travel Res. 55(1), 52–63 (2016). doi: 10.1177/0047287514535847 Google Scholar
  74. 74.
    Wheeler, K., Chang, M., Knuth, K.: Gesture-based control and EMG decomposition. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 36(4), 503–514 (2006). doi: 10.1109/TSMCC.2006.875418 Google Scholar
  75. 75.
    Yang, S.W., Lin, C.S., Lin, S.K., Lee, C.H.: Design of virtual keyboard using blink control method for the severely disabled. Comput. Methods Programs Biomed. 111(2), 410–418 (2013). doi: 10.1016/j.cmpb.2013.04.012 Google Scholar
  76. 76.
    Zhou, J., Rau, P.L.P., Salvendy, G.: Older adults’ use of smart phones: an investigation of the factors influencing the acceptance of new functions. Behav. Inf. Technol. 33(6), 552–560 (2014). doi: 10.1080/0144929X.2013.780637 Google Scholar
  77. 77.
    Zimmerman, T.G., Lanier, J., Blanchard, C., Bryson, S., Harvill, Y.: A hand gesture interface device. ACM SIGCHI Bull. 17(SI), 189–192 (1986). doi: 10.1145/30851.275628 Google Scholar

Copyright information

© Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  1. 1.National University of ChimborazoRiobambaEcuador
  2. 2.University of GranadaGranadaSpain

Personalised recommendations