Journal on Multimodal User Interfaces

, Volume 9, Issue 2, pp 141–151 | Cite as

Low complexity head tracking on portable android devices for real time message composition

  • Laura MontaniniEmail author
  • Enea Cippitelli
  • Ennio Gambi
  • Susanna Spinsante
Original Paper


For the people who are totally or partially unable to move or control their limbs and cannot rely on verbal communication, it is very important to obtain an interface capable of interpreting their limited voluntary movements, in order to allow communications with friends, relatives and care providers, or to send commands to a system. This paper presents a real time software application for disabled subjects, suffering from both motor and speech impairments, that provides message composition and speech synthesis functionalities based on face detection and head tracking. The proposed application runs on portable devices equipped with Android Operating System, and relies upon the O.S.’s native computer vision primitives, without resorting to any external software library. This way, the available camera sensors are exploited, and the device computational requirements accomplished. Experimental results show the effectiveness of the application in recognizing the user’s movements, and the reliability of the message composition and speech synthesis functionalities.


Face detection Head tracking Android O.S. Human-system interface 

Supplementary material



  1. 1.
    Mads B, Gimpel G, Hedman J (2009) The user experience of smart phones: a consumption values approach. In: Proc. 8th Global Mobility Roundtable, GMR, CairoGoogle Scholar
  2. 2.
    World Health Organization (2009) Dept. of Violence and Injury Prevention, Global status report on road safety: time for action, World Health OrganizationGoogle Scholar
  3. 3.
    Wood E, Willoughby T, Rushing A, Bechtel L, Gilbert J (2005) Use of computer input devices by older adults. J Appl Gerontol 24(5):419–438CrossRefGoogle Scholar
  4. 4.
    Spinsante S, Gambi E (2012) Remote health monitoring by OSGi technology and digital TV integration. IEEE Trans Consum Electron 58(4):1434–1441CrossRefGoogle Scholar
  5. 5.
    Mertens A, Koch-Korfges D, Jochems N, Schlick CM (2010) Touchscreen-based input technique for people with intention tremor. In: Proc. of 3rd Conference on Human System Interactions (HSI). pp 236–240Google Scholar
  6. 6.
    Spinsante S, Gambi E (2012) Home automation systems control by head tracking in AAL applications. In: Proc. of IEEE 1st ESTEL Conference, Rome, April 2012Google Scholar
  7. 7.
    Ren J, Rahman M, Kehtarnavaz N, Estevez L (2010) Real-time head pose estimation on mobile platforms. J Syst, Cybern Inf 8(3):56–62Google Scholar
  8. 8.
    Lupu RG, Ungureanu F, Bozomitu RG (2012) Mobile embedded system for human computer communication in assistive technology. In: Proc. IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), 2012, pp 209–212, Aug 30 2012–SeptGoogle Scholar
  9. 9.
    Takahashi K, Mitsukura Y (2013) Head pose tracking system using a mobile device. In: Proc. 2013 IEEE Int. Symposium on Robot and Human Interactive Communication, pp 461–466, 26–29 Aug 2013Google Scholar
  10. 10.
    Montanini L, Cippitelli E, Gambi E, Spinsante S (2014) Real time message composition through head movements on portable Android devices. In: Proc. IEEE 2014 Int. Conf. on Consumer Electronics, pp 526–527, Jan 10–13, 2014, Las Vegas, USAGoogle Scholar
  11. 11.
    Haraikawa T, Oyamada A, Ito H, Oikawa S, Fukui Y (2014) A cost-effective solution for realizing talking appliances for the visually impaired. In: 2014 IEEE International Conference on Consumer Electronics (ICCE), pp 317–318, 10–13 Jan 2014Google Scholar
  12. 12.
    Ivanov R (2014) Blind-environment interaction through voice augmented objects. J Multimodal User Interfaces 8(4):345–365CrossRefGoogle Scholar
  13. 13.
    Batliner A, Hacker C, Nth E (2008) To talk or not to talk with a computer. J Multimodal User Interfaces 2(3–4):171–186CrossRefGoogle Scholar
  14. 14.
    Chandramouli C, Agarwal V (2009) Speech Recognition based Computer Keyboard Replacement for the Quadriplegics, Paraplegics, Paralytics and Amputees, IEEE International Workshop on Medical Measurements and Applications, pp 241–245, 29–30 May 2009Google Scholar
  15. 15.
    Kathirvelan J, Anilkumar R, Alex ZC, Fazul A (2012) Development of low cost automatic wheelchair controlled by oral commands using standalone controlling system, IEEE International Conference on Computational Intelligence & Computing Research (ICCIC), pp 1–4, 18–20 Dec 2012Google Scholar
  16. 16.
    McFarland DJ, Wolpaw JR (2011) Brain-computer interfaces for communication and control. Commun ACM 54(5):60–66CrossRefGoogle Scholar
  17. 17.
    Donegan M, Cotmore S, Holmqvist E, Buchholz M, Lundalv M, Pasian V, Farinetti L, Corno F (2009) Deliverable 3.6: Final User Trials Report. Communication by Gaze Interaction (COGAIN) IST-2003-511598.
  18. 18.
    Beukelman DR, Yorkston KM, Reichle J (2000) Augmentative and alternative communication for adults with acquired neurologic disorders. Paul H Brookes, Baltimore, MDGoogle Scholar
  19. 19.
    Kumar N, Kohlbecher S, Schneider E (2009) A novel approach to video-based pupil tracking. In: Proc. IEEE International Conference on Systems, Man and Cybernetics, 2009. pp 1255–1262, 11–14 Oct 2009Google Scholar
  20. 20.
    Rantanen V, Vanhala T, Tuisku O, Niemenlehto P-H, Verho J, Surakka V, Juhola M, Lekkala J (2011) A wearable, wireless gaze tracker with integrated selection command source for human-computer interaction, IEEE Trans. On Inf Tech Biomed 15(5):795–801CrossRefGoogle Scholar
  21. 21.
    Lupu RG, Ungureanu F, Siriteanu V (2013) Eye tracking mouse for human computer interaction. In: Proc. 2013 E-Health and Bioengineering Conference, pp 1–4, 21–23 Nov 2013Google Scholar
  22. 22.
    La Cascia M, Sclaroff S (1999) Fast, reliable head tracking under varying illumination. In: Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol 1Google Scholar
  23. 23.
    Malassiotis S, Strintzis MG (2003) Real-time head tracking and 3D pose estimation from range data. In: Proc. of International Conference on Image Processing, vol 2, pp 859–862Google Scholar
  24. 24.
    Yun F, Huang TS (2007) hMouse: head tracking driven virtual computer mouse. In: Proc. of IEEE Workshop on Applications of Computer VisionGoogle Scholar
  25. 25.
    Morency LP, Sidner C, Lee C, Darrell T (2005) Contextual recognition of head gestures. In: Proceedings of the International Conference on Multimodal Interactions, Oct 46, 2005, Trento, ItalyGoogle Scholar
  26. 26.
    Song Y, Luo Y, Lin J (2011) Detection of movements of head and mouth to provide computer access for disabled, 2011 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), pp 223–226, 11–13 Nov 2011Google Scholar
  27. 27.
    Bastos-Filho T, Ferreira A, Cavalieri D, Silva R, Muller S, Perez E (2013) Multi-modal interface for communication operated by eye blinks, eye movements, head movements, blowing/sucking and brain waves, 2013 ISSNIP Biosignals and Biorobotics Conference (BRC), pp 1–6, 18–20 Feb 2013Google Scholar
  28. 28.
    Morency LP, Rahimi A, Darrell T (2003) Adaptive view-based appearance model. In: Proceedings IEEE Conf. on Computer Vision and Pattern RecognitionGoogle Scholar
  29. 29.
    Face Tracking on Android O.S. demo video clip, available at:

Copyright information

© OpenInterface Association 2015

Authors and Affiliations

  • Laura Montanini
    • 1
    Email author
  • Enea Cippitelli
    • 1
  • Ennio Gambi
    • 1
  • Susanna Spinsante
    • 1
  1. 1.Dipartimento di Ingegneria dell’InformazioneUniversità Politecnica delle MarcheAnconaItaly

Personalised recommendations