Advertisement

Journal on Multimodal User Interfaces

, Volume 12, Issue 1, pp 55–65 | Cite as

Real-time eye blink and wink detection for object selection in HCI systems

Original Paper
  • 177 Downloads

Abstract

This paper presents an approach for real-time detection of three types of eye blinks: eye blink (blinking both eyes simultaneously), left and right winks. The process of blink detection has been divided into four parts viz. face localization in facial images acquired through a video camera, eye pair localization, pixels’ motion analysis using optical flow technique, and classification of eye blinks. Blink detection has been performed using a video camera and MATLAB software with image processing and computer vision toolbox. The algorithm takes about 60 ms time for processing a frame and 250 ms time for confirmation and classification of the detected blink. An experiment was conducted to evaluate the performance of the proposed approach in which 10 users voluntarily participated. The performance of the proposed method has been tested under two lighting conditions: natural lighting conditions and controlled lighting conditions. Also, the performance has been tested by varying the distance of the user from the camera. Here, it is observed that the system gives best performance when used under controlled lighting conditions and the user sitting at a distance of about 0.5 m. Accuracy of the proposed approach has been found to be 96, 92 and 88% for detection of eye blink, left wink and right wink, respectively. The proposed method has also been tested on ZJU dataset where it has given precision, detection accuracy and false alarm rate of values 94.11, 91.2 and 1.54%, respectively. The proposed system has been used and evaluated for performing various mouse analogous functions using eye blinks and winks. It has given an accuracy of 90, 80 and 90% in performing left click, double click, and right click operations, respectively.

Keywords

Real-time eye blink detection Target selection Analogous mouse operations HCI systems 

Notes

Compliance with ethical standards

Conflict of interest

We have no conflict of interest to declare.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.

References

  1. 1.
    Krolak A, Strumillo P (2012) Eye-blink detection system for human–computer interaction. Univers Access Inf Soc 11:409–419.  https://doi.org/10.1007/s10209-011-0256-6 CrossRefGoogle Scholar
  2. 2.
    Divjak M, Bischof H (2009) eye blink based fatigue detection for prevention of computer vision syndrome. In: IAPR conference on machine vision applications, pp 350–353Google Scholar
  3. 3.
    Danisman T, Bilasco IM, Djeraba C, Ihaddadene N (2010) Drowsy driver detection system using eye blink patterns. In: International conference on machine and web intelligence, pp 230–233Google Scholar
  4. 4.
    Pan G, Sun L, Wu Z (2007) Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In: 11th IEEE international conference on computer vision (ICCV’07)Google Scholar
  5. 5.
    Kumar M, Paepcke A, Winograd T (2007) EyePoint?: practical pointing and selection using gaze and keyboard. In: CHI’07 proceedings of the SIGCHI conference on human factors in computing systems, pp 421–430Google Scholar
  6. 6.
    Singh JV, Prasad G (2015) Enhancing an eye-tracker based human–computer interface with multi-modal accessibility applied for text entry. Int J Comput Appl 130:16–22Google Scholar
  7. 7.
    Guillamet TPA, Teixidó MTM, Viso AF, Palacín CRJ (2013) Implementation of a robust absolute virtual head mouse combining face detection, template matching and optical flow algorithms. Telecommun Syst 52:1479–1489.  https://doi.org/10.1007/s11235-011-9625-y CrossRefGoogle Scholar
  8. 8.
    Huckauf A, Urbina MH (2011) Object selection in gaze controlled systems: what you don’t look at is what you get. ACM Trans Appl Percept 8:1–14.  https://doi.org/10.1145/1870076.1870081 CrossRefGoogle Scholar
  9. 9.
    Urbina MH, Huckauf A (2010) Alternatives to single character entry and dwell time selection on eye typing. In: Proceedings of the 2010 symposium on eye-tracking research & applications, pp 315–322Google Scholar
  10. 10.
    Vertegaal R (2008) A Fitts’ law comparison of eye tracking and manual input in the selection of visual targets. In: ICMI’08 the 10th international conference on multimodal interfaces, pp 241–248Google Scholar
  11. 11.
    Gips J, Olivieri P, Tecce J (1993) Direct control of the computer through electrodes placed around the eyes. In: Proceedings of the fifth international conference on human–computer interaction: applications and case studies, pp 630–635Google Scholar
  12. 12.
    Kraichan C, Pumrin S (2014) Face and eye tracking for controlling computer functions. In: 2014 11th international conference on electrical engineering/electronics, computers, telecommunication and information technology.  https://doi.org/10.1109/ECTICon.2014.6839834
  13. 13.
    Mackenzie IS, Ashtiani B (2011) BlinkWrite?: efficient text entry using eye blinks. Univers Access Inf Soc 10:69–80.  https://doi.org/10.1007/s10209-010-0188-6 CrossRefGoogle Scholar
  14. 14.
    Arai K, Mardiyanto R (2010) Eye-based HCI with Full specification of mouse and keyboard using pupil knowledge in the gaze estimation. In: Proceedings—2011 8th international conference on information technology: new generations, ITNG 2011, pp 423–428.  https://doi.org/10.1109/ITNG.2011.81
  15. 15.
    Khilari R (2010) Iris tracking and blink detection for human-computer interaction using a low resolution webcam. In: ICVGIP’10 proceedings of the 7th Indian conference on computer vision, graphics and image processing, pp 456–463Google Scholar
  16. 16.
    Grauman K, Betke M, Lombardi J et al (2003) Communication via eye blinks and eyebrow raises?: video-based human–computer interfaces. Univers Access Inf Soc 2:359–373CrossRefGoogle Scholar
  17. 17.
    Park KS, Lee KT (1996) Eye controlled human computer interface using the line-of-sight and the intentional blink. Comput Ind Eng 30:463–473CrossRefGoogle Scholar
  18. 18.
    Siriluck W, Kamolphiwong S, Kamolphiwong T (2007) Blink and click. In: Proceedings of the 1st international convention on rehabilitation engineering and assistive technology: in conjunction with 1st Tan Tock Seng Hospital neurorehabilitation meeting, pp 43–46Google Scholar
  19. 19.
    Huckauf A, Urbina MH, Weimar B (2008) On object selection in gaze controlled environments. J Eye Mov Res 2:1–7Google Scholar
  20. 20.
    Yang SW, Lin CS, Lin SK, Lee CH (2013) Design of virtual keyboard using blink control method for the severely disabled. Comput Methods Programs Biomed 111:410–418.  https://doi.org/10.1016/j.cmpb.2013.04.012 CrossRefGoogle Scholar
  21. 21.
    Surakka V, Illi M, Isokoski P (2004) Gazing and frowning as a new human–computer interaction technique. ACM Trans Appl Percept 1:40–56CrossRefGoogle Scholar
  22. 22.
    Gizatdinova Y, Spakov O, Surakka V (2012) Comparison of video-based pointing and selection techniques for hands-free text entry. In: Proceedings of the international working conference on advanced visual interfaces, pp 132–139Google Scholar
  23. 23.
    Zhao XA, Guestrin ED, Sayenko D et al (2012) Typing with eye-gaze and tooth-clicks. In: Proceedings of ETRA 2012, pp 341–344Google Scholar
  24. 24.
    Rantanen V, Verho J, Lekkala J, et al (2012) The effect of clicking by smiling on the accuracy of head-mounted gaze tracking. In: Eye tracking research and applications symposium (ETRA), pp 345–348Google Scholar
  25. 25.
    Usakli AB, Gurkan S, Aloise F et al (2010) On the use of electrooculogram for efficient human computer interfaces. Comput Intell Neurosci 2010:1–5.  https://doi.org/10.1155/2010/135629 Google Scholar
  26. 26.
    Pander T (2008) An application of detection function for the eye blinking detection. In: Human–computer systems interaction part of the advances in intelligent and soft-computing book series, pp 181–191Google Scholar
  27. 27.
    Yamagishi K, Hori J, Miyakawa M (2006) Development of EOG-based communication system controlled by eight-directional eye movements. In: 28th IEEE EMBS annual international conference, pp 2574–2577Google Scholar
  28. 28.
    Pimplaskar D, Nagmode MS, Borkar A (2013) Real time eye blinking detection and tracking using opencv. Int J Eng Res Appl 3:1780–1787Google Scholar
  29. 29.
    Missimer E, Betke M (2010) Blink and wink detection for mouse pointer control. In: PETRA’10 proceedings of the 3rd international conference on pervasive technologies related to assistive environmentsGoogle Scholar
  30. 30.
    Su M, Ye C, Lin S et al (2008) An implementation of an eye-blink-based communication aid for people with severe disabilities. In: International conference on audio, language and image processing 2008, pp 351–356Google Scholar
  31. 31.
    Venkataramanan S, Prabhat P, Choudhury SR, et al (2005) Biomedical instrumentation based on electrooculogram (EOG) signal processing and application to a hospital alarm system. In: Proceedings—2005 international conference on intelligent sensing and information processing, ICISIP’05, pp 535–540Google Scholar
  32. 32.
    Kumar D, Poole E (2002) Classification of EOG for human computer interface. In: Proceedings of the second joint EMBS/BMES conference, pp 64–67Google Scholar
  33. 33.
    Lopez-basterretxea A, Mendez-zorrilla A, Zapirain BG (2015) Eye/head tracking technology to improve HCI with iPad applications. IEEE Sens J 15:2244–2264.  https://doi.org/10.3390/s150202244 CrossRefGoogle Scholar
  34. 34.
    Deepika SS, Murugesan G (2015) A novel approach for human computer interface on eye movements for disabled people. In: 2015 IEEE international conference on electrical, computer and communication technologies (ICECCT 2015)Google Scholar
  35. 35.
    Varona CMJ, Perales FJ, Salinas I (2014) Design recommendations for camera-based head-controlled interfaces that replace the mouse for motion-impaired users. Univers Access Inf Soc 13:471–482.  https://doi.org/10.1007/s10209-013-0326-z CrossRefGoogle Scholar
  36. 36.
    Polatsek P (2013) Eye blink detection. In: Conference of IIT.SRC, pp 1–8Google Scholar
  37. 37.
    Chareonsuk W, Kanhaun S, Khawkam K, Wongsawang D (2016) Face and eyes mouse for ALS Patients. In: 2016 Fifth ICT international student project conference (ICT-ISPC), pp 1–4Google Scholar
  38. 38.
    Gantyala S, Godad W, Phadnis N (2016) Controlling mouse events using eye blink. Int J Adv Res Comput Commun Eng 5:754–756.  https://doi.org/10.17148/IJARCCE.2016.53182 Google Scholar
  39. 39.
    Pallejà T, Rubión E, Tresanchez M, Fernández A (2008) Using the optical flow to implement a relative virtual mouse controlled by head movements. J Univers Comput Sci 14:3127–3141Google Scholar
  40. 40.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition, pp 511–518Google Scholar
  41. 41.
    Mohammed AA, Anwer SA (2014) Efficient eye blink detection method for disabled- helping domain. Int J Adv Comput Sci Appl 5:202–206Google Scholar
  42. 42.
    Won OhL, Eui Chul L, Kang Ryoung P (2010) Blink detection robust to various facial poses. J Neurosci Methods 193:356–372.  https://doi.org/10.1016/j.jneumeth.2010.08.034 CrossRefGoogle Scholar
  43. 43.
    Drutarovsky T, Fogelton A (2014) Eye blink detection using variance of motion vectors. In: ECCV 2014 workshop on computer vision, pp 1–12Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.I.K. Gujral Punjab Technical UniversityJalandharIndia
  2. 2.Department of Electronics and Communication EngineeringBeant College of Engineering and TechnologyGurdaspurIndia
  3. 3.Department of Electronics and Communication EngineeringDAV Institute of Engineering and TechnologyJalandharIndia

Personalised recommendations