Eye pupil localization is one of the indispensable technologies in various computer vision applications such as virtual reality and augmented reality. In general, the algorithm consists of finding the approximate eye region and finding the pupil position by extracting the semantic feature from each eye region. However, the performance of the eye pupil location is affected not only by illumination and image resolution but also by glasses wear. Therefore, this paper proposes an eye pupil localization algorithm which is robust against the above disturbance conditions and provides high accuracy. First, a face is detected from an input image and it is determined whether to wear glasses using the detected face. If glasses are present, the glasses are removed to find the correct eye region. Then, facial landmarks are extracted, and eye regions are detected based on facial landmarks. Next, the pupil region is segmented using fully convolutional networks. Finally, the position of the segmented pupil is calculated. Experimental results show that the proposed algorithm outperforms the state-of-the-art algorithms for public databases such as BioID and GI4E by up to 3.44% 0.5%, respectively.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Abbasi M, Khosravi M (2019) A robust and accurate particle filter-based pupil detection method for big datasets of eye video. J Grid Comput 1–21
Araujo GM, Ribeiro FM, Silva EA, Goldenstein SK (2014) Fast eye localization without a face model using inner product detectors, IEEE International Conference on Image Processing (ICIP), pp. 1366–1370
Baek S, Choi K, Ma C, Kim Y, Ko S (2013) Eyeball model-based iris center localization for visible image-based eye-gaze tracking systems. IEEE Trans Consum Electron 59:415–421
Baldi P, Hornik K (1989) Neural networks and principal component analysis: learning from examples without local minima. Neural Netw 2:53–58
Cai H, Liu B, Ju Z, Thill S, Belpaeme T, Vanderborght B, Liu H (2018) Accurate eye center localization via hierarchical adaptive convolution, British Machine Vision Conference
George A, Routray A (2016) Fast and accurate algorithm for eye localisation for gaze tracking in low-resolution images. IET Comput Vis 10:660–669
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets, Adv Neural Inf Proces Syst, pp. 2672–2680
Gou C, Wu Y, Wang K, Wang K, Wang F, Ji Q (2017) A joint cascaded framework for simultaneous eye detection and eye state estimation. Pattern Recogn 67:23–31
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, Proc IEEE Conf Comput Vis Pattern Recognit, pp. 770–778
Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976
O. Jesorsky, K.J. Kirchberg and R.W. Frischholz (2001) Robust face detection using the hausdorff distance, International conference on audio-and video-based biometric person authentication, pp. 90–95
Kassner M, Patera W, Bulling A (2014) Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction, Proceedings of the ACM international joint conference on pervasive and ubiquitous computing: Adjunct publication, pp. 1151–1160
Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees, Proc IEEE Conf Comput Vis Pattern Recognit, pp. 1867–1874
Levinshtein A, Phung E, Aarabi P (2018) Hybrid eye center localization using cascaded regression and hand-crafted model fitting. Image Vis Comput 71:17–24
Li B, Fu H (2018) Real time eye detector with cascaded convolutional neural networks, Applied Computational Intelligence and Soft Computing, 2018
Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C, Berg AC (2016) Ssd: single shot multibox detector, European conference on computer vision, pp. 21–37
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. Proc IEEE Conf Comput Vis Pattern Recognit:3431–3440
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation, International Conference on Medical image computing and computer-assisted intervention, pp. 234–241
Świrski L, Bulling A, Dodgson N (2012) Robust real-time pupil tracking in highly off-axis images, Proceedings of the Symposium on Eye Tracking Research and Applications, pp. 173–176
Timm F, Barth E (2011) Accurate eye Centre localisation by means of gradients. Visapp 11:125–130
Tonsen M, Zhang X, Sugano Y, Bulling A (2016) Labelled pupils in the wild, Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications, pp. 139–142
Valenti R, Gevers T (2012) Accurate eye center location through invariant isocentric patterns. IEEE Trans Pattern Anal Mach Intell 34:1785–1798
Villanueva A, Ponz V, Sesma-Sanchez L, Ariz M, Porta S, Cabeza R (2013) Hybrid method based on topography for robust detection of iris center and eye corners. ACM Transact Multimed Comput Commun Appl (TOMM) 9:25
Yifan X, Hui Y, Yue F (2019) Accurate and robust eye center localization via fully convolutional networks. IEEE/CAA J Automatic Sin 6:1127–1138
Zhu J, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks, IEEE International Conference on Computer Vision (ICCV)
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Choi, J.H., Lee, K.I. & Song, B.C. Eye pupil localization algorithm using convolutional neural networks. Multimed Tools Appl 79, 32563–32574 (2020). https://doi.org/10.1007/s11042-020-09711-x
- Eye pupil localization
- Fully convolutional networks
- Eyeglasses removal