Constellation-Based Deep Ear Recognition

Part of the Unsupervised and Semi-Supervised Learning book series (UNSESUL)


This chapter introduces COM-Ear, a deep constellation model for ear recognition. Different from competing solutions, COM-Ear encodes global as well as local characteristics of ear images and generates descriptive ear representations that ensure competitive recognition performance. The model is designed as dual-path convolutional neural network (CNN), where one path processes the input in a holistic manner, and the second captures local images characteristics from image patches sampled from the input image. A novel pooling operation, called patch-relevant-information pooling, is also proposed and integrated into the COM-Ear model. The pooling operation helps to select features from the input patches that are locally important and to focus the attention of the network to image regions that are descriptive and important for representation purposes. The model is trained in an end-to-end manner using a combined cross-entropy and center loss. Extensive experiments on the recently introduced Extended Annotated Web Ears (AWEx).


Ear biometrics Ear recognition Part-based models Constellation model Convolutional neural networks 



This research was supported in parts by the ARRS (Slovenian Research Agency) Research Program P2-0250 (B) Meteorology and Biometric Systems, the ARRS Research Program P2-0214 (A) Computer Vision. The authors thank NVIDIA for donating the Titan Xp GPU that was used in the experiments.

This work was also partially supported by the European Commission through the Horizon 2020 research and innovation program under grants 688201 (M2DC) and 690907 (IDENTITY).


  1. 1.
    A.F. Abate, M. Nappi, D. Riccio, S. Ricciardi, Ear recognition by means of a rotation invariant descriptor, in International Conference on Pattern Recognition, vol. 4 (IEEE, Piscataway, 2006), pp. 437–440Google Scholar
  2. 2.
    A. Abaza, A. Ross, C. Hebert, M.A.F. Harrison, M. Nixon, A survey on ear biometrics. ACM Comput. Surv. 45(2), 1–22 (2013)CrossRefGoogle Scholar
  3. 3.
    M. Abdel-Mottaleb, J. Zhou, Human ear recognition from face profile images, in Advances in Biometrics (Springer, Berlin, 2006), pp. 786–792Google Scholar
  4. 4.
    C. Akin, U. Kacar, M. Kirci, A multi-biometrics for twins identification based speech and ear. CoRR abs/1801.09056 (2018)Google Scholar
  5. 5.
    A.A. Almisreb, N. Jamil, N.M. Din, Utilizing AlexNet deep transfer learning for ear recognition, in International Conference on Information Retrieval and Knowledge Management (IEEE, Piscataway, Mar 2018), pp. 1–5Google Scholar
  6. 6.
    H.A. Alshazly, M. Hassaballah, M. Ahmed, A.A. Ali, Ear biometric recognition using gradient-based feature descriptors, in Proceedings of the International Conference on Advanced Intelligent Systems and Informatics, 2018, ed. by A.E. Hassanien, M.F. Tolba, K. Shaalan, A.T. Azar. Advances in Intelligent Systems and Computing (Springer, Berlin, 2019), pp. 435–445Google Scholar
  7. 7.
    B. Arbab-Zavar, M.S. Nixon, Robust log-Gabor filter for ear biometrics, in International Conference on Pattern Recognition (IEEE, Piscataway, 2008), pp. 1–4Google Scholar
  8. 8.
    Z. Baoqing, M. Zhichun, J. Chen, D. Jiyuan, A robust algorithm for ear recognition under partial occlusion, in Chinese Control Conference (2013), pp. 3800–3804Google Scholar
  9. 9.
    A. Basit, M. Shoaib, A human ear recognition method using nonlinear curvelet feature subspace. Int. J. Comput. Math. 91(3), 616–624 (2014)zbMATHCrossRefGoogle Scholar
  10. 10.
    A. Benzaoui, A. Kheider, A. Boukrouche, Ear description and recognition using ELBP and wavelets, in International Conference on Applied Research in Computer Science and Engineering (2015), pp. 1–6Google Scholar
  11. 11.
    M. Burge, W. Burger, Ear biometrics, in Biometrics, ed. by A.K. Jain, R. Bolle, S. Pankanti (IEEE, Piscataway, 1996), pp. 273–285Google Scholar
  12. 12.
    J.D. Bustard, M.S. Nixon, Toward unconstrained ear recognition from two-dimensional images. IEEE Trans. Syst. Man Cybern. A: Syst. Humans 40(3), 486–494 (2010)CrossRefGoogle Scholar
  13. 13.
    T.S. Chan, A. Kumar, Reliable ear identification using 2-D quadrature filters. Pattern Recogn. Lett. 33(14), 1870–1881 (2012)CrossRefGoogle Scholar
  14. 14.
    K. Chang, K.W. Bowyer, S. Sarkar, B. Victor, Comparison and combination of ear and face images in appearance-based biometrics. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1160–1165 (2003)CrossRefGoogle Scholar
  15. 15.
    M. Choraś, Perspective methods of human identification: ear biometrics. Opto-Electr. Rev. 16(1), 85–96 (2008)Google Scholar
  16. 16.
    M. Choras, R.S. Choras, Geometrical algorithms of ear contour shape representation and feature extraction, in International Conference on Intelligent Systems Design and Applications (IEEE, Piscataway, 2006), pp. 451–456Google Scholar
  17. 17.
    D.P. Chowdhury, S. Bakshi, G. Guo, P.K. Sa, On applicability of tunable filter bank based feature for ear biometrics: a study from constrained to unconstrained. J. Med. Syst. 42(1), 11/1–20 (2017)Google Scholar
  18. 18.
    M. Chowdhury, R. Islam, J. Gao, Robust ear biometric recognition using neural network, in Conference on Industrial Electronics and Applications (IEEE, Piscataway, Jun 2017), pp. 1855–1859Google Scholar
  19. 19.
    N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2005), pp. 886–893Google Scholar
  20. 20.
    N. Damar, B. Fuhrer, Ear recognition using multi-scale histogram of oriented gradients, in Conference on Intelligent Information Hiding and Multimedia Signal Processing (2012), pp. 21–24Google Scholar
  21. 21.
    J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, L. Fei-Fei, Imagenet: a large-scale hierarchical image database, in Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2009), pp. 248–255Google Scholar
  22. 22.
    K. Dewi, T. Yahagi, Ear photo recognition using scale invariant keypoints, in Computational Intelligence (2006), pp. 253–258Google Scholar
  23. 23.
    S. Dodge, J. Mounsef, L. Karam, Unconstrained ear recognition using deep neural networks. IET Biom. 7(3), 207–214 (2018)CrossRefGoogle Scholar
  24. 24.
    Ž. Emeršič, D. Štepec, V. Štruc, P. Peer, Training convolutional neural networks with limited training data for ear recognition in the wild, in International Conference on Automatic Face and Gesture Recognition – Workshop on Biometrics in the Wild (IEEE, Piscataway, 2017), pp. 987–994Google Scholar
  25. 25.
    Ž. Emeršič, D. Štepec, V. Štruc, P. Peer, A. George, A. Ahmad, E. Omar, T.E. Boult, R. Safdari, Y. Zhou, S. Zafeiriou, D. Yaman, F.I. Eyiokur, H.K. Ekenel, The unconstrained ear recognition challenge, in International Joint Conference on Biometrics (IEEE/IAPR, Piscataway, 2017), pp. 715–724Google Scholar
  26. 26.
    Ž. Emeršič, V. Štruc, P. Peer, Ear recognition: more than a survey. Neurocomputing 255, 26–39 (2017)CrossRefGoogle Scholar
  27. 27.
    Ž. Emeršič, N.O. Playà, V. Štruc, P. Peer, Towards accessories-aware ear recognition, in International Work Conference on Bioinspired Intelligence (IEEE, Piscataway, Jul 2018), pp. 1–8Google Scholar
  28. 28.
    Ž. Emeršič, B. Meden, P. Peer, V. Štruc, Evaluation and analysis of ear recognition models: performance, complexity and resource requirements. Neural Comput. Applic., 1–16 (2018).
  29. 29.
    Ž. Emeršič, J. Križaj, V. Štruc, P. Peer, Deep ear recognition pipeline, in Recent Advances in Computer Vision, ed. by M. Hassaballah, K.M. Hosny, vol. 804 (Springer, Berlin, 2019), pp. 333–362Google Scholar
  30. 30.
    F.I. Eyiokur, D. Yaman, H.K. Ekenel, Domain adaptation for ear recognition using deep convolutional neural networks. IET Biom. 7(3), 199–206 (2017)CrossRefGoogle Scholar
  31. 31.
    T. Fan, Z. Mu, R. Yang, Multi-modality recognition of human face and ear based on deep learning, in International Conference on Wavelet Analysis and Pattern Recognition (Jul 2017), pp. 38–42Google Scholar
  32. 32.
    P.L. Galdámez, W. Raveane, A. González Arrieta, A brief review of the ear recognition process using deep neural networks. J. Appl. Log. 24, 62–70 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Y. Guo, Z. Xu, Ear recognition using a new local matching approach, in International Conference on Image Processing (IEEE, Piscataway, 2008), pp. 289–292Google Scholar
  34. 34.
    E.E. Hansley, M.P. Segundo, S. Sarkar, Employing fusion of learned and handcrafted features for unconstrained ear recognition. IET Biom. 7(3), 215–223 (2018)CrossRefGoogle Scholar
  35. 35.
    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2016), pp. 770–778Google Scholar
  36. 36.
    A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, H. Adam, MobileNets: efficient convolutional neural networks for mobile vision applications, CoRR abs/1704.04861 (2017)Google Scholar
  37. 37.
    D.J. Hurley, M.S. Nixon, J.N. Carter, Automatic ear recognition by force field transformations, in Colloquium on Visual Biometrics (IET, London, 2000), pp. 7/1–5Google Scholar
  38. 38.
    A. Jain, A. Ross, K. Nandakumar, Introduction to Biometrics (Springer, Berlin, 2011)CrossRefGoogle Scholar
  39. 39.
    U. Kacar, M. Kirci, Ear recognition with score-level fusion based on CMC in long-wave infrared spectrum, CoRR abs/1801.09054 (2018)Google Scholar
  40. 40.
    J. Kannala, E. Rahtu, BSIF: binarized statistical image features, in International Conference on Pattern Recognition (IEEE, Piscataway, 2012), pp. 1363–1366Google Scholar
  41. 41.
    I. Kemelmacher-Shlizerman, S.M. Seitz, D. Miller, E. Brossard, The MegaFace benchmark: 1 million faces for recognition at scale, in Conference on Computer Vision and Pattern Recognition, pp. 4873–4882 (IEEE, Piscataway, 2016)Google Scholar
  42. 42.
    J. Križaj, V. Štruc, N. Pavešić, Adaptation of SIFT features for robust face recognition, in Image Analysis and Recognition (Springer, Berlin, 2010), pp. 394–404Google Scholar
  43. 43.
    D. Laptev, N. Savinov, J.M. Buhmann, M. Pollefeys, TI-POOLING: transformation-invariant pooling for feature learning in convolutional neural networks, in Conference on Computer Vision and Pattern Recognition (IEEE, Piscataway, 2016), pp. 289–297Google Scholar
  44. 44.
    D.G. Lowe, Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  45. 45.
    S. Mahto, T. Arakawa, T. Koshinak, Ear acoustic biometrics using inaudible signals and its application to continuous user authentication, in European Signal Processing Conference (Sep 2018), pp. 1407–1411Google Scholar
  46. 46.
    B. Moreno, A. Sánchez, J. F. Vélez, On the use of outerear images for personal identification in security applications, in: Proceedings of the International Carnahan Conference onSecurity Technology, IEEE, 1999, pp. 469–476.Google Scholar
  47. 47.
    Z. Mu, L. Yuan, Z. Xu, D. Xi, S. Qi, Shape and structural feature based ear recognition, in Advances in Biometric Person Authentication (Springer, Berlin, pp. 663–670, 2004)CrossRefGoogle Scholar
  48. 48.
    I. Naseem, R. Togneri, M. Bennamoun, Sparse representation for ear biometrics, in Advances in Visual Computing (Springer, Berlin, 2008), pp. 336–345CrossRefGoogle Scholar
  49. 49.
    H. Nejati, L. Zhang, T. Sim, E. Martinez-Marroquin, G. Dong, Wonder ears: identification of identical twins from ear images, in International Conference on Pattern Recognition (IEEE, Piscataway, 2012), pp. 1201–1204Google Scholar
  50. 50.
    T. Ojala, M. Pietikainen, D. Harwood, Performance evaluation of texture measures with classification based on Kullback discrimination of distributions, in International Conference on Pattern Recognition, Computer Vision & Image Processing, vol. 1 (IEEE/IAPR, Piscataway, 1994), pp. 582–585Google Scholar
  51. 51.
    V. Ojansivu, J. Heikkilä, Blur insensitive texture classification using local phase quantization, in Image and Signal Processing (Springer, Berlin, 2008), pp. 236–243Google Scholar
  52. 52.
    V. Ojansivu, E. Rahtu, J. Heikkilä, Rotation invariant local phase quantization for blur insensitive texture analysis, in International Conference on Pattern Recognition (IEEE, Piscataway, 2008), pp. 1–4Google Scholar
  53. 53.
    I. Omara, G. Xiao, M. Amrani, Z. Yan, W. Zuo, Deep features for efficient multi-biometric recognition with face and ear images, in International Conference on Digital Image Processing, vol. 10420 (International Society for Optics and Photonics, Bellingham, Jul 2017), pp. 1–6Google Scholar
  54. 54.
    I. Omara, X. Wu, H. Zhang, Y. Du, W. Zuo, Learning pairwise SVM on hierarchical deep features for ear recognition. IET Biom. 7(6), 557–566 (2018)CrossRefGoogle Scholar
  55. 55.
    R.N. Othman, F. Alizadeh, A. Sutherland, A novel approach for occluded ear recognition based on shape context, in International Conference on Advanced Science and Engineering (IEEE, Piscataway, Oct 2018), pp. 93–98Google Scholar
  56. 56.
    A. Pflug, C. Busch, Ear biometrics: a survey of detection, feature extraction and recognition methods. IET Biom. 1(2), 114–129 (2012)CrossRefGoogle Scholar
  57. 57.
    A. Pflug, P.N. Paul, C. Busch, A comparative study on texture and surface descriptors for ear biometrics, in International Carnahan Conference on Security Technology (IEEE, Piscataway, 2014), pp. 1–6Google Scholar
  58. 58.
    M. Pietikäinen, A. Hadid, G. Zhao, T. Ahonen, Computer vision using local binary patterns, in Computational Imaging and Vision (Springer, Berlin, 2011)CrossRefGoogle Scholar
  59. 59.
    S. Prakash, P. Gupta, An efficient ear recognition technique invariant to illumination and pose. Telecommun. Syst. 52(3), 1435–1448 (2013)CrossRefGoogle Scholar
  60. 60.
    M. Rahman, M.R. Islam, N.I. Bhuiyan, B. Ahmed, A. Islam, Person identification using ear biometrics. Int. J. Comput. Intern. Manag. 15(2), 1–8 (2007)Google Scholar
  61. 61.
    M. Ribič, Ž. Emeršič, V. Štruc, P. Peer, Influence of alignment on ear recognition: case study on AWE dataset, in International Electrotechnical and Computer Science Conference, vol. 25-B (IEEE, Piscataway, 2016), pp. 131–134Google Scholar
  62. 62.
    U. Saeed, M.M. Khan, Combining ear-based traditional and soft biometrics for unconstrained ear recognition. J. Electr. Imaging 27(5), 051220/1–10 (2018)MathSciNetCrossRefGoogle Scholar
  63. 63.
    A. Sepas-Moghaddam, F. Pereira, P.L. Correia, Ear recognition in a light field imaging framework: a new perspective. IET Biom. 7(3), 224–231 (2018)CrossRefGoogle Scholar
  64. 64.
    K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  65. 65.
    H. Sinha, R. Manekar, Y. Sinha, P.K. Ajmera, Convolutional neural network-based human identification using outer ear images, in Soft Computing for Problem Solving, ed. by J.C. Bansal, K.N. Das, A. Nagar, K. Deep, A.K. Ojha. Advances in Intelligent Systems and Computing (Springer, Singapore, 2019), pp. 707–719Google Scholar
  66. 66.
    T. Theoharis, G. Passalis, G. Toderici, I.A. Kakadiaris, Unified 3D face and ear recognition using wavelets on geometry images. Pattern Recogn. 41(3), 796–804 (2008)zbMATHCrossRefGoogle Scholar
  67. 67.
    L. Tian, Z. Mu, Ear recognition based on deep convolutional network, in International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (IEEE, Piscataway, Oct 2016), pp. 437–441Google Scholar
  68. 68.
    B. Victor, K. Bowyer, S. Sarkar, An evaluation of face and ear biometrics, in International Conference on Pattern Recognition, vol. 1 (IEEE, Piscataway, 2002), pp. 429–432Google Scholar
  69. 69.
    R. Vidyasri, B. Priyalakshmi, M.R. Raja, S. Priyanka, Recognition of ear based on partial features fusion. Int. J. Biom. 10(2), 105–120 (2018)Google Scholar
  70. 70.
    N.S. Vu, A. Caplier, Face recognition with patterns of oriented edge magnitudes, in European Conference on Computer Vision (Springer, Berlin, 2010), pp. 313–326Google Scholar
  71. 71.
    Y. Wen, K. Zhang, Z. Li, Y. Qiao, A discriminative feature learning approach for deep face recognition, in European Conference on Computer Vision (Springer, Berlin, 2016), pp. 499–515Google Scholar
  72. 72.
    Z. Xie, Z. Mu, Ear recognition using LLE and IDLLE algorithm, in International Conference on Pattern Recognition (IEEE, Piscataway, 2008), pp. 1–4Google Scholar
  73. 73.
    X. Xu, Z. Mu, L. Yuan, Feature-level fusion method based on KFDA for multimodal recognition fusing ear and profile face, in International Conference on Wavelet Analysis and Pattern Recognition (IEEE, Piscataway, 2007), pp. 1306–1310Google Scholar
  74. 74.
    T. Ying, W. Shining, L. Wanxiang, Human ear recognition based on deep convolutional neural network, in Chinese Control And Decision Conference (Jun 2018), pp. 1830–1835Google Scholar
  75. 75.
    L. Yuan, Z.C. Mu, Ear recognition based on 2D images, in Conference on Biometrics: Theory, Applications and Systems (IEEE, Piscataway, 2007), pp. 1–5Google Scholar
  76. 76.
    L. Yuan, H. Zhao, Y. Zhang, Z. Wu, Ear alignment based on convolutional neural network, in Biometric Recognition, ed. by J. Zhou, Y. Wang, Z. Sun, Z. Jia, J. Feng, S. Shan, K. Ubul, Z. Guo, pp. 562–571. Lecture Notes in Computer Science (Springer, Berlin, 2018)CrossRefGoogle Scholar
  77. 77.
    H.J. Zhang, Z.C. Mu, W. Qu, L.M. Liu, C.Y. Zhang, A novel approach for ear recognition based on ICA and RBF network, in International Conference on Machine Learning and Cybernetics, vol. 7 (IEEE, Piscataway, 2005), pp. 4511–4515Google Scholar
  78. 78.
    Y. Zhang, Z. Mu, L. Yuan, H. Zeng, L. Chen, 3D ear normalization and recognition based on local surface variation. Appl. Sci. 7(1), 104/1–21 (2017)Google Scholar
  79. 79.
    Y. Zhang, Z. Mu, L. Yuan, C. Yu, Ear verification under uncontrolled conditions with convolutional neural networks. IET Biom. 7(3), 185–198 (2018)CrossRefGoogle Scholar
  80. 80.
    Y. Zhou, S. Zaferiou, Deformable models of ears in-the-wild for alignment and recognition, in International Conference on Automatic Face and Gesture Recognition (IEEE, Piscataway, May 2017), pp. 626–633Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.XLAB d.o.o.LjubljanaSlovenia
  2. 2.Computer Vision Laboratory, Faculty of Computer and Information ScienceUniversity of LjubljanaLjubljanaSlovenia
  3. 3.Laboratory of Artificial Perception, Systems and Cybernetics, Faculty of Electrical EngineeringUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations