Skip to main content
SpringerLink
Log in

Multimodal soft biometrics: combining ear and face biometrics for age and gender classification

  • 1169: Interdisciplinary Forensics: Government, Academia and Industry Interaction
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present a multimodal, multitask deep convolutional neural network framework for age and gender classification. In the developed framework, we have employed two different biometric modalities: ear and profile face. We have explored three different fusion methods, namely, data, feature, and score fusion, to combine the information extracted from ear and profile face images. In the framework, we have utilized VGG-16 and ResNet-50 models with center loss to obtain more discriminative features. Moreover, we have performed two-stage fine-tuning to increase the representation capacity of the models. To assess the performance of the proposed approach, we have conducted extensive experiments on the FERET, UND-F, and UND-J2 datasets. Experimental results indicate that ear and profile face images contain useful features to extract soft biometric traits. We have shown that when frontal face view of the subject is not available, use of ear and profile face images can be a good alternative for the soft biometric recognition systems. The presented multimodal system achieves very high age and gender classification accuracies, matching the ones obtained by using frontal face images. The multimodal approach has outperformed both the unimodal approaches and the previous state-of-the-art profile face image or ear image-based age and gender classification methods, significantly in both tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://github.com/iremeyiokur/multipie_extended_ear_dataset

  2. https://github.com/yamand16/age_and_gender_classification

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, et al. (2016) Tensorflow: A system for large-scale machine learning. In: 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pp 265–283

  2. Abaza A, Ross A, Hebert C, Harrison MAF, Nixon MS (2013) A survey on ear biometrics. ACM Comput Surv 45(2):22

    Article  Google Scholar 

  3. Antipov G, Baccouche M, Berrani SA, Dugelay JL (2017) Effective training of convolutional neural networks for face-based gender and age prediction. Pattern Recogn 72:15–26

    Article  Google Scholar 

  4. Bradski G, Kaehler A (2000) OpenCV. Dr. Dobb’s journal of software tools, 3

  5. Bukar AM, Ugail H (2017) Automatic age estimation from facial profile view. IET Comput Vis 11(8):650–655

    Article  Google Scholar 

  6. Buolamwini J, Gebru T (2018) Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency, pp 77–91

  7. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In: Computer vision and pattern recognition. IEEE, pp 248–255

  8. Duan M, Li K, Yang C, Li K (2018) A hybrid deep learning cnn–elm for age and gender classification. Neurocomputing 275:448–461

    Article  Google Scholar 

  9. Eidinger E, Enbar R, Hassner T (2014) Age and gender estimation of unfiltered faces. IEEE Trans Inf Forensics Secur 9(12):2170–2179

    Article  Google Scholar 

  10. Emeršič ž, Štruc V, Peer P (2017) Ear recognition: More than a survey. Neurocomputing 255:26–39

    Article  Google Scholar 

  11. Eyiokur FI, Yaman D, Ekenel HK (2017) Domain adaptation for ear recognition using deep convolutional neural networks. IET Biometrics 7(3):199–206

    Article  Google Scholar 

  12. Gnanasivam P, Muttan S (2013) Gender classification using ear biometrics. In: International conference on signal and image processing. Springer, pp 137–148

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Computer vision and pattern recognition. IEEE, pp 770–778

  14. Iannarelli A (1989) Ear identification, forensic identification series. Paramont Publ Company

  15. Introduction to USTB ear image databases: http://www1.ustb.edu.cn/resb/en/index.htm. Access date 30 Oct 2018

  16. Jain AK, Dass SC, Nandakumar K (2004) Soft biometric traits for personal recognition systems. In: Biometric authentication. Springer, pp 731–738

  17. Jain AK, Park U (2009) Facial marks: Soft biometric for face recognition. In: International conference on image processing. IEEE, pp 37–40

  18. Khorsandi R, Abdel-Mottaleb M (2013) Gender classification using 2-D ear images and sparse representation. In: Workshop on applications of computer vision. IEEE, pp 461–466

  19. King DE (2009) Dlib-ml: A machine learning toolkit. J Mach Learn Res 10(Jul):1755–1758

    Google Scholar 

  20. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

  21. Lei J, Zhou J, Abdel-Mottaleb M (2013) Gender classification using automatically detected and aligned 3D ear range data. In: International conference on biometrics. IEEE, pp 1–7

  22. Levi G, Hassner T (2015) Age and gender classification using convolutional neural networks. In: Computer vision and pattern recognition workshops, pp 34–42

  23. Meng D, Mahmoodi S, Nixon MS (2020) Which ear regions contribute to identification and to gender classification?. In: 2020 8th international workshop on biometrics and forensics (IWBF). IEEE, pp 1–6

  24. Ozbulak G, Aytar Y, Ekenel HK (2016) How transferable are CNN-based features for age and gender classification?. In: International conference of the biometrics special interest group. IEEE, pp 1–6

  25. Pflug A, Busch C (2012) Ear biometrics: A survey of detection, feature extraction and recognition methods. IET Biometrics 1(2):114–129

    Article  Google Scholar 

  26. Phillips PJ, Moon H, Rizvi SA, Rauss PJ (2000) The FERET evaluation methodology for face-recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104

    Article  Google Scholar 

  27. Purkait R, Singh P (2007) Anthropometry of the normal human auricle: A study of adult Indian men. Aesthetic Plast Surg 31(4):372–379

    Article  Google Scholar 

  28. Rothe R, Timofte R, Van Gool L (2018) Deep expectation of real and apparent age from a single image without facial landmarks. Int J Comput Vis 126 (2-4):144–157

    Article  MathSciNet  Google Scholar 

  29. Saeed U, Khan MM (2018) Combining ear-based traditional and soft biometrics for unconstrained ear recognition. J Electronic Imag 27(5):051220

    Article  Google Scholar 

  30. Sforza C, Grandi G, Binelli M, Tommasi DG, Rosati R, Ferrario VF (2009) Age-and sex-related changes in the normal human ear. Forensic Sci Int 187(1-3):110–e1

    Article  Google Scholar 

  31. Sharif Razavian A, Azizpour H, Sullivan J, Carlsson S (2014) CNN features off-the-shelf: An astounding baseline for recognition. In: Computer vision and pattern recognition workshops, pp 806–813

  32. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  33. Vaquero DA, Feris RS, Tran D, Brown L, Hampapur A, Turk M (2009) Attribute-based people search in surveillance environments. In: Workshop on applications of computer vision. IEEE, pp 1–8

  34. Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision. Springer, pp 499–515

  35. Yaman D, Eyiokur FI, Sezgin N, Ekenel HK (2018) Age and gender classification from ear images. In: International workshop on biometrics and forensics. IEEE

  36. Yaman D, Irem Eyiokur F, Kemal Ekenel H (2019) Multimodal age and gender classification using ear and profile face images. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 0–0

  37. Yan P, Bowyer KW (2005) Empirical evaluation of advanced ear biometrics. In: Computer vision and pattern recognition workshops. IEEE, p 41

  38. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. In: Advances in neural information processing systems, pp 3320–3328

  39. Zhang G, Wang Y (2011) Hierarchical and discriminative bag of features for face profile and ear based gender classification. In: International joint conference on biometrics. IEEE, pp 1–8

  40. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929

Download references

Acknowledgments

This study is supported by the Istanbul Technical University Research Fund, ITU BAP, project no.42547 and Cost Action CA16101 - MULTI-modal Imaging of FOREnsic SciEnce Evidence - tools for Forensic Science (MULTI-FORESEE).

Author information

Author notes

    Authors and Affiliations

    1. Istanbul Technical University, Istanbul, Turkey

      Dogucan Yaman, Fevziye Irem Eyiokur & Hazım Kemal Ekenel

    Authors
    1. Dogucan Yaman
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Fevziye Irem Eyiokur
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Hazım Kemal Ekenel
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Dogucan Yaman.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Dogucan Yaman and Fevziye Irem Eyiokur have equally contributed.

    Dogucan Yaman and F. Irem Eyiokur did this work as students at Istanbul Technical University. Now, they are working at Karlsruhe Institute of Technology.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Yaman, D., Eyiokur, F.I. & Ekenel, H.K. Multimodal soft biometrics: combining ear and face biometrics for age and gender classification. Multimed Tools Appl 81, 22695–22713 (2022). https://doi.org/10.1007/s11042-021-10630-8

    Download citation

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-021-10630-8

    Keywords

    Search

    Navigation