Skip to main content
SpringerLink
Log in

Learning similarity and dissimilarity in 3D faces with triplet network

  • 1171: Real-time 2D/ 3D Image Processing with Deep Learning
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Face is the most preferred biometric trait used to recognize a person. The 2D face can be considered as a promising biometric trait; however, it may be affected by changes in the age, skin color, texture, or any other environmental factors like illumination variations, occlusion, and low image resolution. The 3D face is an emerging biometric trait, which is being used recently for human recognition. The discriminating power of the 3D face is highly motivating for many tasks such as security, surveillance, and many other technological application in day to day life. Although there are many techniques available for 3D face recognition, most of these techniques are based on volumetric or depth/range images. The conversion of 3D face data, which is originally in point cloud format to volumetric representation makes the data bulkier. Further, some of the geometric properties may be lost when 3D data is converted to a representation of lower dimensions such as depth/range images. The driving objective behind this research is to perform 3D face recognition by directly using faces represented in the form of 3D point cloud. We propose a novel approach for 3D face recognition by learning the similarity and dissimilarity in 3D faces, and for this purpose, introduce a triplet network. The network is an ensemble of our proposed Convolutional PointNet (CPN) network, used for feature extraction and triplet loss. The proposed network maps a 3D face data to Euclidean space where distance based scores represent the similarity among the 3D faces. We also introduce a new evaluation approach for computing the dissimilarity between highly similar 3D face biometric data. Experimentation has been carried out on two databases, namely IIT Indore 3D face database (our in-house database) and Bosphorus 3D face database. To handle the training issues due to the limited availability of samples for each subject in both the databases, we propose a technique for 3D data augmentation. We perform various experiments using the proposed network and show the performance in terms of verification rate and ROC curve. Our point cloud based triplet network shows encouraging performance as compared to other state-of-the-art techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Abate AF, Nappi M, Riccio D, Sabatino G (2007) 2D and 3D face recognition: a survey. Pattern Recogn Lett 28(14):1885–1906

    Article  Google Scholar 

  2. Ahmed E, Saint A, Shabayek AER, Cherenkova K, Das R, Gusev G, Aouada D, Ottersten B (2018) Deep learning advances on different 3D data representations: a survey. arXiv:180801462

  3. Bagchi P, Bhattacharjee D, Nasipuri M (2016) A robust analysis, detection and recognition of facial features in 2.5D images. Multimed Tools and Appl 75 (18):11059–11096

    Article  Google Scholar 

  4. Berretti S, Werghi N, Del Bimbo A, Pala P (2013) Matching 3D face scans using interest points and local histogram descriptors. Comput Graph 37 (5):509–525

    Article  Google Scholar 

  5. Bhople Anagha SA, Prakash S (2020) Point cloud based deep convolutional neural network for 3D face recognition. Multimed Tools Appl. https://doi.org/10.1007/s11042-020-09008-z

  6. Blanz V, Vetter T, et al. (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of SIGGRAPH, pp 187–194

  7. Borisenko G, Denisov A, Krylov A (2004) A diffusion filtering method for image processing. Program Comput Softw 30(5):273–277

    Article  Google Scholar 

  8. Bowyer KW, Chang K, Flynn P (2006) A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition. Comput Vis Image Underst 101(1):1–15

    Article  Google Scholar 

  9. Chelali FZ, Djeradi A, Djeradi R (2009) Linear discriminant analysis for face recognition. In: Proceedings of international conference on multimedia computing and systems, pp 1–10

  10. Chouchane A, Ouamane A, Boutellaa E, Belahcene M, Bourennane S (2018) 3D face verification across pose based on euler rotation and tensors. Multimed Tools Appl 77(16):20697–20714

    Article  Google Scholar 

  11. Dorofeev K, Ruchay A, Kober A, Kober V (2019) 3D face recognition using depth filtering and deep convolutional neural network. In: Proceedings of applications of digital image processing XLII, vol 11137, p 111371Y

  12. Drira H, Amor BB, Srivastava A, Daoudi M, Slama R (2013) 3D face recognition under expressions, occlusions, and pose variations. IEEE Trans Pattern Anal Mach Intell 35(9):2270–2283

    Article  Google Scholar 

  13. Feng J, Guo Q, Guan Y, Wu M, Zhang X, Ti C (2019) 3D face recognition method based on deep convolutional neural network. In: Proceedings of smart innovations in communication and computational sciences, pp 123–130

  14. Garcia-Garcia A, Gomez-Donoso F, Garcia-Rodriguez J, Orts-Escolano S, Cazorla M, Azorin-Lopez J (2016) Pointnet: a 3D convolutional neural network for real-time object class recognition. In: Proceedings of international joint conference on neural networks, pp 1578–1584

  15. Gilani SZ, Mian A, Shafait F, Reid I (2017) Dense 3D face correspondence. IEEE Trans Pattern Anal Mach Intell 40(7):1584–1598

    Article  Google Scholar 

  16. Gupta S, Markey MK, Bovik AC (2010) Anthropometric 3D face recognition. Int J Comput Vis 90(3):331–349

    Article  Google Scholar 

  17. He Y, Liang B, Yang J, Li S, He J (2017) An iterative closest points algorithm for registration of 3D laser scanner point clouds with geometric features. Sensors 17(8):1862

    Article  Google Scholar 

  18. Jayaraman U, Gupta P, Gupta S, Arora G, Tiwari K (2020) Recent development in face recognition. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.08.110

  19. Kang BN, Kim Y, Kim D (2017) Deep convolutional neural network using triplets of faces, deep ensemble, and score-level fusion for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 109–116

  20. Kim D, Hernandez M, Choi J, Medioni G (2017) Deep 3D face identification. In: Proceedings of IEEE international joint conference on biometrics, pp 133–142

  21. Kingkan C, Owoyemi J, Hashimoto K (2018) Point attention network for gesture recognition using point cloud data. In: Proceedings of British machine vision conference, pp 118–130

  22. Koch G, Zemel R, Salakhutdinov R (2015) Siamese neural networks for one-shot image recognition. In: Proceedings of ICML deep learning workshop, vol 2

  23. Koziol Q (2011) HDF5, Encyclopedia of parallel computing. In: Encyclopedia of parallel computing, pp 827–833

  24. Le T, Duan Y (2018) Pointgrid: a deep network for 3D shape understanding. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 9204–9214

  25. Lei Y, Guo Y, Hayat M, Bennamoun M, Zhou X (2016) A two-phase weighted collaborative representation for 3D partial face recognition with single sample. Pattern Recogn 52:218–237

    Article  Google Scholar 

  26. Leng B, Liu Y, Yu K, Xu S, Yuan Z, Qin J (2016) Cascade shallow CNN structure for face verification and identification. Neurocomputing 215:232–240

    Article  Google Scholar 

  27. Leo MJ, Suchitra S (2018) SVM based expression-invariant 3D face recognition system. Procedia Comput Sci 143:619–625

    Article  Google Scholar 

  28. Li H, Huang D, Morvan JM, Wang Y, Chen L (2014) Towards 3D face recognition in the real: a registration-free approach using fine-grained matching of 3D keypoint descriptors. Int J Comput Vis 113:128–142

    Article  MathSciNet  Google Scholar 

  29. Li Y, Wang Y, Liu J, Hao W (2018) Expression-insensitive 3D face recognition by the fusion of multiple subject-specific curves. Neurocomputing 275:1295–1307

    Article  Google Scholar 

  30. Marcolin F, Vezzetti E (2017) Novel descriptors for geometrical 3D face analysis. Multimedia Tools and Applications 76(12):13805–13834

    Article  Google Scholar 

  31. Maturana D, Scherer S (2015) Voxnet: a 3D convolutional neural network for real-time object recognition. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems, pp 922–928

  32. Mian A, Bennamoun M, Owens R (2007) An efficient multimodal 2D-3D hybrid approach to automatic face recognition. IEEE Trans Pattern Anal Mach Intell 29(11):1927–1943

    Article  Google Scholar 

  33. Mian AS, Bennamoun M, Owens R (2008) Keypoint detection and local feature matching for textured 3D face recognition. Int J Comput Vis 79 (1):1–12

    Article  Google Scholar 

  34. Patil H, Kothari A, Bhurchandi K (2015) 3D Face recognition: features, databases, algorithms and challenges. Artif Intell Rev 44(3):393–441

    Article  Google Scholar 

  35. Qi CR, Su H, Mo K, Guibas LJ (2017) Pointnet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 652–660

  36. Qi CR, Yi L, Su H, Guibas LJ (2017) Pointnet++: deep hierarchical feature learning on point sets in a metric space. In: Proceedings of advances in neural information processing systems, pp 5099–5108

  37. Rahim R, Afriliansyah T, Winata H, Nofriansyah D, Aryza S, et al. (2018) Research of face recognition with fisher linear discriminant. In: Proceedings of IOP conference series: materials science and engineering, pp 012–037

  38. Reji R, SojanLal P (2017) Region based 3D face recognition. In: Proceedings of IEEE international conference on computational intelligence and computing research, pp 1–6

  39. Savran A, Alyüz N, Dibeklioğlu H, Çeliktutan O, Gökberk B, Sankur B, Akarun L (2008) Bosphorus database for 3D face analysis. In: Proceedings of European workshop on biometrics and identity management, pp 47–56

  40. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 815–823

  41. Sharma PB, Goyani MM (2012) 3D face recognition techniques-a review. Int J Eng Res Appl 2(1):787–793

    Google Scholar 

  42. Sharma S, Kumar V (2020) Voxel-based 3D face reconstruction and its application to face recognition using sequential deep learning. Multimed Tools Appl 79:17303–17330. https://doi.org/10.1007/s11042-020-08688-x

    Article  Google Scholar 

  43. Singh RD, Mittal A, Bhatia RK (2019) 3D convolutional neural network for object recognition: a review. Multimed Tools Appl 78(12):15951–15995

    Article  Google Scholar 

  44. Soltanpour S, Wu QJ (2016) Multimodal 2D–3D face recognition using local descriptors: pyramidal shape map and structural context. IET Biometrics 6(1):27–35

    Article  Google Scholar 

  45. Soltanpour S, Boufama B, Wu QJ (2017) A survey of local feature methods for 3D face recognition. Pattern Recogn 72:391–406

    Article  Google Scholar 

  46. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1701–1708

  47. Turk MA, Pentland AP (1991) Face recognition using eigenfaces. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp 586–591

  48. Yu Y, Da F, Guo Y (2018) Sparse icp with resampling and denoising for 3D face verification. IEEE Trans on Inf Forensics Secur 14(7):1917–1927

    Article  Google Scholar 

  49. Zhang Z, Da F, Wang C, Yu J, Yu Y (2019) Face recognition on 3D point clouds. In: Proceedings of seventh international conference on optical and photonic engineering (icOPEN 2019), vol 11205, pp 350–355

  50. Zhang Z, Da F, Yu Y (2019) Data-free point cloud network for 3D face recognition. arXiv:1911.04731

  51. Zhou H, Mian A, Wei L, Creighton D, Hossny M, Nahavandi S (2014) Recent advances on singlemodal and multimodal face recognition: a survey. IEEE Trans Hum-Mach Syst 44(6):701–716

    Article  Google Scholar 

  52. Zulqarnain Gilani S, Mian A (2018) Learning from millions of 3D scans for large-scale 3D face recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1896–1905

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology Indore, Indore, 453552, India

    Anagha R. Bhople & Surya Prakash

Authors
  1. Anagha R. Bhople
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Surya Prakash
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anagha R. Bhople.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bhople, A.R., Prakash, S. Learning similarity and dissimilarity in 3D faces with triplet network. Multimed Tools Appl 80, 35973–35991 (2021). https://doi.org/10.1007/s11042-020-10160-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10160-9

Keywords

Search

Navigation