Abstract
Automatic facial beauty analysis has become an emerging research topic. Despite some achieved advances, current methods and systems suffer from at least two limitations. Firstly, many developed systems rely on the use of ad-hoc hand-crafted features that were designed for generic pattern recognition problems. Secondly, while Deep Convolutional Neural Nets (DCNN) have been recently demonstrated to be a promising area of research in statistical machine learning, their use for automatic face beauty analysis may not guarantee optimal performances due to the use of a limited amount of face images with beauty scores. In this paper, we attempt to overcome these two main limitations by jointly exploiting two tricks. First, instead of using hand-crafted face features we use deep features of a pre-trained DCNN able to generate a high-level representation of a face image. Second, we exploit manifold learning theory and deploy three graph-based semi-supervised learning methods in order to enrich model learning without the need of additional labeled face images. These schemes perform graph-based score propagation. The proposed schemes were tested on three public datasets for beauty analysis: SCUT-FBP, M2B, and SCUT-FBP5500. These experiments, as well as many comparisons with supervised schemes, show that the scheme coined Kernel Flexible Manifold Embedding compares favorably with many supervised schemes. They also show that its performances in terms of error prediction and Pearson Correlation are better than those reported for the used datasets.
Similar content being viewed by others
Notes
The network can be downloaded from http://www.vlfeat.org/matconvnet/pretrained/.
References
Aarabi P, Hughes D, Mohajer K, Emami M (2001) The automatic measurement of facial beauty. In: 2001 IEEE international conference on systems, man, and cybernetics, vol 4. IEEE, pp 2644–2647
Bainbridge W, Isola P, Oliva A (2013) The intrinsic memorability of face images. J Exp Psychol Gen 142(4):1323–1334
Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396
Bi J, Bennett KP (2003) Regression error characteristic curves. In: Proceedings of the 20th international conference on machine learning (ICML-03), pp 43–50
Bottino A, Laurentini A (2010) The analysis of facial beauty: an emerging area of research in pattern analysis. In: ICIAR. Part I, LNCS (6111), pp 425–435
Cheng B, Yang J, Yan S, Fu Y, Huang T (2010) Learning with l1-graph for image analysis. Trans Img Proc 19(4):858–866
datatang. http://datatang.com/
De Bacco C, Larremore DB, Moore C (2018) A physical model for efficient ranking in networks. arXiv:1709.09002
Dornaika F, Bosaghzadeh A (2015) Adaptive graph construction using data self-representativeness for pattern classification. Inf Sci 325:118–139
Dornaika F, Dhabi R, Bosaghzadeh A, Ruichek Y (2016) Dynamic adaptive graph construction: application to graph-based multi-observation classification. In: IEEE international conference on pattern recognition
Dornaika F, El Traboulsi Y (2016) Learning flexible graph-based semi-supervised embedding. IEEE Trans Cybern 46(1):206–218
Dornaika F, Traboulsi YE, Assoum A (2013) Adaptive two phase sparse representation classifier for face recognition. In: LNCS 8192. Advanced concepts for intelligent vision systems
Eisenthal Y, Dror G, Ruppin E (2006) Facial attractiveness: beauty and the machine. Neural Comput 18(1):119–142
Gan J, Li L, Zhai Y, Liu Y (2014) Deep self-taught learning for facial beauty prediction. Neurocomputing 144:295–303
Gan J, Wang B, Xu Y (2015) A novel method for predicting facial beauty under unconstrained condition. Springer, Cham, pp 350–360
Gray D, Yu K, Xu W, Gong Y (2010) Predicting facial beauty without landmarks. Comput Vision–ECCV 2010, 434–447
He R, Zheng W-S, Hu B-G, Kong X-W (2011) Nonnegative sparse coding for discriminative semi-supervised learning. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 2849–2856
Kagian A, Dror G, Leyvand T, Cohen-Or D, Ruppin E (2007) A humanlike predictor of facial attractiveness, pp 649–656
Kagian A, Dror G, Leyvand T, Meilijson I, Cohen-Or D, Ruppin E (2008) A machine learning predictor of facial attractiveness revealing human-like psychophysical biases. Vision Res 48(2):235–243
Laurentini A, Bottino A (2014) Computer analysis of face beauty a survey. Computer Vision and Image Understanding 125(Supplement C):184–199
Liang L, Lin L, Jin L, Xie D, Li M (2018) SCUT-FBP5500: a diverse benchmark dataset for multi-paradigm facial beauty prediction. arXiv:1801.06345v1[cs.CV] 19 Jan 2018
Liu S, Fan Y-Y, Samal A, Guo Z (2016) Advances in computational facial attractiveness methods. Multimed Tools Appl 75(23):16633–16663
Mu Y (2013) Computational facial attractiveness prediction by aesthetics-aware features. Neurocomputing 99:59–64
Nguyen TV, Liu S, Ni B, Tan J, Rui Y, Yan S (2013) Towards decrypting attractiveness via multi-modality cues. ACM Trans Multimed Comput, Commun, Appl (TOMM) 9(4):28
Nie F, Wang X, Jordan MI, Huang H (2016) The constrained laplacian rank algorithm for graph-based clustering. In: AAAI conference on artificial intelligence
Nie F, Xu D, Tsang IW-H, Zhang C (2010) Flexible manifold embedding: a framework for semi-supervised and unsupervised dimension reduction. IEEE Trans Image Process 19(7):1921–1932
Nie L, Song X, Chua T-S (2016) Learning from multiple social networks. Morgan and Claypool Publishers, San Rafael
Nie L, Wang M, Zha Z-J, Chua T-S (2012) Oracle in image search: a content-based approach to performance prediction. ACM Trans Inf Syst 30:13:1–13:23
Parkhi OM, Vedaldi A, Zisserman A, et al. (2015) Deep face recognition. In: BMVC, vol 1, p 6
Raducanu B, Dornaika F (2014) Embedding new observations via sparse-coding for non-linear manifold learning. Pattern Recogn 47(1):480–492
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Fei-Fei L (2018) Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014
Schmid K, Marx D, Samal A (2008) Computation of a face attractiveness index based on neoclassical canons, symmetry, and golden ratios. Pattern Recogn 41 (8):2710–2717
Sharif Razavian A, Azizpour H, Sullivan J, Carlsson S (2014) Cnn features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 806–813
Sutic D, Rreskovic I, Huic R, Jukic I (2010) Automatic evaluation of facial attractiveness. In: Proceedings of the 33rd international convention, pp 1339–1342
Traboulsi YE, Dornaika F, Assoum A (2015) Kernel flexible manifold embedding for pattern classification. Neurocomputing 167:517–527
Wang S, Shao M, Fu Y (2014) Attractive or not?: beauty prediction with attractiveness-aware encoders and robust late fusion. In: Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 805–808
Whitehill J, Movellan JR (2008) Personalized facial attractiveness prediction. In: IEEE international conference on automatic face gesture recognition, pp 1–7
Xie D, Liang L, Jin L, Xu J, Li M (2015) Scut-fbp: a benchmark dataset for facial beauty perception. In: 2015 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 1821–1826
Xu J, Jin L, Liang L, Feng Z, Xie D (2015) A new humanlike facial attractiveness predictor with cascaded fine-tuning deep learning model. arXiv:1511.02465
Yan M, Duan Y, Deng S, Zhu W, Wu X (2016) Facial beauty assessment under unconstrained conditions. In: 2016 8th international conference on electronics, computers and artificial intelligence (ECAI), pp 1–6
Zhang D, Chen F, Xu Y (2016) Computer models for facial beauty analysis. Springer, Berlin
Zhang D, Zhao Q, Chen F (2011) Quantitative analysis of human facial beauty using geometric features. Pattern Recogn 44:940–950
Zhou D, Bousquet O, Lal TN, Weston J, Schölkopf B (2004) Learning with local and global consistency. In: Advances in neural information processing systems, pp 321–328
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dornaika, F., Elorza, A., Wang, K. et al. Image-based face beauty analysis via graph-based semi-supervised learning. Multimed Tools Appl 79, 3005–3030 (2020). https://doi.org/10.1007/s11042-019-08206-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-08206-8