Abstract
Generating and recognizing facial expressions has numerous applications, however, those are limited by the scarcity of datasets containing labeled nuanced expressions. In this paper, we describe the use of Delaunay triangulation combined with simple morphing techniques to blend images of faces, which allows us to create and automatically label facial expressions portraying controllable intensities of emotion. We have applied this approach on the RafD dataset consisting of 67 participants and 8 categorical emotions and evaluated the augmentation in a facial expression generation and recognition tasks using deep learning models. For the generation task, we used a deconvolution neural network which learns to encode the input images in a high-dimensional feature space and generate realistic expressions at varying intensities. The augmentation significantly improves the quality of images compared to previous comparable experiments and it allows to create images with a higher resolution. For the recognition task, we evaluated pre-trained Densenet121 and Resnet50 networks with either the original or augmented dataset. Our results indicate that the augmentation alone has a similar or better performance compared to the original. Implications of this method and its role in improving existing facial expression generation and recognition approaches are discussed.
This work has been supported by AffecTech: Personal Technologies for Affective Health, Innovative Training Network funded by the H2020 People Programme under Marie Skłodowska-Curie grant agreement No 722022.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cogsdill, E.J., Todorov, A.T., Spelke, E.S., Banaji, M.R.: Inferring character from faces: a developmental study. Psychol. Sci. 25(5), 1132–1139 (2014)
Buck, R.: Social and emotional functions in facial expression and communication: the readout hypothesis. Biol. Psychol. 38(2–3), 95–115 (1994)
Kruszka, P., et al.: 22q11. 2 deletion syndrome in diverse populations. Am. J. Med. Genet. Part A 173(4), 879–888 (2017)
Chang, K.I., Bowyer, K.W., Flynn, P.J.: Multiple nose region matching for 3D face recognition under varying facial expression. IEEE Trans. Pattern Anal. Mach. Intell. 28(10), 1695–1700 (2006)
Valev, H., Leufkens, T., Sas, C., Westerink, J., Dotsch, R.: Evaluation of a self-report system for assessing mood using facial expressions. In: Cipresso, P., Serino, S., Villani, D. (eds.) MindCare 2019. LNICST, vol. 288, pp. 231–241. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25872-6_19
Sanches, P., et al.: HCI and affective health: taking stock of a decade of studies and charting future research directions. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–17 (2019)
Colombo, D., et al.: The need for change: understanding emotion regulation antecedents and consequences using ecological momentary assessment. Emotion 20(1), 30 (2020)
Alfaras, M., et al.: From biodata to somadata. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2020)
Hu, L., et al.: Avatar digitization from a single image for real-time rendering. ACM Trans. Graph. (ToG) 36(6), 1–14 (2017)
Gallucci, A., Znamenskiy, D., Petkovic, M.: Prediction of 3D body parts from face shape and anthropometric measurements. J. Image Graph. 8(3), 67–74 (2020)
Lombardi, S., Saragih, J., Simon, T., Sheikh, Y.: Deep appearance models for face rendering. ACM Trans. Graph. (TOG) 37(4), 1–13 (2018)
Chu, C., Zhmoginov, A., Sandler, M.: Cyclegan, a master of steganography. arXiv preprint arXiv:1712.02950 (2017)
Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 8789–8797 (2018)
Ding, H., Sricharan, K., Chellappa, R.: Exprgan: Facial expression editing with controllable expression intensity. arXiv preprint arXiv:1709.03842 (2017)
Pumarola, A., Agudo, A., Martinez, A.M., Sanfeliu, A., Moreno-Noguer, F.: Ganimation: anatomically-aware facial animation from a single image. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 818–833 (2018)
Rychlowska, M., Jack, R.E., Garrod, O.G., Schyns, P.G., Martin, J.D., Niedenthal, P.M.: Functional smiles: tools for love, sympathy, and war. Psychol. Sci. 28(9), 1259–1270 (2017)
Yitzhak, N., et al.: Gently does it: humans outperform a software classifier in recognizing subtle, nonstereotypical facial expressions. Emotion 17(8), 1187 (2017)
Le Ngo, A.C., Phan, R.C.-W., See, J.: Spontaneous subtle expression recognition: imbalanced databases and solutions. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 33–48. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_3
Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H., Hawk, S.T., Van Knippenberg, A.: Presentation and validation of the radboud faces database. Cogn. Emotion 24(8), 1377–1388 (2010)
Lee, D.-T., Schachter, B.J.: Two algorithms for constructing a delaunay triangulation. Int. J. Comput. Inf. Sci. 9(3), 219–242 (1980)
Dosovitskiy, A., Springenberg, J.T., Tatarchenko, M., Brox, T.: Learning to generate chairs, tables and cars with convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 692–705 (2016)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Ravuri, S., Vinyals, O.: Classification accuracy score for conditional generative models. In: Advances in Neural Information Processing Systems, pp. 12268–12279 (2019)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Valev, H., Gallucci, A., Leufkens, T., Westerink, J., Sas, C. (2021). Applying Delaunay Triangulation Augmentation for Deep Learning Facial Expression Generation and Recognition. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12663. Springer, Cham. https://doi.org/10.1007/978-3-030-68796-0_53
Download citation
DOI: https://doi.org/10.1007/978-3-030-68796-0_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68795-3
Online ISBN: 978-3-030-68796-0
eBook Packages: Computer ScienceComputer Science (R0)
