Skip to main content
SpringerLink
Log in

Improved Obstructed Facial Feature Reconstruction for Emotion Recognition with Minimal Change CycleGANs

  • Conference paper
  • First Online:
Advanced Concepts for Intelligent Vision Systems (ACIVS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14124))

  • 225 Accesses

Abstract

Comprehending facial expressions is essential for human interaction and closely linked to facial muscle understanding. Typically, muscle activation measurement involves electromyography (EMG) surface electrodes on the face. Consequently, facial regions are obscured by electrodes, posing challenges for computer vision algorithms to assess facial expressions. Conventional methods are unable to assess facial expressions with occluded features due to lack of training on such data. We demonstrate that a CycleGAN-based approach can restore occluded facial features without fine-tuning models and algorithms. By introducing the minimal change regularization term to the optimization problem for CycleGANs, we enhanced existing methods, reducing hallucinated facial features. We reached a correct emotion classification rate up to \(90\%\) for individual subjects. Furthermore, we overcome individual model limitations by training a single model for multiple individuals. This allows for the integration of EMG-based expression recognition with existing computer vision algorithms, enriching facial understanding and potentially improving the connection between muscle activity and expressions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abramian, D., Eklund, A.: Refacing: reconstructing anonymized facial features using GANS. 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1104–1108 (2019). https://doi.org/10.1109/ISBI.2019.8759515

  2. Arnold, D.: Selective surface electrostimulation of the denervated zygomaticus muscle. Diagnostics 11(2), 188 (2021). https://doi.org/10.3390/diagnostics11020188

    Article  Google Scholar 

  3. Büchner, T., Sickert, S., Volk, G.F., Anders, C., Guntinas-Lichius, O., Denzler, J.: Let’s get the FACS straight - reconstructing obstructed facial features. In: International Conference on Computer Vision Theory and Applications (VISAPP), pp. 727–736. SciTePress (2023). https://doi.org/10.5220/0011619900003417

  4. Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press (1978)

    Google Scholar 

  5. Fridlund, A.J., Cacioppo, J.T.: Guidelines for human electromyographic research. Psychophysiology 23(5), 567–589 (1986). https://doi.org/10.1111/j.1469-8986.1986.tb00676.x

    Article  Google Scholar 

  6. Goodfellow, I.J.,et al.: Generative Adversarial Networks. In: Advances in Neural Information Processing Systems, vol. 27 (2014). https://doi.org/10.48550/arXiv.1406.2661

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://doi.org/10.48550/arXiv.1512.03385

  8. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs Trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017). https://doi.org/10.48550/arXiv.1706.08500

  9. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017). https://doi.org/10.48550/arXiv.1611.07004

  10. Kammoun, A., Slama, R., Tabia, H., Ouni, T., Abid, M.: Generative adversarial networks for face generation: a survey. ACM Comput. Surv. 55(5), 1–37 (2023)

    Article  Google Scholar 

  11. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8107–8116 (2020). https://doi.org/10.1109/CVPR42600.2020.00813

  12. Klingner, C.M., Guntinas-Lichius, O.: Mimik und Emotion. Laryngo-Rhino-Otologie 102(S 01), S115–S125 (2023). https://doi.org/10.1055/a-2003-5687

  13. Kuramoto, E., Yoshinaga, S., Nakao, H., Nemoto, S., Ishida, Y.: Characteristics of facial muscle activity during voluntary facial expressions: imaging analysis of facial expressions based on myogenic potential data. Neuropsychopharmacol. Rep. 39(3), 183–193 (2019). https://doi.org/10.1002/npr2.12059

    Article  Google Scholar 

  14. Kurz, A., Volk, G.F., Arnold, D., Schneider-Stickler, B., Mayr, W., Guntinas-Lichius, O.: Selective electrical surface stimulation to support functional recovery in the early phase after unilateral acute facial nerve or vocal fold paralysis. Frontiers Neurol. 13, 869900 (2022)

    Article  Google Scholar 

  15. Li, Y., Liu, S., Yang, J., Yang, M.H.: Generative face completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3911–3919 (2017)

    Google Scholar 

  16. Liu, M., Li, Q., Qin, Z., Zhang, G., Wan, P., Zheng, W.: BlendGAN: implicitly GAN blending for arbitrary stylized face generation. In: Neural Information Processing Systems ( 2021)

    Google Scholar 

  17. Liu, S., Wei, Y., Lu, J., Zhou, J.: An improved evaluation framework for generative adversarial networks. CoRR (2018). https://doi.org/10.48550/arXiv.1803.07474

  18. Liu, Y., Li, Q., Sun, Z., Tan, T.: A 3 GAN: an attribute-aware attentive generative adversarial network for face aging. IEEE Trans. Inf. Forensics Secur. 16, 2776–2790 (2021). https://doi.org/10.1109/TIFS.2021.3065499

    Article  Google Scholar 

  19. Loyo, M., McReynold, M., Mace, J.C., Cameron, M.: Protocol for randomized controlled trial of electric stimulation with high-volt twin peak versus placebo for facial functional recovery from acute Bell’s palsy in patients with poor prognostic factors. J. Rehabil. Assistive Technol. Eng. 7, 2055668320964142 (2020). https://doi.org/10.1177/2055668320964142

  20. Luan, P., Huynh, V., Tuan Anh, T.: Facial expression recognition using residual masking network. In: IEEE 25th International Conference on Pattern Recognition, pp. 4513–4519 (2020)

    Google Scholar 

  21. Mueller, N., Trentzsch, V., Grassme, R., Guntinas-Lichius, O., Volk, G.F., Anders, C.: High-resolution surface electromyographic activities of facial muscles during mimic movements in healthy adults: a prospective observational study. Frontiers Hum. Neurosci. 16, 1029415 (2022)

    Article  Google Scholar 

  22. Taigman, Y., Polyak, A., Wolf, L.: Unsupervised cross-domain image generation. CoRR (2016). https://doi.org/10.48550/arXiv.1611.02200

  23. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. CoRR (2017). https://doi.org/10.48550/arXiv.1607.08022

  24. Wang, X., Guo, H., Hu, S., Chang, M.C., Lyu, S.: GAN-generated Faces Detection: a survey and new perspectives. CoRR (2022)

    Google Scholar 

  25. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  26. Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: a review. Med. Image Anal. 58, 101552 (2019). https://doi.org/10.1016/j.media.2019.101552

    Article  Google Scholar 

  27. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018). https://doi.org/10.48550/arXiv.1801.03924

  28. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2223–2232 (2017). https://doi.org/10.48550/arXiv.1703.10593

Download references

Acknowledgments

This work has been funded by the Deutsche Forschungsgemeinschaft (DFG - German Research Foundation) project 427899908 BRIDGING THE GAP: MIMICS AND MUSCLES (DE 735/15-1 and GU 463/12-1).

Author information

Authors and Affiliations

  1. Computer Vision Group, Friedrich Schiller University Jena, 07745, Jena, Germany

    Tim Büchner & Joachim Denzler

  2. Department of Otorhinolaryngology, Jena University Hospital, 07747, Jena, Germany

    Orlando Guntinas-Lichius

Authors
  1. Tim Büchner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Orlando Guntinas-Lichius
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joachim Denzler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tim Büchner .

Editor information

Editors and Affiliations

  1. DGA TA, Toulouse, France

    Jaques Blanc-Talon

  2. University of Auckland, Auckland, New Zealand

    Patrice Delmas

  3. Ghent University, Ghent, Belgium

    Wilfried Philips

  4. University of Antwerp, Wilrijk, Belgium

    Paul Scheunders

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Büchner, T., Guntinas-Lichius, O., Denzler, J. (2023). Improved Obstructed Facial Feature Reconstruction for Emotion Recognition with Minimal Change CycleGANs. In: Blanc-Talon, J., Delmas, P., Philips, W., Scheunders, P. (eds) Advanced Concepts for Intelligent Vision Systems. ACIVS 2023. Lecture Notes in Computer Science, vol 14124. Springer, Cham. https://doi.org/10.1007/978-3-031-45382-3_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45382-3_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45381-6

  • Online ISBN: 978-3-031-45382-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation