Automatic and Objective Facial Palsy Grading Index Prediction Using Deep Feature Regression

Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1248)


One of the main reasons for a half-sided facial paralysis is a dysfunction of the facial nerve. Physicians have to assess such a unilateral facial palsy with the help of standardized grading scales to evaluate the treatment. However, such assessments are usually very subjective and they are prone to variance and inconsistency between physicians due to their varying experience. We propose an automatic non-biased method using deep features combined with a linear regression method for facial palsy grading index prediction. With an extension of the free software tool Auto-eFace we annotated images of facial palsy patients and healthy subjects according to a common facial palsy grading scale. In our experiments, we obtained an average grading error of 11%.


Automatic assessment Facial palsy Regression Deep learning 



The research was supported by grant DE 735/15-1 and GU 463/12-1 of the German Research Foundation (DFG). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of Titan Xp GPUs used for this research.


  1. 1.
    Banks, C.A., Bhama, P.K., Park, J., Hadlock, C.R., Hadlock, T.A.: Clinician-graded electronic facial paralysis assessment: the eface. Plast. Reconstr. Surg. 136(2), 223e–230e (2015)CrossRefGoogle Scholar
  2. 2.
    Barbosa, J., et al.: Efficient quantitative assessment of facial paralysis using iris segmentation and active contour-based key points detection with hybrid classifier. BMC Med. Imaging 16(1), 23 (2016)CrossRefGoogle Scholar
  3. 3.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  4. 4.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  5. 5.
    Daugman, J.: How iris recognition works. In: The Essential Guide to Image Processing, pp. 715–739. Elsevier (2009)Google Scholar
  6. 6.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)Google Scholar
  7. 7.
    Donahue, J., et al.: Decaf: a deep convolutional activation feature for generic visual recognition. In: International Conference on Machine Learning, pp. 647–655 (2014)Google Scholar
  8. 8.
    Finsterer, J.: Management of peripheral facial nerve palsy. Eur. Arch. Otorhinolaryngol. 265(7), 743–752 (2008)CrossRefGoogle Scholar
  9. 9.
    Gaber, A., Faher, M.F., Wahed, M.A.: Automated grading of facial paralysis using the kinect v2: a proof of concept study. In: International Conference on Virtual Rehabilitation, pp. 258–264. IEEE (2015)Google Scholar
  10. 10.
    Gaber, A., Taher, M.F., Wahed, M.A.: Quantifying facial paralysis using the kinect v2. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2497–2501. IEEE (2015)Google Scholar
  11. 11.
    Guarin, D.L., Dusseldorp, J., Hadlock, T.A., Jowett, N.: A machine learning approach for automated facial measurements in facial palsy. JAMA Facial Plast. Surg. 20(4), 335–337 (2018)CrossRefGoogle Scholar
  12. 12.
    Guarin, D.L., et al.: Toward an automatic system for computer-aided assessment in facial palsy. arXiv preprint arXiv:1910.11497 (2019)
  13. 13.
    Haase, D., Kemmler, M., Guntinas-Lichius, O., Denzler, J.: Efficient measuring of facial action unit activation intensities using active appearance models. In: Machine Vision Applications, pp. 141–144 (2013)Google Scholar
  14. 14.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778. IEEE (2016)Google Scholar
  15. 15.
    House, J.W., Brackmann, D.E.: Facial nerve grading system. Otolaryngol. Head Neck Surg. 93(2), 146–147 (1985)CrossRefGoogle Scholar
  16. 16.
    Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  17. 17.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)Google Scholar
  18. 18.
    King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  19. 19.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105 (2012)Google Scholar
  20. 20.
    Lankton, S., Tannenbaum, A.: Localizing region-based active contours. IEEE Trans. Image Process. 17(11), 2029–2039 (2008)MathSciNetCrossRefGoogle Scholar
  21. 21.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  22. 22.
    Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear CNN models for fine-grained visual recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1449–1457 (2015)Google Scholar
  23. 23.
    Modersohn, L., Denzler, J.: Facial paresis index prediction by exploiting active appearance models for compact discriminative features. In: International Conference on Computer Vision Theory and Applications, pp. 271–278 (2016)Google Scholar
  24. 24.
    Morales, D.R., Donnan, P.T., Daly, F., Staa, T.V., Sullivan, F.M.: Impact of clinical trial findings on bell’s palsy management in general practice in the uk 2001–2012: interrupted time series regression analysis. BMJ Open 3(7), e003121 (2013)CrossRefGoogle Scholar
  25. 25.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference, p. 6 (2015)Google Scholar
  26. 26.
    Plumbaum, K., et al.: Inpatient treatment of patients with acute idiopathic peripheral facial palsy: a population-based healthcare research study. Clin. Otolaryngol. 42(6), 1267–1274 (2017)CrossRefGoogle Scholar
  27. 27.
    Ross, B.G., Fradet, G., Nedzelski, J.M.: Development of a sensitive clinical facial grading system. Otolaryngol. Head Neck Surg. 114(3), 380–386 (1996)CrossRefGoogle Scholar
  28. 28.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: IEEE International Conference on Computer Vision - Workshops, pp. 397–403 (2013)Google Scholar
  29. 29.
    Schaede, R.A., Volk, G.F., Modersohn, L., Barth, J.M., Denzler, J., Guntinas-Lichius, O.: Video instruction for synchronous video recording of mimic movement of patients with facial palsy. Laryngo-Rhino-Otologie (2017)Google Scholar
  30. 30.
    Simon, M., Rodner, E.: Neural activation constellations: unsupervised part model discovery with convolutional networks. In: IEEE International Conference on Computer Vision, pp. 1143–1151. IEEE (2015)Google Scholar
  31. 31.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  32. 32.
    Stennert, E., Limberg, C., Frentrup, K.: An index for paresis and defective healing-an easily applied method for objectively determining therapeutic results in facial paresis (author’s transl). HNO 25(7), 238–245 (1977)Google Scholar
  33. 33.
    Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, Long Beach, California, USA, 09–15 June 2019, vol. 97, pp. 6105–6114 (2019)Google Scholar
  34. 34.
    Thielker, J., Geißler, K., Granitzka, T., Klingner, C., Volk, G., Guntinas-Lichius, O.: Acute management of bell’s palsy. Curr. Otorhinolaryngol. Rep. 6(2), 161–170 (2018)CrossRefGoogle Scholar
  35. 35.
    Vapnik, V.N.: The Nature of Statistical Learning Theory. Springer, New York (1995). Scholar
  36. 36.
    Volk, G.F., et al.: Functional outcome and quality of life after hypoglossal-facial jump nerve suture. Front. Surg. 7, 11 (2020)CrossRefGoogle Scholar
  37. 37.
    Volk, G.F., et al.: Reliability of grading of facial palsy using a video tutorial with synchronous video recording. The Laryngoscope 129(10), 2274–2279 (2019)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Computer Vision GroupFriedrich Schiller University JenaJenaGermany
  2. 2.Department of OtorhinolaryngologyJena University HospitalJenaGermany

Personalised recommendations