Skip to main content
SpringerLink
Log in

Automated and objective action coding of facial expressions in patients with acute facial palsy

  • Miscellaneous
  • Published:
European Archives of Oto-Rhino-Laryngology Aims and scope Submit manuscript

Abstract

Aim of the present observational single center study was to objectively assess facial function in patients with idiopathic facial palsy with a new computer-based system that automatically recognizes action units (AUs) defined by the Facial Action Coding System (FACS). Still photographs using posed facial expressions of 28 healthy subjects and of 299 patients with acute facial palsy were automatically analyzed for bilateral AU expression profiles. All palsies were graded with the House–Brackmann (HB) grading system and with the Stennert Index (SI). Changes of the AU profiles during follow-up were analyzed for 77 patients. The initial HB grading of all patients was 3.3 ± 1.2. SI at rest was 1.86 ± 1.3 and during motion 3.79 ± 4.3. Healthy subjects showed a significant AU asymmetry score of 21 ± 11 % and there was no significant difference to patients (p = 0.128). At initial examination of patients, the number of activated AUs was significantly lower on the paralyzed side than on the healthy side (p < 0.0001). The final examination for patients took place 4 ± 6 months post baseline. The number of activated AUs and the ratio between affected and healthy side increased significantly between baseline and final examination (both p < 0.0001). The asymmetry score decreased between baseline and final examination (p < 0.0001). The number of activated AUs on the healthy side did not change significantly (p = 0.779). Radical rethinking in facial grading is worthwhile: automated FACS delivers fast and objective global and regional data on facial motor function for use in clinical routine and clinical trials.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Linstrom CJ (2002) Objective facial motion analysis in patients with facial nerve dysfunction. The Laryngoscope 112:1129–1147

    Article  PubMed  Google Scholar 

  2. House JW, Brackmann DE (1985) Facial nerve grading system. Otolaryngol Head Neck Surg 93:146–147

    CAS  PubMed  Google Scholar 

  3. Ross BG, Fradet G, Nedzelski JM (1996) Development of a sensitive clinical facial grading system. Otolaryngol Head Neck Surg 114:380–386

    Article  CAS  PubMed  Google Scholar 

  4. Vrabec JT, Backous DD, Djalilian HR et al (2009) Facial Nerve Grading System 2.0. Otolaryngol Head Neck Surg 140:445–450

    Article  PubMed  Google Scholar 

  5. Stennert E, Fisch U (1977) Facial nerve paralysis scoring system. In: Facial Nerve Surgery. Aesculapius Publishing, Zurich

  6. Hato N, Fujiwara T, Gyo K et al (2014) Yanagihara facial nerve grading system as a prognostic tool in Bell’s Palsy. Otol Neurot 35:1669–1672

    Article  Google Scholar 

  7. Lee LN, Susarla SM, HH M et al (2013) A comparison of facial nerve grading systems. Ann Plast Surg 70:313–316

    CAS  PubMed  Google Scholar 

  8. Berg T, Marsk E, Engstrom M et al (2009) The effect of study design and analysis methods on recovery rates in Bell’s palsy. Laryngoscope 119:2046–2050

    Article  PubMed  Google Scholar 

  9. Neely JG, Cheung JY, Wood M et al (1992) Computerized quantitative dynamic analysis of facial motion in the paralyzed and synkinetic face. Am J Otol 13:97–107

    CAS  PubMed  Google Scholar 

  10. Frey M, Jenny A, Giovanoli P et al (1994) Development of a new documentation system for facial movements as a basis for the international registry for neuromuscular reconstruction in the face. Plast Reconstruct Surg 93:1334–1349

    Article  CAS  Google Scholar 

  11. Meier-Gallati V, Scriba H, Fisch U (1998) Objective scaling of facial nerve function based on area analysis (OSCAR). Otolaryngol Head Neck Surg 118:545–550

    CAS  PubMed  Google Scholar 

  12. Wachtman GS, Cohn JF, Vanswearingen JM et al (2001) Automated tracking of facial features in patients with facial neuromuscular dysfunction. Plast Reconstruct Surg 107:1124–1133

    Article  CAS  Google Scholar 

  13. Hadlock TA, Urban LS (2012) Toward a universal, automated facial measurement tool in facial reanimation. Arch Facial Plast Surg 14:277–282

    Article  PubMed  Google Scholar 

  14. Neely JG, Wang KX, Shapland CA et al (2010) Computerized objective measurement of facial motion: normal variation and test-retest reliability. Otol Neurotol 31:1488–1492

    PubMed  Google Scholar 

  15. O’reilly BF, Soraghan JJ, Mcgrenary S et al (2010) Objective method of assessing and presenting the House-Brackmann and regional grades of facial palsy by production of a facogram. Otol Neurotol 31:486–491

    Article  PubMed  Google Scholar 

  16. Ekman P, Friesen WV (1978) Manual of the Facial Action Coding System (FACS). Consulting Psychologists Press, Palo Alto

    Google Scholar 

  17. Cohn JF, Zlochower AJ, Lien J et al (1999) Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. Psychophysiology 36:35–43

    Article  CAS  PubMed  Google Scholar 

  18. Hamm J, Kohler CG, Gur RC et al (2011) Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Meth 200:237–256

    Article  Google Scholar 

  19. Rogers CR, Schmidt KL, Vanswearingen JM et al (2007) Automated facial image analysis: detecting improvement in abnormal facial movement after treatment with botulinum toxin A. Ann Plast Surg 58:39–47

    Article  CAS  PubMed  Google Scholar 

  20. Haase D, Kemmler M, Guntinas Lichius O et al (2012) Measuring Facial Action Unit Activation Intensities using Active Appearance Models. In: German Association for Pattern Recognition (DAGM) Conference, August 28–31, 2012. Graz, Austria

  21. Volk GF, Klingner C, Finkensieper M et al (2013) Prognostication of recovery time after acute peripheral facial palsy: a prospective cohort study. BMJ Open 3. doi:10.1136/bmjopen-2013-003007 (pii: e003007)

    Google Scholar 

  22. Grosheva M, Wittekindt C, Guntinas-Lichius O (2008) Prognostic value of electroneurography and electromyography in facial palsy. Laryngoscope 118:394–397

    Article  PubMed  Google Scholar 

  23. Sullivan FM, Swan IR, Donnan PT et al (2007) Early treatment with prednisolone or acyclovir in Bell’s palsy. New Engl J Med 357:1598–1607

    Article  CAS  PubMed  Google Scholar 

  24. Stennert E, Limberg CH, Frentrup KP (1977) An index for paresis and defective healing: an easily applied method for objectively determining therapeutic results in facial paresis (author’s transl). HNO 25:238–245

    CAS  PubMed  Google Scholar 

  25. Cootes TF, Edwards CA (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intel 23:681–685

    Article  Google Scholar 

  26. Lucey P, Cohn JF, Kanade T et al (2010) The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. Comput Vision Pattern Recog Workshops:94–101

  27. Matthews I, Baker S (2004) Active appearance models revisited. Int J Comput Vis 60:135–164

    Article  Google Scholar 

  28. Rasmussen CE, Williams CKI (2005) Gaussian processes for machine learning. MIT Press, Cambridge

    Google Scholar 

  29. Ashraf AB, Lucey S, Cohn JF et al (2009) The painful face-pain expression recognition using active appearance models. Image Vision Comput 27:1788–1796

    Article  Google Scholar 

  30. Ekman P (1993) Facial expression and emotion. Am Psychol 48:384–392

    Article  CAS  PubMed  Google Scholar 

  31. Van Gelder RS, Borod JC (1990) Neurobiological and cultural aspects of facial asymmetry. J Commun Dis 23:273–286

    Article  Google Scholar 

  32. Richardson CK, Bowers D, Bauer RM et al (2000) Digitizing the moving face during dynamic displays of emotion. Neuropsychologia 38:1028–1039

    Article  CAS  PubMed  Google Scholar 

  33. Hager JC, Ekman P (2005) The asymmetry of facial actions is inconsistent with models of hemispheric specialization. In: Ekman P, Rosenberg EL (eds) What the face reveals. Oxford University Press, Oxford

    Google Scholar 

  34. Volk GF, Karamyan I, Klingner CM et al (2014) Quantitative magnetic resonance imaging volumetry of facial muscles in healthy patients with facial palsy. Plast Reconstruct Surg Glob Open 2(6):e173. doi:10.1097/GOX.0000000000000128

  35. Volk GF, Sauer M, Pohlmann M et al (2014) Reference values for dynamic facial muscle ultrasonography in adults. Muscle Nerve 50:348–357

    Article  PubMed  Google Scholar 

  36. Schumann NP, Bongers K, Guntinas-Lichius O et al (2010) Facial muscle activation patterns in healthy male humans: a multi-channel surface EMG study. J Neuroscience Meth 187:120–128

    Article  Google Scholar 

  37. Kim SW, Heller ES, Hohman MH et al (2013) Detection and perceptual impact of side-to-side facial movement asymmetry. Facial Plast Surg 15:411–416

    Article  Google Scholar 

  38. Hohman MH, Kim SW, Heller ES et al (2014) Determining the threshold for asymmetry detection in facial expressions. Laryngoscope 124:860–865

    Article  PubMed  Google Scholar 

  39. Engstrom M, Berg T, Stjernquist-Desatnik A et al (2008) Prednisolone and valaciclovir in Bell’s palsy: a randomised, double-blind, placebo-controlled, multicentre trial. Lancet Neurol 7:993–1000

    Article  PubMed  Google Scholar 

  40. Heckmann JG, Lang C, Glocker FX et al (2012) The new S2 k AWMF guideline for the treatment of Bell’s palsy in commented short form. Laryngorhinootol 91:686–692

    Article  CAS  Google Scholar 

  41. Gronseth GS, Paduga R, American Academy Of N (2012) Evidence-based guideline update: steroids and antivirals for Bell palsy: report of the Guideline Development Subcommittee of the American Academy of Neurology. Neurology 79:2209–2213

    Article  PubMed  Google Scholar 

  42. Baugh RF, Basura GJ, Ishii LE et al (2013) Clinical practice guideline: Bell’s palsy. Otolaryngol Head Neck Surg 149:S1–S27

    Article  PubMed  Google Scholar 

  43. Deleyiannis FW, Askari M, Schmidt KL et al (2005) Muscle activity in the partially paralyzed face after placement of a fascial sling: a preliminary report. Ann Plast Surg 55:449–455

    Article  CAS  PubMed  Google Scholar 

  44. He S, Soraghan JJ, O’reilly BF et al (2009) Quantitative analysis of facial paralysis using local binary patterns in biomedical videos. IEEE Trans Biomed Engin 56:1864–1870

    Article  Google Scholar 

  45. Sawai N, Hato N, Hakuba N et al (2012) Objective assessment of the severity of unilateral facial palsy using OKAO Vision® facial image analysis software. Acta Otolaryngol 132:1013–1017

    Article  PubMed  Google Scholar 

  46. Mabvuure NT, Hallam MJ, Venables V et al (2013) Validation of a new photogrammetric technique to monitor the treatment effect of Botulinum toxin in synkinesis. Eye 27:860–864

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  47. Dulguerov P, Wang D, Perneger TV et al (2003) Videomimicography: the standards of normal revised. Arch Otolaryngol Head Neck Surg 129:960–965

    Article  PubMed  Google Scholar 

  48. Ferrario VF, Sforza C (2007) Anatomy of emotion: a 3D study of facial mimicry. Eur J Histochem EJH 51(Suppl 1):45–52

    Google Scholar 

  49. Denlinger RL, Vanswearingen JM, Cohn JF et al (2008) Puckering and blowing facial expressions in people with facial movement disorders. Phys Ther 88:909–915

    Article  PubMed Central  PubMed  Google Scholar 

  50. Beurskens CH, Oosterhof J, Nijhuis-Van Der Sanden MW (2010) Frequency and location of synkineses in patients with peripheral facial nerve paresis. Otol Neurotol 31:671–675

    PubMed  Google Scholar 

Download references

Acknowledgments

We thank Astrid Wetzel (Media Center, Jena University Hospital) for the photographs of all healthy subjects and patients. We thank Wolfgang H. Miltner (Department of Biological and Clinical Psychology, Friedrich Schiller University Jena) for critical reading of the manuscript.

Conflict of interest

The authors indicate that they have no conflict of interest.

Author information

Authors and Affiliations

  1. Department of Computer Science, Friedrich Schiller University, Jena, Germany

    Daniel Haase & Joachim Denzler

  2. Department of Otorhinolaryngology, University Hospital Jena, Lessingstrasse 2, 07740, Jena, Germany

    Laura Minnigerode, Gerd Fabian Volk & Orlando Guntinas-Lichius

  3. Facial Nerve Center, University Hospital Jena, Jena, Germany

    Laura Minnigerode, Gerd Fabian Volk & Orlando Guntinas-Lichius

Authors
  1. Daniel Haase
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Laura Minnigerode
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerd Fabian Volk
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joachim Denzler
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Orlando Guntinas-Lichius
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Orlando Guntinas-Lichius.

Electronic supplementary material

Below is the link to the electronic supplementary material.

405_2014_3385_MOESM1_ESM.jpg

Supplement Fig. 1. Standard set of photographs of facial expression for each healthy subject and patient. (JPEG 1883 kb)

405_2014_3385_MOESM2_ESM.jpg

Supplement Fig. 2. Example of automated feature point localization on the healthy side (A) and on the paralyzed side (B; mirrored to the right side!) in the standardized photography with the patient showing his teeth

Supplementary material 3 (DOCX 69 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haase, D., Minnigerode, L., Volk, G.F. et al. Automated and objective action coding of facial expressions in patients with acute facial palsy. Eur Arch Otorhinolaryngol 272, 1259–1267 (2015). https://doi.org/10.1007/s00405-014-3385-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00405-014-3385-8

Keywords

Search

Navigation