European Archives of Oto-Rhino-Laryngology

, Volume 272, Issue 5, pp 1259–1267 | Cite as

Automated and objective action coding of facial expressions in patients with acute facial palsy

  • Daniel Haase
  • Laura Minnigerode
  • Gerd Fabian Volk
  • Joachim Denzler
  • Orlando Guntinas-LichiusEmail author


Aim of the present observational single center study was to objectively assess facial function in patients with idiopathic facial palsy with a new computer-based system that automatically recognizes action units (AUs) defined by the Facial Action Coding System (FACS). Still photographs using posed facial expressions of 28 healthy subjects and of 299 patients with acute facial palsy were automatically analyzed for bilateral AU expression profiles. All palsies were graded with the House–Brackmann (HB) grading system and with the Stennert Index (SI). Changes of the AU profiles during follow-up were analyzed for 77 patients. The initial HB grading of all patients was 3.3 ± 1.2. SI at rest was 1.86 ± 1.3 and during motion 3.79 ± 4.3. Healthy subjects showed a significant AU asymmetry score of 21 ± 11 % and there was no significant difference to patients (p = 0.128). At initial examination of patients, the number of activated AUs was significantly lower on the paralyzed side than on the healthy side (p < 0.0001). The final examination for patients took place 4 ± 6 months post baseline. The number of activated AUs and the ratio between affected and healthy side increased significantly between baseline and final examination (both p < 0.0001). The asymmetry score decreased between baseline and final examination (p < 0.0001). The number of activated AUs on the healthy side did not change significantly (p = 0.779). Radical rethinking in facial grading is worthwhile: automated FACS delivers fast and objective global and regional data on facial motor function for use in clinical routine and clinical trials.


Facial nerve Mimic muscles Asymmetry Facial Action Coding System Grading Palsy 



We thank Astrid Wetzel (Media Center, Jena University Hospital) for the photographs of all healthy subjects and patients. We thank Wolfgang H. Miltner (Department of Biological and Clinical Psychology, Friedrich Schiller University Jena) for critical reading of the manuscript.

Conflict of interest

The authors indicate that they have no conflict of interest.

Supplementary material

405_2014_3385_MOESM1_ESM.jpg (1.8 mb)
Supplement Fig. 1. Standard set of photographs of facial expression for each healthy subject and patient. (JPEG 1883 kb)
405_2014_3385_MOESM2_ESM.jpg (430 kb)
Supplement Fig. 2. Example of automated feature point localization on the healthy side (A) and on the paralyzed side (B; mirrored to the right side!) in the standardized photography with the patient showing his teeth
405_2014_3385_MOESM3_ESM.docx (69 kb)
Supplementary material 3 (DOCX 69 kb)


  1. 1.
    Linstrom CJ (2002) Objective facial motion analysis in patients with facial nerve dysfunction. The Laryngoscope 112:1129–1147CrossRefPubMedGoogle Scholar
  2. 2.
    House JW, Brackmann DE (1985) Facial nerve grading system. Otolaryngol Head Neck Surg 93:146–147PubMedGoogle Scholar
  3. 3.
    Ross BG, Fradet G, Nedzelski JM (1996) Development of a sensitive clinical facial grading system. Otolaryngol Head Neck Surg 114:380–386CrossRefPubMedGoogle Scholar
  4. 4.
    Vrabec JT, Backous DD, Djalilian HR et al (2009) Facial Nerve Grading System 2.0. Otolaryngol Head Neck Surg 140:445–450CrossRefPubMedGoogle Scholar
  5. 5.
    Stennert E, Fisch U (1977) Facial nerve paralysis scoring system. In: Facial Nerve Surgery. Aesculapius Publishing, ZurichGoogle Scholar
  6. 6.
    Hato N, Fujiwara T, Gyo K et al (2014) Yanagihara facial nerve grading system as a prognostic tool in Bell’s Palsy. Otol Neurot 35:1669–1672CrossRefGoogle Scholar
  7. 7.
    Lee LN, Susarla SM, HH M et al (2013) A comparison of facial nerve grading systems. Ann Plast Surg 70:313–316PubMedGoogle Scholar
  8. 8.
    Berg T, Marsk E, Engstrom M et al (2009) The effect of study design and analysis methods on recovery rates in Bell’s palsy. Laryngoscope 119:2046–2050CrossRefPubMedGoogle Scholar
  9. 9.
    Neely JG, Cheung JY, Wood M et al (1992) Computerized quantitative dynamic analysis of facial motion in the paralyzed and synkinetic face. Am J Otol 13:97–107PubMedGoogle Scholar
  10. 10.
    Frey M, Jenny A, Giovanoli P et al (1994) Development of a new documentation system for facial movements as a basis for the international registry for neuromuscular reconstruction in the face. Plast Reconstruct Surg 93:1334–1349CrossRefGoogle Scholar
  11. 11.
    Meier-Gallati V, Scriba H, Fisch U (1998) Objective scaling of facial nerve function based on area analysis (OSCAR). Otolaryngol Head Neck Surg 118:545–550PubMedGoogle Scholar
  12. 12.
    Wachtman GS, Cohn JF, Vanswearingen JM et al (2001) Automated tracking of facial features in patients with facial neuromuscular dysfunction. Plast Reconstruct Surg 107:1124–1133CrossRefGoogle Scholar
  13. 13.
    Hadlock TA, Urban LS (2012) Toward a universal, automated facial measurement tool in facial reanimation. Arch Facial Plast Surg 14:277–282CrossRefPubMedGoogle Scholar
  14. 14.
    Neely JG, Wang KX, Shapland CA et al (2010) Computerized objective measurement of facial motion: normal variation and test-retest reliability. Otol Neurotol 31:1488–1492PubMedGoogle Scholar
  15. 15.
    O’reilly BF, Soraghan JJ, Mcgrenary S et al (2010) Objective method of assessing and presenting the House-Brackmann and regional grades of facial palsy by production of a facogram. Otol Neurotol 31:486–491CrossRefPubMedGoogle Scholar
  16. 16.
    Ekman P, Friesen WV (1978) Manual of the Facial Action Coding System (FACS). Consulting Psychologists Press, Palo AltoGoogle Scholar
  17. 17.
    Cohn JF, Zlochower AJ, Lien J et al (1999) Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. Psychophysiology 36:35–43CrossRefPubMedGoogle Scholar
  18. 18.
    Hamm J, Kohler CG, Gur RC et al (2011) Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders. J Neurosci Meth 200:237–256CrossRefGoogle Scholar
  19. 19.
    Rogers CR, Schmidt KL, Vanswearingen JM et al (2007) Automated facial image analysis: detecting improvement in abnormal facial movement after treatment with botulinum toxin A. Ann Plast Surg 58:39–47CrossRefPubMedGoogle Scholar
  20. 20.
    Haase D, Kemmler M, Guntinas Lichius O et al (2012) Measuring Facial Action Unit Activation Intensities using Active Appearance Models. In: German Association for Pattern Recognition (DAGM) Conference, August 28–31, 2012. Graz, AustriaGoogle Scholar
  21. 21.
    Volk GF, Klingner C, Finkensieper M et al (2013) Prognostication of recovery time after acute peripheral facial palsy: a prospective cohort study. BMJ Open 3. doi: 10.1136/bmjopen-2013-003007 (pii: e003007)Google Scholar
  22. 22.
    Grosheva M, Wittekindt C, Guntinas-Lichius O (2008) Prognostic value of electroneurography and electromyography in facial palsy. Laryngoscope 118:394–397CrossRefPubMedGoogle Scholar
  23. 23.
    Sullivan FM, Swan IR, Donnan PT et al (2007) Early treatment with prednisolone or acyclovir in Bell’s palsy. New Engl J Med 357:1598–1607CrossRefPubMedGoogle Scholar
  24. 24.
    Stennert E, Limberg CH, Frentrup KP (1977) An index for paresis and defective healing: an easily applied method for objectively determining therapeutic results in facial paresis (author’s transl). HNO 25:238–245PubMedGoogle Scholar
  25. 25.
    Cootes TF, Edwards CA (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intel 23:681–685CrossRefGoogle Scholar
  26. 26.
    Lucey P, Cohn JF, Kanade T et al (2010) The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. Comput Vision Pattern Recog Workshops:94–101Google Scholar
  27. 27.
    Matthews I, Baker S (2004) Active appearance models revisited. Int J Comput Vis 60:135–164CrossRefGoogle Scholar
  28. 28.
    Rasmussen CE, Williams CKI (2005) Gaussian processes for machine learning. MIT Press, CambridgeGoogle Scholar
  29. 29.
    Ashraf AB, Lucey S, Cohn JF et al (2009) The painful face-pain expression recognition using active appearance models. Image Vision Comput 27:1788–1796CrossRefGoogle Scholar
  30. 30.
    Ekman P (1993) Facial expression and emotion. Am Psychol 48:384–392CrossRefPubMedGoogle Scholar
  31. 31.
    Van Gelder RS, Borod JC (1990) Neurobiological and cultural aspects of facial asymmetry. J Commun Dis 23:273–286CrossRefGoogle Scholar
  32. 32.
    Richardson CK, Bowers D, Bauer RM et al (2000) Digitizing the moving face during dynamic displays of emotion. Neuropsychologia 38:1028–1039CrossRefPubMedGoogle Scholar
  33. 33.
    Hager JC, Ekman P (2005) The asymmetry of facial actions is inconsistent with models of hemispheric specialization. In: Ekman P, Rosenberg EL (eds) What the face reveals. Oxford University Press, OxfordGoogle Scholar
  34. 34.
    Volk GF, Karamyan I, Klingner CM et al (2014) Quantitative magnetic resonance imaging volumetry of facial muscles in healthy patients with facial palsy. Plast Reconstruct Surg Glob Open 2(6):e173. doi: 10.1097/GOX.0000000000000128
  35. 35.
    Volk GF, Sauer M, Pohlmann M et al (2014) Reference values for dynamic facial muscle ultrasonography in adults. Muscle Nerve 50:348–357CrossRefPubMedGoogle Scholar
  36. 36.
    Schumann NP, Bongers K, Guntinas-Lichius O et al (2010) Facial muscle activation patterns in healthy male humans: a multi-channel surface EMG study. J Neuroscience Meth 187:120–128CrossRefGoogle Scholar
  37. 37.
    Kim SW, Heller ES, Hohman MH et al (2013) Detection and perceptual impact of side-to-side facial movement asymmetry. Facial Plast Surg 15:411–416CrossRefGoogle Scholar
  38. 38.
    Hohman MH, Kim SW, Heller ES et al (2014) Determining the threshold for asymmetry detection in facial expressions. Laryngoscope 124:860–865CrossRefPubMedGoogle Scholar
  39. 39.
    Engstrom M, Berg T, Stjernquist-Desatnik A et al (2008) Prednisolone and valaciclovir in Bell’s palsy: a randomised, double-blind, placebo-controlled, multicentre trial. Lancet Neurol 7:993–1000CrossRefPubMedGoogle Scholar
  40. 40.
    Heckmann JG, Lang C, Glocker FX et al (2012) The new S2 k AWMF guideline for the treatment of Bell’s palsy in commented short form. Laryngorhinootol 91:686–692CrossRefGoogle Scholar
  41. 41.
    Gronseth GS, Paduga R, American Academy Of N (2012) Evidence-based guideline update: steroids and antivirals for Bell palsy: report of the Guideline Development Subcommittee of the American Academy of Neurology. Neurology 79:2209–2213CrossRefPubMedGoogle Scholar
  42. 42.
    Baugh RF, Basura GJ, Ishii LE et al (2013) Clinical practice guideline: Bell’s palsy. Otolaryngol Head Neck Surg 149:S1–S27CrossRefPubMedGoogle Scholar
  43. 43.
    Deleyiannis FW, Askari M, Schmidt KL et al (2005) Muscle activity in the partially paralyzed face after placement of a fascial sling: a preliminary report. Ann Plast Surg 55:449–455CrossRefPubMedGoogle Scholar
  44. 44.
    He S, Soraghan JJ, O’reilly BF et al (2009) Quantitative analysis of facial paralysis using local binary patterns in biomedical videos. IEEE Trans Biomed Engin 56:1864–1870CrossRefGoogle Scholar
  45. 45.
    Sawai N, Hato N, Hakuba N et al (2012) Objective assessment of the severity of unilateral facial palsy using OKAO Vision® facial image analysis software. Acta Otolaryngol 132:1013–1017CrossRefPubMedGoogle Scholar
  46. 46.
    Mabvuure NT, Hallam MJ, Venables V et al (2013) Validation of a new photogrammetric technique to monitor the treatment effect of Botulinum toxin in synkinesis. Eye 27:860–864CrossRefPubMedCentralPubMedGoogle Scholar
  47. 47.
    Dulguerov P, Wang D, Perneger TV et al (2003) Videomimicography: the standards of normal revised. Arch Otolaryngol Head Neck Surg 129:960–965CrossRefPubMedGoogle Scholar
  48. 48.
    Ferrario VF, Sforza C (2007) Anatomy of emotion: a 3D study of facial mimicry. Eur J Histochem EJH 51(Suppl 1):45–52Google Scholar
  49. 49.
    Denlinger RL, Vanswearingen JM, Cohn JF et al (2008) Puckering and blowing facial expressions in people with facial movement disorders. Phys Ther 88:909–915CrossRefPubMedCentralPubMedGoogle Scholar
  50. 50.
    Beurskens CH, Oosterhof J, Nijhuis-Van Der Sanden MW (2010) Frequency and location of synkineses in patients with peripheral facial nerve paresis. Otol Neurotol 31:671–675PubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Daniel Haase
    • 1
  • Laura Minnigerode
    • 2
    • 3
  • Gerd Fabian Volk
    • 2
    • 3
  • Joachim Denzler
    • 1
  • Orlando Guntinas-Lichius
    • 2
    • 3
    Email author
  1. 1.Department of Computer ScienceFriedrich Schiller UniversityJenaGermany
  2. 2.Department of OtorhinolaryngologyUniversity Hospital JenaJenaGermany
  3. 3.Facial Nerve CenterUniversity Hospital JenaJenaGermany

Personalised recommendations