Advertisement

The Utility of Facial Analysis Algorithms in Detecting Melancholia

Chapter

Abstract

Facial expressions reliably reflect an individual’s internal emotional state and form an important part of effective social interaction and communication. In clinical psychiatry, facial affect is routinely assessed, and any identified deviations from normal affective range and reactivity may signal the presence of a potential psychiatric disorder. An example is melancholic depression or ‘melancholia’ where facial immobility and non-reactivity are viewed as sensitive diagnostic indicators of the illness. However, affect in depressive disorders such as melancholia, and indeed psychiatric conditions more broadly, is largely assessed by clinicians, without biological or computational quantification. While such clinical assessment provides useful qualitative descriptors of illness features, the inherent subjectivity of this approach raises concerns regarding diagnostic reliability, and may hinder communication between clinicians. Methodological advances and algorithm development in the field of affective computing have the potential to overcome such limitations through objective characterization of facial features. Among these methods are implicit face analysis techniques, which are based on local spatio-temporal descriptors such as the space-time interest points and Bag-of-Words framework, and explicit face analysis techniques based on deformable model fitting methods such as Constrained Local Models and Active Appearance Models. In this chapter we overview these approaches and discuss their application toward detection and diagnosis of depressive disorders, in particular their capacity to delineate melancholia from the residual non-melancholic conditions.

Keywords

Facial Imaging Local Binary Pattern Interest Point Active Appearance Model Facial Part 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    S. Bloch, B.S. Singh, Foundations of Clinical Psychiatry, 2nd edn. (Melbourne University Publishing, Melbourne, 2001)Google Scholar
  2. 2.
    W. James, What is an emotion? Mind 9, 188–205 (1894)Google Scholar
  3. 3.
    R. Reisenzein, A short history of psychological perspectives on emotion, in The Oxford Handbook of Affective Computing, ed. by R. Calvo, S. D’Mello, J. Gratch, A. Kappas (Oxford University Press, Oxford, 2015)Google Scholar
  4. 4.
    M.B. Arnold, Emotion and Personality (Columbia University Press, New York, 1960)Google Scholar
  5. 5.
    R.S. Lazarus, Psychological Stress and the Coping Process (McGraw-Hill, New York, 1966)Google Scholar
  6. 6.
    G.E. Berrios, The History of Mental Symptoms: Descriptive Psychopathology Since the Nineteenth Century (Cambridge University Press, Cambridge, UK, 1996)CrossRefGoogle Scholar
  7. 7.
    W. McDougall, An Introduction to Social Psychology (Methuen, London, 1908/1960)Google Scholar
  8. 8.
    K.L. Davis, J. Panksepp, The brain’s emotional foundations of human personality and the affective neuroscience personality scales. Neurosci. Biobehav. Rev. 35, 1946–1958 (2011)CrossRefGoogle Scholar
  9. 9.
    S.S. Tomkins, Affect Imagery Consciousness: Volume I: The Positive Affects (Springer Publishing Company, New York, 1962)Google Scholar
  10. 10.
    S.S. Tomkins, Affect Imagery Consciousness: Volume II: The Negative Affects (Springer Publishing Company, New York, 1963)Google Scholar
  11. 11.
    S.S. Tomkins, Affect Imagery Consciousness: Volume III: Anger and Fear (Springer Publishing Company, New York, 1991)Google Scholar
  12. 12.
    C.E. Izard, The Face of Emotion (Appleton-Century Crofts, New York, 1971)Google Scholar
  13. 13.
    P. Ekman, Facial expression and emotion. Am. Psychol. 48, 384 (1993)CrossRefGoogle Scholar
  14. 14.
    C.E. Izard, Basic emotions, natural kinds, emotion schemas, and a new paradigm. Perspect. Psychol. Sci. 2, 260–280 (2007)CrossRefGoogle Scholar
  15. 15.
    J.A. Russell, A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)CrossRefGoogle Scholar
  16. 16.
    D. Keltner, P. Ekman, Facial expressions of emotions, in Handbook of Emotions, ed. by M. Lewis, J. Haviland-Jones, 2nd edn. (Guildford Press, New York, 2000)Google Scholar
  17. 17.
    P. Ekman, W.V. Friesen, The Facial Action Coding System: A Technique for the Measurement of Facial Movement (Consulting Psychologists, Palo Alto, 1978)Google Scholar
  18. 18.
    G. Parker, Classifying depression: should paradigms lost be regained? Am. J. Psychiatry 157, 1195–1203 (2000)CrossRefGoogle Scholar
  19. 19.
    APA, Diagnostic and Statistical Manual of Mental Disorders (DSM-III), 3rd edn. (American Psychiatric Association, Washington, DC, 1980)Google Scholar
  20. 20.
    E. Kraepelin, Manic-Depressive Insanity and Paranoia (E. & S. Livingstone, Edinburgh, 1921)Google Scholar
  21. 21.
    E. Mapother, Discussion on manic-depressive psychosis. Br. Med. J. ii, 872–879 (1926)Google Scholar
  22. 22.
    A. Lewis, Melancholia: a clinical survey of depressive states. J. Ment. Sci. 80, 277–378 (1934)Google Scholar
  23. 23.
    G. Parker, The Bonds of Depression (Angus & Robertson Publishers, Sydney, 1978)Google Scholar
  24. 24.
    L.G. Kiloh, R.F. Garside, The independence of neurotic depression and endogenous depression. Br. J. Psychiatry 109, 451–463 (1963)CrossRefGoogle Scholar
  25. 25.
    M.W.P. Carney, M. Roth, R.F. Garside, The diagnosis of depressive syndromes and the prediction of ECT response. Br. J. Psychiatry 111, 659–674 (1965)CrossRefGoogle Scholar
  26. 26.
    C. Gurney, M. Roth, R.F. Garside, T.A. Kerr, K. Schapira, Studies in the classification of affective disorders. The relationship between anxiety states and depressive illnesses. II. Br. J. Psychiatry 121, 162–166 (1972)CrossRefGoogle Scholar
  27. 27.
    G. Parker, P.D. Hadzi, Melancholia: A Disorder of Movement and Mood (Cambridge University Press, New York, 1996)CrossRefGoogle Scholar
  28. 28.
    G. Parker, D. Hadzi-Pavlovic, K. Wilhelm et al., Defining melancholia: properties of a refined sign-based measure. Br. J. Psychiatry 164, 316–326 (1994)CrossRefGoogle Scholar
  29. 29.
    C.M. Pariante, S.L. Lightman, The HPA axis in major depression: classical theories and new developments. Trends Neurosci. 31, 464–468 (2008)CrossRefGoogle Scholar
  30. 30.
    J.P. Herman, W.E. Cullinan, Neurocircuitry of stress: central control of the hypothalamo-pituitary-adrenocortical axis. Trends Neurosci. 20, 78–84 (1997)CrossRefGoogle Scholar
  31. 31.
    B.J. Carroll, G.C. Curtis, J. Mendels, Neuroendocrine regulation in depression. II. Discrimination of depressed from nondepressed patients. Arch. Gen. Psychiatry 33, 1051–1058 (1976)CrossRefGoogle Scholar
  32. 32.
    B.J. Carroll, M. Feinberg, J.F. Greden et al., A specific laboratory test for the diagnosis of melancholia. Standardization, validation, and clinical utility. Arch. Gen. Psychiatry 38, 15–22 (1981)CrossRefGoogle Scholar
  33. 33.
    E. Shorter, M. Fink, Endocrine Psychiatry: Solving the Riddle of Melancholia (Oxford University Press, New York, 2010)Google Scholar
  34. 34.
    G. Parker, M. Fink, E. Shorter et al., Issues for DSM-5: whither melancholia? The case for its classification as a distinct mood disorder. Am. J. Psychiatry 167, 745–747 (2010)CrossRefGoogle Scholar
  35. 35.
    M.P. Hyett, M.J. Breakspear, K.J. Friston, C.C. Guo, G.B. Parker, Disrupted effective connectivity of cortical systems supporting attention and interoception in melancholia. JAMA Psychiatry 72, 350–358 (2015)CrossRefGoogle Scholar
  36. 36.
    I. Cohen, N. Sebe, L. Chen, A. Garg, T.S. Huang, Facial expression recognition from video sequences: temporal and static modelling. Comput. Vis. Image Underst. 91, 160–187 (2003)CrossRefGoogle Scholar
  37. 37.
    B. Fasel, J. Luettin, Automatic facial expression analysis: a survey. Pattern Recogn. 36, 259–275 (2003)CrossRefzbMATHGoogle Scholar
  38. 38.
    M. Pantic, L.J.M. Rothkrantz, Facial action recognition for facial expression analysis from static face images. IEEE Trans. Syst. Man Cybern. B Cybern. 34, 1449–1461 (2004)CrossRefGoogle Scholar
  39. 39.
    E. Sariyanidi, H. Gunes, A. Cavallaro, Automatic analysis of facial affect: a survey of registration, representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37, 39–58 (2015)Google Scholar
  40. 40.
    Z. Zeng, M. Pantic, G.I. Roisman, T.S. Huang, A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)CrossRefGoogle Scholar
  41. 41.
    P.A. Viola, M.J. Jones, Rapid object detection using a boosted cascade of simple features, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2001) pp. 511–518Google Scholar
  42. 42.
    T.F. Cootes, G.J. Edwards, C.J. Taylor, Active appearance models, in Proceedings of the European Conference on Computer Vision (ECCV) (Springer,1998), pp. 484–498Google Scholar
  43. 43.
    Y. Freund, R.E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, in Computational Learning Theory, 23–27, Springer (Springer, 1995), pp. 23–27Google Scholar
  44. 44.
    M. Jones, P. Viola, Fast multi-view face detection, in Mitsubishi Electric Research Lab TR- 20003-96, vol. 3 (MERL, 2003), p. 14Google Scholar
  45. 45.
    M. Osadchy, Y.L. Cun, M.L. Miller, Synergistic face detection and pose estimation with energy-based models. J. Mach. Learn. Res. 8, 1197–1215 (2007)Google Scholar
  46. 46.
    C. Huang, H. Ai, Y. Li, S. Lao, Vector boosting for rotation invariant multi-view face detection, in Proceedings of the IEEE International Conference on Computer Vision (ICCV) 446–453. (IEEE, 2005), pp. 446–453Google Scholar
  47. 47.
    M. Kass, A. Witkin, D. Terzopoulos, Snakes: active contour models. Int. J. Comput. Vis. 1, 321–331 (1988)CrossRefzbMATHGoogle Scholar
  48. 48.
    T.F. Cootes, C.J. Taylor, D.H. Cooper, J. Graham, Active shape models-their training and application. Comput. Vis. Image Underst. 61, 38–59 (1995)CrossRefGoogle Scholar
  49. 49.
    S. Baker, I. Matthews, Equivalence and efficiency of image alignment algorithms, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2001), pp. 1090–1097Google Scholar
  50. 50.
    S. Baker, R. Gross, I. Matthews, Lucas-Kanade 20 Years on: A Unifying Framework: Part 3 (Carnegie Mellon University, RI, USA, 2003)Google Scholar
  51. 51.
    J. Saragih, R. Göcke, Iterative error bound minimisation for AAM alignment, in Proceedings of the International Conference on Pattern Recognition (ICPR) (IEEE, 2006), pp. 1192–1195Google Scholar
  52. 52.
    J. Saragih, R. Göcke, A Nonlinear discriminative approach to AAM fitting, in Proceedings of the IEEE International Conference on Computer Vision (ICCV) (IEEE, 2007), pp. 1–8Google Scholar
  53. 53.
    G. McIntyre, R. Goecke, M. Hyett, M. Green, M. Breakspear, An approach for automatically measuring facial activity in depressed subjects, in Proceedings of Affective Computing and Intelligent Interaction (ACII) (Springer, 2009), pp. 1–8Google Scholar
  54. 54.
    J.F. Cohn, T.S. Kreuz, I. Matthews, et al. Detecting depression from facial actions and vocal prosody, in Proceedings of Affective Computing and Intelligent Interaction (ACII), (Springer, 2009), pp. 1–7Google Scholar
  55. 55.
    J.M. Saragih, S. Lucey, J. Cohn, Face alignment through subspace constrained mean-shifts, in Proceedings of the IEEE International Conference of Computer Vision (ICCV), September (IEEE, 2009), pp. 1034–1041Google Scholar
  56. 56.
    S.W. Chew, P. Lucey, S. Lucey et al., In the pursuit of effective affective computing: the relationship between features and registration. IEEE Trans. Syst. Man Cybern. B Cybern. 42, 1006–1016 (2012)CrossRefGoogle Scholar
  57. 57.
    P.F. Felzenszwalb, D.P. Huttenlocher, Pictorial structures for object recognition. Int. J. Comput. Vis. 61, 55–79 (2005)CrossRefGoogle Scholar
  58. 58.
    M. Everingham, J. Sivic, A. Zisserman, Hello! My name is: Buffy” – automatic naming of characters in TV Video, in Proceedings of the British Machine and Vision Conference (BMVC) (Springer, 2006), pp. 899–908Google Scholar
  59. 59.
    X. Zhu, D. Ramanan, Face detection, pose estimation, and landmark localization in the wild, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2012), pp. 2879–2886Google Scholar
  60. 60.
    J. Joshi, A. Dhall, R. Goecke, M. Breakspear, G. Parker, Neural-net classification for spatio-temporal descriptor based depression analysis, in Proceedings of the International Conference on Pattern Recognition (ICPR) (IAPR, 2012), pp. 2634–2638Google Scholar
  61. 61.
    G. Zhao, M. Pietikainen, Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 915–928 (2007)CrossRefGoogle Scholar
  62. 62.
    A. Asthana, M. de la Hunty, A. Dhall, R. Goecke, Facial performance transfer via deformable models and parametric correspondence. IEEE Trans. Vis. Comput. Graph. 18, 1511–1519 (2012)Google Scholar
  63. 63.
    A. Dhall, A. Asthana, R. Goecke, Facial expression based automatic album creation, in Proceedings of the Neural Information Processing. Models and Applications (ICONIP), (Springer, 2010), pp. 485–492Google Scholar
  64. 64.
    M. Pantic, L. Rothkrantz, Expert system for automatic analysis of facial expression. Image Vision Comput. J. 18, 881–905 (2000)CrossRefGoogle Scholar
  65. 65.
    L. Zhang, D. Tjondronegoro, V. Chandran, Evaluation of texture and geometry for dimensional facial expression recognition, in Proceedings of the International Conference on Digital Image Computing Techniques and Applications (DICTA) (IEEE, 2011), pp. 620–626Google Scholar
  66. 66.
    A. Dhall, A. Asthana, R. Goecke, T. Gedeon, Emotion recognition using PHOG and LPQ features, in Proceedings of the IEEE Conference Automatic Faces & Gesture Recognition workshop FERA (IEEE, 2011), pp. 878–883Google Scholar
  67. 67.
    K. Sikka, T. Wu, J. Susskind, M. Bartlett, Exploring bag of words architectures in the facial expression domain, in Proceedings of the European Conference on Computer Vision and Workshops (ECCVW) (Springer, 2012), pp. 250–259Google Scholar
  68. 68.
    F. Lavagetto, R. Pockaj, The facial animation engine: towards a high-level interface for the design of MPEG-4 compliant animated faces. IEEE Trans. Circuits Syst. Video Technol. 9, 277–289 (1999)CrossRefGoogle Scholar
  69. 69.
    N. Sebe, I. Cohen, T. Gevers, T.S. Huang, Emotion recognition based on joint visual and audio cues, in Proceedings of the International Conference on Pattern Recognition (ICPR), 2006Google Scholar
  70. 70.
    A. Asthana, J. Saragih, M. Wagner, R. Goecke, Evaluating AAM fitting methods for facial expression recognition, in Proceedings of the IEEE International Conference on Affective Computing and Intelligent Interaction (ACII) (Springer, 2009), pp. 598–605Google Scholar
  71. 71.
    Z. Wang, A.C. Bovik, H.R. Sheikh, S. Member, E.P. Simoncelli, S. Member, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004)CrossRefGoogle Scholar
  72. 72.
    M. Valstar, M. Pantic, Fully automatic facial action unit detection and temporal analysis, in Proceedings of the International Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), (IEEE, 2006), pp. 149–149Google Scholar
  73. 73.
    T. Ojala, M. Pietikinen, T. Menp, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24, 971–987 (2002)CrossRefGoogle Scholar
  74. 74.
    V. Ojansivu, J. Heikkilä, Blur insensitive texture classification using local phase quantization, in Proceedings of the Image and Signal Processing (ICISP) (Springer, 2008), pp. 236–243Google Scholar
  75. 75.
    J. Joshi, R. Goecke, M. Breakspear, G. Parker, Can body expressions contribute to automatic depression analysis? in Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (IEEE, 2013), pp. 1–7Google Scholar
  76. 76.
    Z. Li, J.-I. Imai, M. Kaneko, Facial-component-based bag of words and PHOG descriptor for facial expression recognition, in Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC) (IEEE, 2009), pp. 1353–1358Google Scholar
  77. 77.
    V. Bettadapura, G. Schindler, T. Plötz, I. Essa, Augmenting bag-of-words: data-driven discovery of temporal and structural information for activity recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2013), pp. 2619–2626Google Scholar
  78. 78.
    I. Laptev, T. Lindeberg, Space-time interest points, in International Conference on Computer Vision (ICCV) (IEEE, 2003), pp. 432–439Google Scholar
  79. 79.
    F. Wallhoff, Facial Expressions and Emotion Database, Technical Report (2006)Google Scholar
  80. 80.
    C. Vondrick, A. Khosla, T. Malisiewicz, A. Torralba, HOGgles: Visualizing object detection features, in Proceedings of the IEEE International Conference on Computer Vision (ICCV) (IEEE, 2013), pp. 1–8Google Scholar
  81. 81.
    J.R. Williamson, T.F. Quatieri, B.S. Helfer, G. Ciccarelli, D.D. Mehta, Vocal and facial biomarkers of depression based on motor incoordination and timing, in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge (AVEC) (ACM, 2014), pp. 65–72Google Scholar
  82. 82.
    R. Gupta, N. Malandrakis, B. Xiao, et al. Multimodal prediction of affective dimensions and depression in human-computer interactions, in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge (AVEC) (ACM, 2014), pp. 33–40Google Scholar
  83. 83.
    H. Pérez-Espinosa, H.J. Escalante, L. Villaseñor-Pineda, M. Montes-y-Gómez, D. Pinto Avedaño, V. Reyez-Meza, Fusing affective dimensions and audio-visual features from segmented video for depression recognition, in Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge (ACM, 2014), pp. 49–55Google Scholar
  84. 84.
    J. Joshi, R. Goecke, S. Alghowinem et al., Multimodal assistive technologies for depression diagnosis and monitoring. J. Multimodel User Interfaces 7, 217–228 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.School of PsychiatryUniversity of New South WalesSydneyAustralia
  2. 2.Black Dog InstitutePrince of Wales HospitalRandwickAustralia
  3. 3.Human-Centred Technology Research CentreUniversity of CanberraCanberraAustralia
  4. 4.Research School of Computer ScienceAustralian National UniversityActonAustralia

Personalised recommendations