Coupled HMM-based multimodal fusion for mood disorder detection through elicited audio–visual signals

  • Tsung-Hsien Yang
  • Chung-Hsien Wu
  • Kun-Yi Huang
  • Ming-Hsiang Su
Original Research


Mood disorders encompass a wide array of mood issues, including unipolar depression (UD) and bipolar disorder (BD). In diagnostic evaluation on the outpatients with mood disorder, a high percentage of BD patients are initially misdiagnosed as having UD. It is crucial to establish an accurate distinction between BD and UD to make a correct and early diagnosis, leading to improvements in treatment and course of illness. In this study, eliciting emotional videos are firstly used to elicit the patients’ emotions. After watching each video clips, their facial expressions and speech responses are collected when they are interviewing with a clinician. In mood disorder detection, the facial action unit (AU) profiles and speech emotion profiles (EPs) are obtained, respectively, by using the support vector machines (SVMs) which are built via facial features and speech features adapted from two selected databases using a denoising autoencoder-based method. Finally, a Coupled Hidden Markov Model (CHMM)-based fusion method is proposed to characterize the temporal information. The CHMM is modified to fuse the AUs and the EPs with respect to six emotional videos. Experimental results show the promising advantage and efficacy of the CHMM-based fusion approach for mood disorder detection.


Mood disorder detection Coupled Hidden Markov Model Multimodal fusion Autoencoder adaptation 


  1. American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders (DSM-5R): American Psychiatric PubGoogle Scholar
  2. Barnes TR (1989) A rating scale for drug-induced akathisia. Br J Psychiatry 154:672–676CrossRefGoogle Scholar
  3. Bersani G, Polli E, Valeriani G, Zullo D, Melcore C, Capra E, Quartini A, Marino P, Minichino A, Bernabei L (2013) Facial expression in patients with bipolar disorder and schizophrenia in response to emotional stimuli: a partially shared cognitive and social deficit of the two disorders. J Neuropsychiatr Dis Treat 9:1137CrossRefGoogle Scholar
  4. Bou-Ghazale SE, Assaleh K (2002) A robust endpoint detection of speech for noisy environments with application to automatic speech recognition. IEEE international conference on acoustics, speech, and signal processing (ICASSP), pp IV-3808–IV-3811Google Scholar
  5. Bozikas VP, Kosmidis MH, Tonia T, Andreou C, Focas K, Karavatos A (2007) Impaired perception of affective prosody in remitted patients with bipolar disorder. J Neuropsychiatry Clin Neurosci 19:436–440CrossRefGoogle Scholar
  6. Brand M, Oliver N, Pentland A (1997) Coupled hidden Markov models for complex action recognition. In: Proc. Computer Vision Pattern Recognition, pp. 201–206Google Scholar
  7. Chen CH, Lennox B, Jacob R, Calder A, Lupson V, Bisbrown-Chippendale R, Suckling J, Bullmore E (2006) Explicit and implicit facial affect recognition in manic and depressed states of bipolar disorder: a functional magnetic resonance imaging study. Biol Psychiatry 59:31–39CrossRefGoogle Scholar
  8. Cheng C, Chen H, Chan Y, Su Y, Tseng C (2013) Taiwan corpora of Chinese emotions and relevant psychophysiological data—normative data for Chinese jokes. Chin J Psychol 55:555–569Google Scholar
  9. Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, La Torre FD (2009) Detecting depression from facial actions and vocal prosody. In: Proc. IEEE international conference on affective computing and intelligent interaction and workshops, pp 1–7Google Scholar
  10. David DP, Soeiro-de-Souza MG, Moreno RA, Bio DS (2014) Facial emotion recognition and its correlation with executive functions in bipolar I patients and healthy controls. J Affect Disord 152:288–294CrossRefGoogle Scholar
  11. Ekman P (1999) Basic emotions. Handbook of cognition and emotion. Wiley, New York, pp 45–60Google Scholar
  12. Erguzel TT, Sayar GH, Tarhan N (2016) Artificial intelligence approach to classify unipolar and bipolar depressive disorders. Neural Comput Appl 27:1607–1616CrossRefGoogle Scholar
  13. Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the international conference on multimedia, 2010, pp. 1459–1462Google Scholar
  14. Furui S (1989) Unsupervised speaker adaptation based on hierarchical spectral clustering. IEEE Trans Acoust Speech Signal Process 37(12):1923–1930CrossRefGoogle Scholar
  15. Giannakopoulos T (2009) A method for silence removal and segmentation of speech signals, implemented in Matlab, Department of Informatics and Telecommunications, University of Athens, Greece, Computational Intelligence Laboratory (CIL), Insititute of Informatics and Telecommunications (IIT), NCSR DEMOKRITOS, GreeceGoogle Scholar
  16. Glorot X, Bordes A, Bengio Y (2011) Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Proceedings of the 28th international conference on machine learning (ICML-11) pp. 513–520Google Scholar
  17. Goghari VM, Sponheim SR (2013) More pronounced deficits in facial emotion recognition for schizophrenia than bipolar disorder. Compr Psychiatry 54:388–397CrossRefGoogle Scholar
  18. Greco A, Valenza G, Lanata A, Rota G, Scilingo EP (2014) Electrodermal activity in bipolar patients during affective elicitation. IEEE J Biomed Health Inform 18(6):1865–1873CrossRefGoogle Scholar
  19. Gross JJ, Levenson RW (1995) Emotion elicitation using films. Cogn Emot 9:87–108CrossRefGoogle Scholar
  20. Gunes H, Pantic M (2010) Automatic, dimensional and continuous emotion recognition. Int J Synth Emot (IJSE) 1(1):68–99CrossRefGoogle Scholar
  21. Guy W (1976) Clinical global impression scale, The ECDEU assessment manual for psychopharmacology-revised. Volume DHEW Publ No ADM 76, vol 338, pp. 218–222Google Scholar
  22. Hamilton M (1960) A rating scale for depression. J Neurol Neurosurg Psychiatry 23:56CrossRefGoogle Scholar
  23. Hermansky H, Morgan N (1994) RASTA processing of speech. IEEE Trans Speech Audio Process 2:578–589CrossRefGoogle Scholar
  24. Hirschfeld RM, Williams JB, Spitzer RL, Calabrese JR, Flynn L, Keck PE Jr, Lewis L, McElroy SL, Post RM, Rapport DJ (2000) Development and validation of a screening instrument for bipolar spectrum disorder: the mood disorder questionnaire. Am J Psychiatry 157:1873–1875CrossRefGoogle Scholar
  25. Howard N (2013) Approach towards a natural language analysis for diagnosing mood disorders and comorbid conditions. In: Proc. mexican international conference on artificial intelligence (MICAI), pp. 234–243Google Scholar
  26. Lanata A, Greco A, Valenza G, Scilingo EP (2014) A pattern recognition approach based on electrodermal response for pathological mood identification in bipolar disorders. In: Proc. IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 3601–3605Google Scholar
  27. Langenecker SA, Bieliauskas LA, Rapport LJ, Zubieta JK, Wilde EA, Berent S (2005) Face emotion perception and executive functioning deficits in depression. J Clin Exp Neuropsychol 27:320–333CrossRefGoogle Scholar
  28. Leucht S, Pitschel-Walz G, Abraham D, Kissling W (1999) Efficacy and extrapyramidal side-effects of the new antipsychotics olanzapine, quetiapine, risperidone, and sertindole compared to conventional antipsychotics and placebo. A meta-analysis of randomized controlled trials. Schizophr Res 35:51–68CrossRefGoogle Scholar
  29. Low LSA, Maddage NC, Lech M, Sheeber L, Allen NB (2010) Influence of acoustic low-level descriptors in the detection of clinical depression in adolescents. In: Proc. IEEE international conference on acoustics speech and signal processing (ICASSP), pp. 5154–5157Google Scholar
  30. Low LSA, Maddage NC, Lech M, Sheeber L, Allen NB (2011) Detection of clinical depression in adolescents’ speech during family interactions. IEEE Trans Biomed Eng 58(3):574–586CrossRefGoogle Scholar
  31. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), pp. 94–101Google Scholar
  32. Martin O, Kotsia I, Macq B, Pitas I (2006) The eNTERFACE’05 audio-visual emotion database. In: Proceedings of the 22nd international conference on data engineering workshops (ICDEW’06), Atlanta, USA, 3–7 April 2006Google Scholar
  33. Ooi KEB, Lech M, Allen NB (2013) Multichannel weighted speech classification system for prediction of major depression in adolescents. IEEE Trans Biomed Eng 60(2):497–506CrossRefGoogle Scholar
  34. Perlis RH (2005) Misdiagnosis of bipolar disorder. Am J Managed Care 11:S271–S274Google Scholar
  35. Rattani A, Kisku DR, Bicego M, Tistarelli M (2007) Feature level fusion of face and fingerprint biometrics. In: Proceedings of the first IEEE international conference on biometrics: theory, applications, and systems, 2007, BTAS 2007, IEEE, pp 1–6Google Scholar
  36. Rezek L, Sykacek P, Roberts SJ (2000) Learning interaction dynamics with coupled hidden Markov models. IEE proceedings—science, measurement and technology, vol 147. no. 6Google Scholar
  37. Saha G, Chakroborty S, Senapati S (2005) A new silence removal and endpoint detection algorithm for speech and speaker recognition applications. In: Proceedings of the 11th national conference on communications (NCC), pp. 291–295Google Scholar
  38. Sanchez MH, Vergyri D, Ferrer L, Richey C, Garcia P, Knoth B, Jarrold W (2011) Using prosodic and spectral features in detecting depression in elderly males. In: Proc. INTERSPEECH, pp. 3001–3004Google Scholar
  39. Schuller B, Valstar M, Eyben F, McKeown G, Cowie R, Pantic M (2011) AVEC 2011 the first international audio/visual emotion challenge. In: Proc. First int. audio/visual emotion challenge and workshop (ACII), pp. 415–424Google Scholar
  40. Schuller B, Valstar M, Eyben F, Cowie R, Pantic M (2012) AVEC 2012—the continuous audio/visual emotion challenge. In: Proc. of int. audio/visual emotion challenge and workshop (AVEC), ACM ICMIGoogle Scholar
  41. Summers M, Papadopoulou K, Bruno S, Cipolotti L, Ron MA (2006) Bipolar I and bipolar II disorder: cognition and emotion processing. Psychol Med 36:1799–1809CrossRefGoogle Scholar
  42. Surguladze SA, Young AW, Senior C, Brébion G, Travis MJ, Phillips ML (2004) Recognition accuracy and response bias to happy and sad facial expressions in patients with major depression. Neuropsychology 18:212CrossRefGoogle Scholar
  43. Tzimiropoulos G, Pantic M (2014) Gauss-newton deformable part models for face alignment in-the-wild. In: Computer vision and pattern recognition (CVPR), 2014 IEEE conference, pp. 1851–1858Google Scholar
  44. Valstar M, Schuller B, Smith K, Eyben F, Jiang B, Bilakhia S et al (2013) AVEC 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on audio/visual emotion challenge, ACM, pp 3–10Google Scholar
  45. Vederman AC, Weisenbach SL, Rapport LJ, Leon HM, Haase BD, Franti LM, Schallmo MP, Saunders EF, Kamali MM, Zubieta JK (2012) Modality-specific alterations in the perception of emotional stimuli in bipolar disorder compared to healthy controls and major depressive disorder. Cortex 48:1027–1034CrossRefGoogle Scholar
  46. Wu CH, Lin JC, Wei WL (2014) A survey on audiovisual emotion recognition: databases, features, and data fusion strategies. APSIPA transactions on signal and information processing, vol. 3, e12Google Scholar
  47. Young R, Biggs J, Ziegler V, Meyer D (1978) A rating scale for mania: reliability, validity and sensitivity. Br J Psychiatry 133:429–435CrossRefGoogle Scholar
  48. Zeng Z, Pantic M, Roisman G, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence, 31.1 pp. 39–58Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Computer Science and Information EngineeringNational Cheng-Kung UniversityTainanTaiwan

Personalised recommendations