Skip to main content

Artificial intelligence for suicide assessment using Audiovisual Cues: a review

Artificial Intelligence Review (2022)Cite this article

Abstract

Death by suicide is the seventh leading death cause worldwide. The recent advancement in Artificial Intelligence (AI), specifically AI applications in image and voice processing, has created a promising opportunity to revolutionize suicide risk assessment. Subsequently, we have witnessed fast-growing literature of research that applies AI to extract audiovisual non-verbal cues for mental illness assessment. However, the majority of the recent works focus on depression, despite the evident difference between depression symptoms and suicidal behavior non-verbal cues. In this paper, we review the recent works that study suicide ideation and suicide behavior detection through audiovisual feature analysis, mainly suicidal voice/speech acoustic features analysis and suicidal visual cues. Automatic suicide assessment is a promising research direction that is still in the early stages. Accordingly, there is a lack of large datasets that can be used to train machine leaning and deep learning models proven to be effective in other, similar tasks.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2

References

  • Airas M (2008) TKK Aparat: An environment for voice inverse filtering and parameterization. Logop Phoniatr Vocology 33:49–64

    Article  Google Scholar 

  • Akkaralaertsest T, Yingthawornsuk T (2015) Comparative analysis of vocal characteristics in speakers with depression and high-risk suicide. Int J Comput Theory Eng 7:448

    Article  Google Scholar 

  • Alghowinem S, Goecke R, Wagner M et al (2016) Multimodal depression detection: fusion analysis of paralinguistic, head pose and eye gaze behaviors. IEEE Trans Affect Comput 9:478–490

    Article  Google Scholar 

  • Anunvrapong P, Yingthawornsuk T (2014) Characterization of ∆MFCC in depressed speech sample as assessment of suicidal risk. In: International Conference on Advanced Computational Technologies \& Creative Media (ICACTCM ‘2014). pp 119–123

  • Barsties B, De Bodt M (2015) Assessment of voice quality: current state-of-the-art. Auris Nasus Larynx 42:183–188

    Article  Google Scholar 

  • Belouali A, Gupta S, Sourirajan V et al (2021) Acoustic and language analysis of speech for suicidal ideation among US veterans. BioData Min 14:1–17

    Article  Google Scholar 

  • Benton A, Coppersmith G, Dredze M (2017) Ethical research protocols for social media health research. In: Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. pp 94–102

  • Blanchard M, Farber BA (2020) “It is never okay to talk about suicide”: patients’ reasons for concealing suicidal ideation in psychotherapy. Psychother Res 30:124–136

    Article  Google Scholar 

  • Castillo-Sánchez G, Marques G, Dorronzoro E et al (2020) Suicide Risk Assessment Using Machine Learning and Social Networks: a Scoping Review. J Med Syst 44:205. https://doi.org/10.1007/s10916-020-01669-5

    Article  Google Scholar 

  • Chakravarthula SN, Nasir M, Tseng S-Y et al (2020) Automatic prediction of suicidal risk in military couples using multimodal interaction cues from couples conversations. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp 6539–6543

  • Cummins N, Scherer S, Krajewski J et al (2015) A review of depression and suicide risk assessment using speech analysis. Speech Commun 71:10–49

    Article  Google Scholar 

  • Degottex G, Kane J, Drugman T et al (2014) COVAREP—A collaborative voice analysis repository for speech technologies. In: 2014 ieee international conference on acoustics, speech and signal processing (icassp). pp 960–964

  • Dhelim S, Chen L, Aung N et al (2022) A hybrid personality-aware recommendation system based on personality traits and types models. J Ambient Intell Humaniz Comput 1–14. https://doi.org/10.1007/s12652-022-04200-5

  • Dhelim S, Ning H, Aung N (2020) ComPath: User Interest Mining in Heterogeneous Signed Social Networks for Internet of People. IEEE Internet Things J 1–1. https://doi.org/10.1109/JIOT.2020.3037109

  • Doval B, d’Alessandro C, Henrich N (2006) The spectrum of glottal flow models. Acta Acust united with Acust 92:1026–1046

    Google Scholar 

  • Drugman T, Bozkurt B, Dutoit T (2012) A comparative study of glottal source estimation techniques. Comput Speech \& Lang 26:20–34

    Article  Google Scholar 

  • Eigbe N, Baltrusaitis T, Morency L-P, Pestian J (2018) Toward Visual Behavior Markers of Suicidal Ideation. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, pp 530–534

  • Eyben F, Scherer KR, Schuller BW et al (2015) The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Trans Affect Comput 7:190–202

    Article  Google Scholar 

  • France DJ, Shiavi RG, Silverman S et al (2000) Acoustical properties of speech as indicators of depression and suicidal risk. IEEE Trans Biomed Eng 47:829–837

    Article  Google Scholar 

  • Galatzer-Levy I, Abbas A, Ries A et al (2021) Validation of Visual and Auditory Digital Markers of Suicidality in Acutely Suicidal Psychiatric Inpatients: Proof-of-Concept Study. J Med Internet Res 23:e25199

    Article  Google Scholar 

  • Gideon J, Schatten HT, McInnis MG, Provost EM (2019) Emotion recognition from natural phone conversations in individuals with and without recent suicidal ideation. In: Interspeech

  • Gratch I, Choo T-H, Galfalvy H et al (2021) Detecting suicidal thoughts: The power of ecological momentary assessment. Depress Anxiety 38:8–16

    Article  Google Scholar 

  • Gratch J, Artstein R, Lucas G et al (2014) The distress analysis interview corpus of human and computer interviews. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). pp 3123–3128

  • Gupta R, Malandrakis N, Xiao B et al (2014) Multimodal prediction of affective dimensions and depression in human-computer interactions. In: Proceedings of the 4th international workshop on audio/visual emotion challenge. pp 33–40

  • He L, Cao C (2018) Automated depression analysis using convolutional neural networks from speech. J Biomed Inform 83:103–111

    Article  Google Scholar 

  • He L, Niu M, Tiwari P et al (2022) Deep learning for depression recognition with audiovisual cues: A review. Inf Fusion 80:56–86

    Article  Google Scholar 

  • Ji S, Pan S, Li X et al (2020) Suicidal ideation detection: A review of machine learning methods and applications. IEEE Trans Comput Soc Syst

  • Joshi J, Dhall A, Goecke R et al (2012) Neural-net classification for spatio-temporal descriptor based depression analysis. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012). pp 2634–2638

  • Joshi J, Goecke R, Parker G, Breakspear M (2013) Can body expressions contribute to automatic depression analysis? In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2013

  • Kane J, Aylett M, Yanushevskaya I, Gobl C (2014) Phonetic feature extraction for context-sensitive glottal source processing. Speech Commun 59:10–21

    Article  Google Scholar 

  • Kaymaz Keskinpala H, Yingthawornsuk T, Salomon RM et al (2007) Distinguishing High Risk Suicidal Subjects among Depressed Subjects Using Mel-Frequency Cepstrum Coefficients and Cross Validation Technique. Disting High Risk Suicidal Subj among Depress Subj Using Mel-Frequency Cepstrum Coefficients Cross Valid Tech1000–1004

  • Keskinpala HK, Yingthawornsuk T, Wilkes DM et al (2007) Screening for high risk suicidal states using mel-cepstral coefficients and energy in frequency bands. In: 2007 15th European Signal Processing Conference. pp 2229–2233

  • Kleiman S, Rule NO (2013) Detecting Suicidality From Facial Appearance. Soc Psychol Personal Sci 4:453–460. https://doi.org/10.1177/1948550612466115

    Article  Google Scholar 

  • Laksana E, Baltrusaitis T, Morency L-P, Pestian JP (2017) Investigating Facial Behavior Indicators of Suicidal Ideation. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). IEEE, pp 770–777

  • Liu RT, Bettis AH, Burke TA (2020) Characterizing the phenomenology of passive suicidal ideation: a systematic review and meta-analysis of its prevalence, psychiatric comorbidity, correlates, and comparisons with active suicidal ideation. Psychol Med 50:367–383

    Article  Google Scholar 

  • Moher D, Liberati A, Tetzlaff J et al (2010) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 8:336–341

    Article  Google Scholar 

  • Nasir M, Baucom BR, Bryan CJ et al (2017) Complexity in speech and its relation to emotional bond in therapist-patient interactions during suicide risk assessment interviews. In: Interspeech. pp 3296–3300

  • Nasir M, Jati A, Shivakumar PG et al (2016) Multimodal and multiresolution depression detection from speech and facial landmark features. In: Proceedings of the 6th international workshop on audio/visual emotion challenge. pp 43–50

  • NH NNW, Wilkes MD, Salomon RM (2015a) Investigating the Course of Recovery in High Risk Suicide using Power Spectral Density.Asian J Appl Sci3

  • NH NNW, Wilkes MD, Salomon RM (2015b) Timing Patterns of Speech as Potential Indicators of Near-Term Suicidal Risk.Int J Multidiscip Curr Res3

  • Ozdas A, Shiavi RG, Silverman SE et al (2004a) Investigation of Vocal Jitter and Glottal Flow Spectrum as Possible Cues for Depression and Near-Term Suicidal Risk. IEEE Trans Biomed Eng 51:1530–1540. https://doi.org/10.1109/TBME.2004.827544

    Article  Google Scholar 

  • Ozdas A, Shiavi RG, Wilkes DM et al (2004b) Analysis of vocal tract characteristics for near-term suicidal risk assessment. Methods Inf Med 43:36–38

    Article  Google Scholar 

  • Pampouchidou A, Simos PG, Marias K et al (2017) Automatic assessment of depression based on visual cues: A systematic review. IEEE Trans Affect Comput 10:445–470

    Article  Google Scholar 

  • Pestian J, Santel D, Sorter M et al (2018) A Machine Learning Approach to Identifying Future Suicide Risk. SSRN Electron J. https://doi.org/10.2139/ssrn.3279211

    Article  Google Scholar 

  • Quatieri TF, Malyska N (2012) Vocal-source biomarkers for depression: A link to psychomotor activity. In: Thirteenth Annual Conference of the International Speech Communication Association

  • Reddy PP, Suresh C, Rao VK et al (2021) Vocal Analysis to Predict Suicide Tendency. In: Proceedings of International Conference on Advances in Computer Engineering and Communication Systems. pp 481–488

  • Ringeval F, Schuller B, Valstar M et al (2019) AVEC 2019 workshop and challenge: state-of-mind, detecting depression with AI, and cross-cultural affect recognition. In: Proceedings of the 9th International on Audio/visual Emotion Challenge and Workshop. pp 3–12

  • Rohlfing ML, Buckley DP, Piraquive J et al (2021) Hey Siri: How Effective are Common Voice Recognition Systems at Recognizing Dysphonic Voices? Laryngoscope 131:1599–1607

    Article  Google Scholar 

  • Scherer S, Pestian J, Morency L-P (2013) Investigating the speech characteristics of suicidal adolescents. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, pp 709–713

  • Scherer S, Stratou G, Lucas G et al (2014) Automatic audiovisual behavior descriptors for psychological disorder analysis. Image Vis Comput 32:648–658

    Article  Google Scholar 

  • Sebald DJ, Bucklew JA (2000) Support vector machine techniques for nonlinear equalization. IEEE Trans signal Process 48:3217–3226

    Article  Google Scholar 

  • Shah AP, Vaibhav V, Sharma V et al (2019) Multimodal Behavioral Markers Exploring Suicidal Intent in Social Media Videos. In: 2019 International Conference on Multimodal Interaction. ACM, New York, NY, USA, pp 409–413

  • Silverman MM, Berman AL, Sanddal ND et al (2007) Rebuilding the tower of Babel: a revised nomenclature for the study of suicide and suicidal behaviors. Part 2: Suicide-related ideations, communications, and behaviors. Suicide Life-Threatening Behav 37:264–277

    Article  Google Scholar 

  • Silverman SE, others Silverman MK (2006) Methods and apparatus for evaluating near-term suicidal risk using vocal parameters

  • Silvia PJ, Allan WD, Beauchamp DL et al (2006) Biased recognition of happy facial expressions in social anxiety. J Soc Clin Psychol 25:585–602

    Article  Google Scholar 

  • Skerrett DM, Kõlves K, De Leo D (2015) Are LGBT populations at a higher risk for suicidal behaviors in Australia? Research findings and implications. J Homosex 62:883–901

    Article  Google Scholar 

  • Stasak B, Epps J, Schatten HT et al (2021) Read speech voice quality and disfluency in individuals with recent suicidal ideation or suicide attempt. Speech Commun 132:10–20

    Article  Google Scholar 

  • Sundberg J, Patel S, Bjorkner E, Scherer KR (2011) Interdependencies among voice source parameters in emotional speech. IEEE Trans Affect Comput 2:162–174

    Article  Google Scholar 

  • Valstar M, Schuller B, Smith K et al (2013) Avec 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge. pp 3–10

  • Venek V, Scherer S, Morency L-P et al (2017) Adolescent Suicidal Risk Assessment in Clinician-Patient Interaction. IEEE Trans Affect Comput 8:204–215. https://doi.org/10.1109/TAFFC.2016.2518665

    Article  Google Scholar 

  • Venek V, Scherer S, Morency L-P et al (2014) Adolescent suicidal risk assessment in clinician-patient interaction: A study of verbal and acoustic behaviors. In: 2014 IEEE Spoken Language Technology Workshop (SLT). IEEE, pp 277–282

  • Venn HR, Gray JM, Montagne B et al (2004) Perception of facial expressions of emotion in bipolar disorder. Bipolar Disord 6:286–293

    Article  Google Scholar 

  • Walker J, Murphy P (2007) A review of glottal waveform analysis. Prog nonlinear speech Process 1–21

  • Wang Q, Yang H, Yu Y (2018) Facial expression video analysis for depression detection in Chinese patients. J Vis Commun Image Represent 57:228–233

    Article  Google Scholar 

  • Wang W, Ning H, Shi F et al (2021a) A Survey of Hybrid Human-Artificial Intelligence for Social Computing. IEEE Trans Human-Machine Syst

  • Wang Y, Guobule N, Li M, Li J (2021b) The correlation of facial emotion recognition in patients with drug-na{\“\i}ve depression and suicide ideation. J Affect Disord 295:250–254

    Article  Google Scholar 

  • Warner CH, Appenzeller GN, Grieger T et al (2011) Importance of anonymity to encourage honest reporting in mental health screening after combat deployment. Arch Gen Psychiatry 68:1065–1071

    Article  Google Scholar 

  • Williamson JR, Godoy E, Cha M et al (2016) Detecting depression using vocal, facial and semantic communication cues. In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. pp 11–18

  • World Health Organization (2014) Preventing suicide: A global imperative. World Health Organization

  • World Health Organization (2019) Suicide key facts. https://www.who.int/news-room/fact-sheets/detail/suicide. Accessed 14 Apr 2021

  • Yingthawornsuk T, Keskinpala HK, France D et al (2006) Objective estimation of suicidal risk using vocal output characteristics. In: Ninth International Conference on Spoken Language Processing

  • Yingthawornsuk T, Keskinpala HK, Wilkes DM et al (2007) Direct acoustic feature using iterative EM algorithm and spectral energy for classifying suicidal speech. In: Eighth Annual Conference of the International Speech Communication Association

  • Yingthawornsuk T, Shiavi RG (2008) Distinguishing depression and suicidal risk in men using GMM based frequency contents of affective vocal tract response. In: 2008 International Conference on Control, Automation And Systems. pp 901–904

  • Zhang T, Schoene AM, Ji S, Ananiadou S (2022) Natural language processing applied to mental illness detection: a narrative review. NPJ Digit Med 5:1–13

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded by the National Natural Science Foundation of China (Grant Number: 61872038).

Author information

Authors and Affiliations

  1. School of Computer Science, Dublin College University, Dublin, Ireland

    Sahraoui Dhelim

  2. School of Computing, Ulster University, Ulster, UK

    Liming Chen & Chris Nugent

  3. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China

    Huansheng Ning

Authors
  1. Sahraoui Dhelim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liming Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huansheng Ning
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chris Nugent
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huansheng Ning.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dhelim, S., Chen, L., Ning, H. et al. Artificial intelligence for suicide assessment using Audiovisual Cues: a review. Artif Intell Rev (2022). https://doi.org/10.1007/s10462-022-10290-6

Download citation

  • Published:

  • DOI: https://doi.org/10.1007/s10462-022-10290-6

Keywords

  • Suicide detection
  • Machine learning
  • Speech analysis
  • Visual cues
  • Suicide ideation detection