Skip to main content

Advertisement

SpringerLink
Log in

Multimodal time-aware attention networks for depression detection

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Depression is a common mental disorder, which may lead to suicide when the condition is severe. With the advancement of technology, there are billions of people who share their thoughts and feelings on social media at any time and from any location. Social media data has therefore become a valuable resource to study and detect the depression of the user. In our work, we use Instagram as the platform to study depression detection. We use hashtags to find users and label them as depressive or non-depressive according to their self-statement. Text, image, and posting time are used jointly to detect depression. Furthermore, the time interval between posts is important information when studying medical-related data. In this paper, we use time-aware LSTM to handle the irregularity of time intervals in social media data and use an attention mechanism to pay more attention to the posts that are important for detecting depression. Experiment results show that our model outperforms previous work with an F1-score of 95.6%. In addition to the good performance on Instagram, our model also outperforms state-of-the-art methods in detecting depression on Twitter with an F1-score of 90.8%. This indicates the potential of our model to be a reference for psychiatrists to assess the patient; or for users to know more about their mental health condition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The datasets generated during the current study are available from the corresponding author on reasonable request.

Notes

  1. https://napoleoncat.com/stats/social-media-users-in-taiwan/2021/06

  2. https://github.com/ckiplab/ckip-transformers

  3. https://drive.google.com/file/d/11ye00sHFY5re2NOBRKreg-tVbDNrc7Xd/view

References

  • Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., & et al (2016). TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX conference on operating systems design and implementation (pp. 265–283). USENIX Association, DOI https://doi.org/10.5555/3026877.3026899, (to appear in print).

  • American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed).

  • An, M., Wang, J., Li, S., & Zhou, G. (2020). Multimodal topic-enriched auxiliary learning for depression detection. In Proceedings of the 28th international conference on computational linguistics (pp. 1078–1089). International Committee on Computational Linguistics, DOI https://doi.org/10.18653/v1/2020.coling-main.94, (to appear in print).

  • Bahdanau, D., Cho, K.H., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd international conference on learning representations. arXiv:1409.0473.

  • Baytas, I.M., Xiao, C., Zhang, X., Wang, F., Jain, A.K., & Zhou, J. (2017). Patient subtyping via time-aware LSTM networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 65–74). https://doi.org/10.1145/3097983.3097997: Association for Computing Machinery.

  • Beck, A.T., Steer, R.A., & Brown, G.K. (1996). Manual for the Beck Depression Inventory-II. Psychological Corporation.

  • Chiu, C.Y., Lane, H.Y., Koh, J.L., & Chen, A.L. (2021). Multimodal depression detection on instagram considering time interval of posts. Journal of intelligent information systems (Vol. 56, pp. 25–47). Netherlands: Springer.

    Google Scholar 

  • Coppersmith, G., Dredze, M., & Harman, C. (2014). Quantifying mental health signals in Twitter. In Proceedings of the workshop on computational linguistics and clinical psychology: From linguistic signal to clinical reality (pp. 51–60). Association for Computational Linguistics, DOI https://doi.org/10.3115/v1/W14-3207, (to appear in print).

  • De Choudhury, M., Gamon, M., Counts, S., & Horvitz, E. (2013). Predicting depression via social media. Proceedings of the International AAAI Conference on Web and Social Media, 7(1), 128–137. https://ojs.aaai.org/index.php/ICWSM/article/view/14432.

    Google Scholar 

  • Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Li, F.-F. (2009). Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 248–255), DOI https://doi.org/10.1109/CVPR.2009.5206848, (to appear in print).

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, (Vol. 1 pp. 4171–4186). Association for Computational Linguistics, DOI https://doi.org/10.18653/v1/n19-1423.

  • Gui, T., Zhu, L., Zhang, Q., Peng, M., Zhou, X., Ding, K., & Chen, Z. (2019). Cooperative multimodal approach to depression detection in twitter. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 110–117. https://doi.org/10.1609/aaai.v33i01.3301110.

    Article  Google Scholar 

  • Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.

    Article  Google Scholar 

  • Huang, Y., Chiang, C.-F., & Chen, A.L. (2019). Predicting Depression Tendency based on Image, Text and Behavior Data from Instagram. In Proceedings of the 8th international conference on data science technology and applications (pp. 32–40), DOI https://doi.org/10.5220/0007833600320040, (to appear in print).

  • James, S.L., Abate, D., Abate, K.H., Abay, S.M., Abbafati, C., Abbasi, N., & et al (2018). Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990–2017: a systematic analysis for the global burden of disease study 2017. Lancet, 392(10159), 1789–1858. https://doi.org/10.1016/S0140-6736(18)32279-7.

    Article  Google Scholar 

  • Kingma, D.P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the 3rd international conference on learning representations. arXiv:1412.6980.

  • Kroenke, K., Spitzer, R.L., & Williams, J.B. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16(9), 606–613. https://doi.org/10.1046/j.1525-1497.2001.016009606.x.

    Article  Google Scholar 

  • Mann, P., Paes, A., & Matsushima, E.H. (2020). See and read: Detecting depression symptoms in higher education students using multimodal social media data. Proceedings of the International AAAI Conference on Web and Social Media, 14(1), 440–451. https://ojs.aaai.org/index.php/ICWSM/article/view/7313.

    Google Scholar 

  • Novak, P.K., Smailović, J., Sluban, B., & Mozetič, I. (2015). Sentiment of emojis. PLoS ONE, 10(12), e0144296. https://doi.org/10.1371/journal.pone.0144296.

    Article  Google Scholar 

  • Park, M., Cha, C., & Cha, M. (2012). Depressive moods of users portrayed in twitter. In Proceedings of the 18th ACM international conference on knowledge discovery and data mining (pp. 1–8).

  • Radloff, L.S. (1977). The CES-d scale: a Self-Report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385–401. https://doi.org/10.1177/014662167700100306.

    Article  Google Scholar 

  • Reece, A.G., & Danforth, C.M. (2017). Instagram photos reveal predictive markers of depression. EPJ Data Science, 6(15).

  • Shen, G., Jia, J., Nie, L., Feng, F., Zhang, C., Hu, T., & et al. (2017). Depression detection via harvesting social media: a multimodal dictionary learning solution. In Proceedings of the twenty-sixth international joint conference on artificial intelligence (pp. 3838–3844), DOI https://doi.org/10.24963/ijcai.2017/536, (to appear in print).

  • Shen, T., Jia, J., Shen, G., Feng, F., He, X., Luan, H., & et al (2018). Cross-domain depression detection via harvesting social media. In Proceedings of the twenty-seventh international joint conference on artificial intelligence (pp. 1611–1617). International Joint Conferences on Artificial Intelligence Organization, DOI https://doi.org/10.24963/ijcai.2018/223, (to appear in print).

  • Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A.A. (2017). Inception-v4, Inception-ResNet and the impact of residual connections on learning. In Proceedings of the Thirty-First AAAI conference on artificial intelligence (pp. 4278–4284). AAAI Press, DOI https://doi.org/10.5555/3298023.3298188, (to appear in print).

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., & et al (2017). Attention is all you need. In Proceedings of the 31st international conference on neural information processing systems (pp. 6000–6010). Curran Associates Inc, DOI https://doi.org/10.5555/3295222.3295349, (to appear in print).

  • Wang, P.S., Aguilar-Gaxiola, S., Alonso, J., Angermeyer, M.C., Borges, G., Bromet, E.J., & et al (2007). Use of mental health services for anxiety, mood, and substance disorders in 17 countries in the WHO world mental health surveys. Lancet, 370(9590), 841–850. https://doi.org/10.1016/S0140-6736(07)61414-7.

    Article  Google Scholar 

  • Wu, M.Y., Shen, C.Y., Wang, E.T., & Chen, A.L. (2020). A deep architecture for depression detection using posting, behavior, and living environment data. Journal of Intelligent Information Systems, 54, 225–244. https://doi.org/10.1007/s10844-018-0533-4.

    Article  Google Scholar 

  • Yates, A., Cohan, A., & Goharian, N. (2017). Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 2968–2978). Association for Computational Linguistics, DOI https://doi.org/10.18653/v1/D17-1322, (to appear in print).

Download references

Acknowledgements

This work was partially supported by the Ministry of Science and Technology, ROC (Grant Number: 109-2221-E-468-014-MY3). We thank National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational and storage resources.

Author information

Authors and Affiliations

  1. Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu, Taiwan

    Ju Chun Cheng

  2. Department of Computer Science and Information Engineering, Asia University/Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan

    Arbee L. P. Chen

Authors
  1. Ju Chun Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arbee L. P. Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ju Chun Cheng.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, J.C., Chen, A.L.P. Multimodal time-aware attention networks for depression detection. J Intell Inf Syst 59, 319–339 (2022). https://doi.org/10.1007/s10844-022-00704-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-022-00704-w

Keywords

Search

Navigation