Skip to main content

Advertisement

Log in

Talking Mental Health: a Battle of Wits Between Humans and AI

  • Published:
Journal of Technology in Behavioral Science Aims and scope Submit manuscript

Abstract

This study aimed to investigate the differences between responses generated by ChatGPT and those produced by humans in terms of authenticity, professionalism, and practicality. It involved 140 participants of the age group ranging from 18 to 43 (101 females, 37 males, and 2 preferring not to disclose their gender). The participants were presented with the 10 solution statements against the 10 problem statements, generated by human participants and ChatGPT3.5 (gpt-3.5-turbo) and asked to rate the responses on a 5-point Likert scale, with higher scores indicating higher levels of authenticity, professionalism, and practicality. Paired sample t-test was conducted to compare the scores of the ChatGPT-generated responses and the human-generated responses. The results of the study indicated that there was significant difference between the two types of responses in the given dimensions. These findings suggest that ChatGPT-generated responses can be considered a reliable alternative to human-generated responses in certain applications. Additionally, the use of ChatGPT-generated responses can reduce the response time and workload of human responders, as well as the associated costs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  • Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020). Effectiveness and safety of using chatbots to improve mental health: Systematic review and meta-analysis. Journal of Medical Internet Research, 22(7), e16021. https://doi.org/10.2196/16021

    Article  PubMed  PubMed Central  Google Scholar 

  • Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. In IFIP international conference on artificial intelligence applications and innovations (pp. 373–383). Springer, Cham.

  • Agarwal, J., & Sharma, S. (2023). Artificial Intelligence Enabled Cognitive Computer-Centered Digital Analysis Model for Examination of the Children’s Mental Health.

  • American Psychological Association. (2020). Ethical Principles of Psychologists and Code of Conduct. Including 2010 and 2016 Amendments. Washington, DC: American Psychological Association 2020. https://www.apa.org/ethics/code

  • Aminah, S., Hidayah, N., & Ramli, M. (2023). Considering ChatGPT to be the first aid for young adults on mental health issues. Journal of Public Health. https://doi.org/10.1093/pubmed/fdad065

  • Andersson, G., & Titov, N. (2014). Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry, 13(1), 4–11.

    Article  PubMed  PubMed Central  Google Scholar 

  • Asada, M. (2015). Development of artificial empathy. Neuroscience Research, 90, 41–50.

    Article  PubMed  Google Scholar 

  • Balaji, D., He, L., Giani, S., Bosse, T., Wiers, R., & de Bruijn, G. J. (2022). Effectiveness and acceptability of conversational agents for sexual health promotion: A systematic review and meta-analysis. Sexual Health, 19(5), 391–405. https://doi.org/10.1071/sh22016

  • Baumeister, H., Reichler, L., Munzinger, M., & Lin, J. (2019). The impact of guidance on internet-based mental health interventions—A systematic review. Internet Interventions, 14, 1–10.

    Google Scholar 

  • Bilquise, G., Ibrahim, S., & Shaalan, K. (2022). September 26). Emotionally intelligent chatbots: A systematic literature review. Human Behavior and Emerging Technologies, 2022, 1–23. https://doi.org/10.1155/2022/9601630

    Article  Google Scholar 

  • Bird, T., Mansell, W., Wright, J., Gaffney, H., & Tai, S. (2018). Manage your life online: A web-based randomized controlled trial evaluating the effectiveness of a problem-solving intervention in a student sample. Behavioural and Cognitive Psychotherapy, 46(5), 570–582.

    Article  PubMed  Google Scholar 

  • Bohr, A., & Memarzadeh, K. (2020). The rise of artificial intelligence in healthcare applications. Artificial Intelligence in Healthcare, 25–60. https://doi.org/10.1016/b978-0-12-818438-7.00002-2

  • Bond, R. R., O'Neill, S., Ennis, E., & Mulvenna, M. (2019). Chatbots for mental health and suicide prevention: Is this even ethical?. In European Conference on Mental Health.

  • Bradley, & Alhajjar. (n.d.). AI ethics: Assessing and correcting conversational bias in machine learning based chatbots. icwsm.org. https://workshop-proceedings.icwsm.org/pdf/2022_67.pdf

  • Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.

  • Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arxiv.org/abs/2303.12712

  • Bucci, S., Schwannauer, M., & Berry, N. (2019). The digital revolution and its impact on mental health care. Psychology and Psychotherapy: Theory, Research and Practice, 92(2), 277–297.

    Article  Google Scholar 

  • Cameron, G., Cameron, D., Megaw, G., Bond, R., Mulvenna, M., O’Neill, S., & McTear, M. (2017). Towards a chatbot for digital counselling. In Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017). BCS Learning & Development.

  • Carlbring, P., Hadjistavropoulos, H., Kleiboer, A., & Andersson, G. (2023). A new era in internet interventions: The advent of Chat-GPT and AI-assisted therapist guidance. Internet Interventions, 32, 100621. https://doi.org/10.1016/j.invent.2023.100621

    Article  PubMed  PubMed Central  Google Scholar 

  • Christensen, H., Griffiths, K. M., & Farrer, L. (2009). Adherence in internet interventions for anxiety and depression: Systematic review. Journal of Medical Internet Research, 11(2), e13. https://doi.org/10.2196/jmir.1194

    Article  PubMed  PubMed Central  Google Scholar 

  • Coghlan, S., Leins, K., Sheldrick, S., Cheong, M., Gooding, P., & D’Alfonso, S. (2023). To chat or bot to chat: Ethical issues with using chatbots in mental health. DIGITAL HEALTH, 9, 20552076231183544.

    Article  PubMed  PubMed Central  Google Scholar 

  • D’Alfonso, S. (2020). AI in mental health. Current Opinion in Psychology, 36, 112–117.

    Article  PubMed  Google Scholar 

  • Davenport, T., Guha, A., Grewal, D., & Bressgott, T. (2020). How artificial intelligence will change the future of marketing. Journal of the Academy of Marketing Science, 48, 24–42.

    Article  Google Scholar 

  • Denecke, K., Abd-Alrazaq, A., & Househ, M. (2021). Artificial intelligence for chatbots in mental health: Opportunities and challenges. Multiple perspectives on artificial intelligence in healthcare: Opportunities and challenges, 115–128.

  • Diederich, S., Brendel, A. B., Morana, S., & Kolbe, L. (2022). On the design of and interaction with conversational agents: An organizing and assessing review of human-computer interaction research. Journal of the Association for Information Systems, 23(1), 96–138. https://doi.org/10.17705/1jais.00724

  • Ee, C., Lake, J., Firth, J., Hargraves, F., de Manincor, M., Meade, T., Marx, W., & Sarris, J. (2020). An integrative collaborative care model for people with mental illness and physical comorbidities. International Journal of Mental Health Systems, 14(1), 83. https://doi.org/10.1186/s13033-020-00410-6

    Article  PubMed  PubMed Central  Google Scholar 

  • Erbe, D., Eichert, H. C., Riper, H., & Ebert, D. D. (2017). Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: Systematic review. Journal of Medical Internet Research, 19(9), e306.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ernsting, C., Dombrowski, S. U., Oedekoven, M., Sullivan, J. L., Kanzler, M., & Kuhlmey, A. (2020). Using smartphones and health apps to change and manage health behaviors: A population-based survey. Journal of Medical Internet Research, 22(2), e16956.

    Google Scholar 

  • Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017). Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): A randomized controlled trial. JMIR Mental Health, 4(2), e19. https://doi.org/10.2196/mental.7785

    Article  PubMed  PubMed Central  Google Scholar 

  • Freeman, D., Haselton, P., Freeman, J., Spanlang, B., Kishore, S., Albery, E., & Nickless, A. (2018). Automated psychological therapy using immersive virtual reality for treatment of fear of heights: A single-blind, parallel-group, randomised controlled trial. The Lancet Psychiatry, 5(8), 625–632.

  • Fulmer, R., Joerin, A., Gentile, B., Lakerink, L., & Rauws, M. (2018). Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: Randomized controlled trial. JMIR Mental Health, 5(4), e9782.

    Article  Google Scholar 

  • Gaffney, H., Mansell, W., Edwards, R., & Wright, J. (2013). Manage your life online (MYLO): A pilot trial of a conversational computer-based intervention for problem solving in a student sample. Behavioral and Cognitive Psychotherapy, 42(6), 731–746. https://doi.org/10.1017/s135246581300060x

  • Gaffney, H., Mansell, W., & Tai, S. (2019). Conversational agents in the treatment of mental health problems: Mixed-method systematic review. JMIR Mental Health, 6(10), e14166. https://doi.org/10.2196/14166

    Article  PubMed  PubMed Central  Google Scholar 

  • García-Peñalvo, F. J., Llorens Largo, F., & Vidal, J. (2023). The new reality of education in the face of advances in generative artificial intelligence.

  • Ghandeharioun, A., McDuff, D., Czerwinski, M., & Rowan, K. (2019). Towards understanding emotional intelligence for behavior change chatbots. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 8–14). IEEE.

  • Giger, J. C., Piçarra, N., Alves-Oliveira, P., Oliveira, R., & Arriaga, P. (2019). Humanization of robots: Is it really such a good idea? Human Behavior and Emerging Technologies, 1(2), 111–123.

    Article  Google Scholar 

  • Harrington, L., et al. (2021). AI-powered chatbots in mental health: Benefits, limitations, and ethical considerations. Ethics in Health Informatics and Health Information Management, 6(2), 67–72.

    Google Scholar 

  • He, Y., Yang, L., Zhu, X., Wu, B., Zhang, S., Qian, C., & Tian, T. (2022). Mental health chatbot for young adults with depressive symptoms during the COVID-19 pandemic: Single-blind, three-arm randomized controlled trial. Journal of Medical Internet Research, 24(11), e40719. https://doi.org/10.2196/40719

  • Hedman-Lagerlöf, E., Carlbring, P., Svärdman, F., Riper, H., Cuijpers, P., & Andersson, G. (2023). Therapist-supported Internet-based cognitive behaviour therapy yields similar effects as face-to-face therapy for psychiatric and somatic disorders: An updated systematic review and meta-analysis. World Psychiatry : Official Journal of the World Psychiatric Association (WPA), 22(2), 305–314. https://doi.org/10.1002/wps.21088

    Article  PubMed  Google Scholar 

  • Hsu, J. (2023) Mental health service criticised for experiment with AI chatbot. New Scientist. https://www.newscientist.com/article/2354077-mental-health-service-criticised-for-experiment-with-ai-chatbot/

  • Huang, M. H., & Rust, R. T. (2018). Artificial intelligence in service. Journal of Service Research, 21(2), 155–172.

    Article  Google Scholar 

  • Ingram, D. (2023). A mental health tech company ran an AI experiment on real users. Nothing’s stopping apps from conducting more. https://www.nbcnews.com/tech/internet/chatgpt-ai-experiment-mental-health-tech-app-koko-rcna65110

  • Inkster, B., Sarda, S., & Subramanian, V. (2018). An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: Real-world data evaluation mixed-methods study. JMIR mHealth and uHealth, 6(11), e12106.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kaddari, Z., Mellah, Y., Berrich, J., Belkasmi, M. G., & Bouchentouf, T. (2020, March). Natural language processing: Challenges and future directions. In International Conference on Artificial Intelligence & Industrial Applications, 236–246. Cham: Springer International Publishing.

  • Kavakli, M., Li, M., & Rudra, T. (2012). Towards the development of a virtual counselor to tackle students’ exam stress. Journal of Integrated Design and Process Science, 16(1), 5–26.

    Article  Google Scholar 

  • Kelly, S., Kaye, S. A., & Oviedo-Trespalacios, O. (2023). What factors contribute to the acceptance of artificial intelligence? A systematic review. Telematics and Informatics, 77, 101925. https://doi.org/10.1016/j.tele.2022.101925

  • Koelen, J. A., Vonk, A., Klein, A., de Koning, L., Vonk, P., de Vet, S., & Wiers, R. (2022). Man vs. Machine: A meta-analysis on the added value of human support in text-based internet treatments (“e-therapy”) for mental disorders. Clinical Psychology Review, 96, 102179.

  • Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., & Bashir, R. (2018). Conversational agents in healthcare: A systematic review. Journal of the American Medical Informatics Association, 25(9), 1248–1258.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lindqvist, K., Mechler, J., Midgley, N., Carlbring, P., Carstorp, K., Neikter, H. K., & Philips, B. (2022). “I didn’t have to look her in the eyes”—Participants’ experiences of the therapeutic relationship in internet-based psychodynamic therapy for adolescent depression. Psychotherapy Research, 1–15.

  • Luxton, D. D., McCann, R. A., Bush, N. E., Mishkind, M. C., & Reger, G. M. (2011). mHealth for mental health: Integrating smartphone technology in behavioral healthcare. Professional Psychology: Research and Practice, 42(6), 505–512. https://doi.org/10.1037/a0024485

    Article  Google Scholar 

  • Ly, K. H., Ly, A. M., Andersson, G., & Cuijpers, P. (2020). A fully automated conversational agent for promoting mental well-being: A pilot RCT using mixed methods. Internet Interventions, 20, 100321. https://doi.org/10.1016/j.invent.2020.100321

    Article  Google Scholar 

  • Maciejewski, J., & Smoktunowicz, E. (2023). Low-effort internet intervention to reduce students’ stress delivered with Meta’s Messenger chatbot (Stressbot): A randomized controlled trial. Internet Interventions, 33, 100653. https://doi.org/10.1016/j.invent.2023.100653

    Article  PubMed  PubMed Central  Google Scholar 

  • Miloff, A., Carlbring, P., Hamilton, W., Andersson, G., Reuterskiöld, L., & Lindner, P. (2020). Measuring alliance toward embodied virtual therapists in the era of automated treatments with the virtual therapist alliance scale (VTAS): Development and psychometric evaluation. Journal of Medical Internet Research, 22(3), e16660.

    Article  PubMed  PubMed Central  Google Scholar 

  • Minerva, F., & Giubilini, A. (2023). Is AI the future of mental healthcare?. Topoi, 42(3), 809–817. https://doi.org/10.1007/s11245-023-09932-3

  • Morris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018). Towards an artificially empathic conversational agent for mental health applications: System design and user perceptions. Journal of Medical Internet Research, 20(6), e10148.

    Article  PubMed  PubMed Central  Google Scholar 

  • Patel, V., Saxena, S., Lund, C., Thornicroft, G., Baingana, F., Bolton, P., & Unutzer, J. (2018). The Lancet Commission on global mental health and sustainable development. The Lancet, 392(10157), 1553–1598. https://doi.org/10.1016/s0140-6736(18)31612-x

  • Pathare, S., Brazinova, A., & Levav, I. (2018). Care gap: A comprehensive measure to quantify unmet needs in mental health. Epidemiology and Psychiatric Sciences, 27(5), 463–467. https://doi.org/10.1017/S2045796018000100

    Article  PubMed  PubMed Central  Google Scholar 

  • Paul, S. C., Bartmann, N., & Clark, J. L. (2021). Customizability in conversational agents and their impact on health engagement. Human Behavior and Emerging Technologies, 3(5), 1141–1152.

    Article  Google Scholar 

  • Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023). Is ChatGPT a general-purpose natural language processing task solver?. arXiv preprint arXiv:2302.06476.

  • Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.

    Google Scholar 

  • Rana, J. (2023). The pros and cons of healthcare chatbots | REVE Chat. REVE Chat. https://www.revechat.com/blog/healthcare-chatbots/

  • Rapp, A., Curti, L., & Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, 151, 102630.

    Article  Google Scholar 

  • Rashkin, H., Smith, E. M., Li, M., & Boureau, Y. L. (2018). Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207.

  • Reddy, M., Reddy, T., & Kumar, N. (2021). AI chatbot for mental health : PTSD assessment . International Journal of Creative Research Thoughts (IJCRT), 9(5), f145–f150. https://ijcrt.org/papers/IJCRT2105554.pdf

  • Sabour, S., Zhang, W., Xiao, X., Zhang, Y., Zheng, Y., Wen, J., Zhao, J., & Huang, M. (2023). A chatbot for mental health support: exploring the impact of Emohaa on reducing mental distress in China. Frontiers in Digital Health, 5. https://doi.org/10.3389/fdgth.2023.1133987

  • Santoro, E., & Monin, B. (2023). The AI effect: People rate distinctively human attributes as more essential to being human after learning about artificial intelligence advances. Journal of Experimental Social Psychology, 107, 104464. https://doi.org/10.1016/j.jesp.2023.104464

  • Seiferth, C., Vogel, L., Aas, B., Brandhorst, I., Carlbring, P., Conzelmann, A., Esfandiari, N., Finkbeiner, M., Hollmann, K., Lautenbacher, H., Meinzinger, E., Newbold, A., Opitz, A., Renner, T. J., Sander, L. B., Santangelo, P. S., Schoedel, R., Schuller, B., Stachl, C. & Löchner, J. (2023). How to e-mental health: A guideline for researchers and practitioners using digital technology in the context of mental health. Nature Mental Health, 1, 542–554. https://doi.org/10.1038/s44220-023-00085-1

  • Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023). Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1), 46–57.

    Article  Google Scholar 

  • Singh, O. P. (2023). Artificial intelligence in the era of ChatGPT-Opportunities and challenges in mental health care. Indian Journal of Psychiatry, 65(3), 297.

    Article  PubMed  PubMed Central  Google Scholar 

  • Smith, K. A., Blease, C., Faurholt-Jepsen, M., Firth, J., Van Daele, T., Moreno, C., Carlbring, P., Ebner-Priemer, U. W., Koutsouleris, N., Riper, H., Mouchabac, S., Torous, J., & Cipriani, A. (2023). Digital mental health: Challenges and next steps. BMJ Mental Health, 26(1), e300670. https://doi.org/10.1136/bmjment-2023-300670

    Article  PubMed  PubMed Central  Google Scholar 

  • Suganuma, S., Sakamoto, D., & Shimoyama, H. (2018). An embodied conversational agent for unguided internet-based cognitive behavior therapy in preventative mental health: Feasibility and acceptability pilot trial. JMIR Mental Health, 5(3), e10454.

    Article  PubMed  PubMed Central  Google Scholar 

  • Suta, P., Lan, X., Wu, B., Mongkolnam, P., & Chan, J. H. (2020). An overview of machine learning in chatbots. International Journal of Mechanical Engineering and Robotics.

  • Thompson, D. C., & Bentzien, J. (2020). Crowdsourcing and open innovation in drug discovery: Recent contributions and future directions. Drug Discovery Today, 25(12), 2284–2293.

  • Tielman, M. L., Jonker, C. M., & van Riemsdijk, M. B. (2018, July). What Should I Do? Deriving Norms from Actions, Values and Context. In MRC@ IJCAI (pp. 35–40).

  • Trautmann, S., Rehm, J., & Wittchen, H. U. (2016). The economic costs of mental disorders: Do our societies react appropriately to the burden of mental disorders? EMBO Reports, 17(9), 1245–1249.

    Article  PubMed  PubMed Central  Google Scholar 

  • Vaidyam, A., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. (2019). Chatbots and conversational agents in mental health: A review of the psychiatric landscape. Canadian Journal of Psychiatry, 64(7), 456–464. https://doi.org/10.1177/0706743719839389

    Article  PubMed  PubMed Central  Google Scholar 

  • Vis, C., Schuurmans, J., Aouizerate, B., Atipei Craggs, M., Batterham, P., Bührmann, L., Calear, A., Cerga Pashoja, A., Christensen, H., Dozeman, E., Duedal Pedersen, C., Ebert, D. D., Etzelmueller, A., Fanaj, N., Finch, T. L., Hanssen, D., Hegerl, U., Hoogendoorn, A., Mathiasen, K., Riper, H. (2023). Effectiveness of self-guided tailored implementation strategies in integrating and embedding internet-based cognitive behavioral therapy in routine mental health care: Results of a multicenter stepped-wedge cluster randomized trial. Journal of Medical Internet Research, 25, e41532. https://doi.org/10.2196/41532

  • Walsh, C. G., Ribeiro, J. D., & Franklin, J. C. (2017). Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science, 5(3), 457–469.

    Article  Google Scholar 

  • Wampold, B. E., & Flückiger, C. (2023). The alliance in mental health care: Conceptualization, evidence, and clinical applications. World Psychiatry, 22(1), 25–41.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wang, X., & Nakatsu, R. (2013). How do people talk with a virtual philosopher: Log analysis of a real-world application. In Entertainment Computing–ICEC 2013: 12th International Conference, ICEC 2013, São Paulo, Brazil, October 16–18. (2013). Proceedings 12, 132–137. Springer.

    Google Scholar 

  • World Health Organization. (2021). Mental disorders. Retrieved from https://www.who.int/news-room/fact-sheets/detail/mental-disorders

  • Wu, F., Lu, C., Zhu, M., Chen, H., Zhu, J., Yu, K., ... & Pan, Y. (2020). Towards a new generation of artificial intelligence in China. Nature Machine Intelligence, 2(6), 312–316.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gagan Jain.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lopes, E., Jain, G., Carlbring, P. et al. Talking Mental Health: a Battle of Wits Between Humans and AI. J. technol. behav. sci. (2023). https://doi.org/10.1007/s41347-023-00359-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s41347-023-00359-6

Keywords

Navigation