Skip to main content
SpringerLink
Log in

Equivalence of chatbot and paper-and-pencil versions of the De Jong Gierveld loneliness scale

  • Published:
Current Psychology Aims and scope Submit manuscript

Abstract

Technological progress provides health professionals with an excellent opportunity to take advantage of these developments and contribute to the development of efficient ways of diagnosing, monitoring, treating and assisting users. The purpose of this work is to present the results of a study conducted to examine the quantitative equivalence of paper-and-pencil and a voice-based conversational assistant, popularly known as a “chatbot”, as means to administer tests. One hundred and eight undergraduate university students completed both versions of the De Jong Gierveld Loneliness Scale. The interval between the first and second administration was set at four days. Validity, internal structure, internal consistency and equivalence of chatbot administration mode were assessed. A confirmatory factor analysis was used to verify the factor structure and provided a two-factor structure. Validity and internal consistency are adequate. These results support the feasibility of using chatbots for loneliness assessment in a sample of undergraduate university students and other populations in future.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abd-Alrazaq, A. A., Rababeh, A., Alajlani, M., Bewick, B. M., & Househ, M. (2020). Effectiveness and safety of using Chatbots to improve mental health: Systematic review and meta-analysis. Journal of Medical Internet Research, 22(7), e16021. https://doi.org/10.2196/16021.

    Article  PubMed  PubMed Central  Google Scholar 

  • Alberdi, A., Weakley, A., Aztiria, A., Schmitter-Edgecombe, M., & Cook, D. J. (2018). Automatic assessment of functional health decline in older adults based on smart home data. Journal of Biomedical Informatics, 81, 119–130. https://doi.org/10.1016/j.jbi.2018.03.009.

    Article  Google Scholar 

  • Bazzano, A. N., Martin, J., Hicks, E., Faughnan, M., & Murphy, L. (2017). Human-centred design in global health: A scoping review of applications and contexts. PLoS One, 12(11), e0186744. https://doi.org/10.1371/journal.pone.0186744.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barrigón, M. L., Rico-Romano, A. M., Ruiz-Gómez, M., Delgado-Gómez, D., Barahona, I., Aroca, F., & Baca-García, E. (2017). Comparative study of pencil-and-paper and electronic formats of GHQ-12, WHO-5 and PHQ-9 questionnaires. Revista de Psiquiatría y Salud Mental, 10, 160–167. https://doi.org/10.1016/j.rpsm.2016.12.002.

    Article  PubMed  Google Scholar 

  • Bellagente, P., Crema, C., Depari, A., Ferrari, P., Flammini, A., Lanfranchi, G., Lenzi, G., Maddiona, M., Rinaldi, S., Sisinni, E., & Ziliani, G. (2018). Remote and non-invasive monitoring of elderly in a smart city context. In 2018 IEEE sensors applications symposium (SAS) (pp. 1-6). IEEE. https://doi.org/10.1109/SAS.2018.8336732.

  • Bentler, P. M. (2006). EQS 6 structural equations program manual (Vol. 6). Encino: Multivariate software.

    Google Scholar 

  • Bonsaksen, T., Opseth, T. M., Misund, A. R., Geirdal, A. Ø., Fekete, O. R., & Nordli, H. (2019). The de Jong Gierveld loneliness scale used with Norwegian clubhouse members: Psychometric properties and associated factors. International Journal of Psychosocial Rehabilitation, 22(2), 88–100.

    Google Scholar 

  • Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of covariance structures. British Journal of Mathematical and Statistical Psychology, 37(1), 62–83. https://doi.org/10.1111/j.2044-8317.1984.tb00789.x.

    Article  PubMed  Google Scholar 

  • Cacioppo, J. T., & Cacioppo, S. (2018). The growing problem of loneliness. The Lancet, 391, 426. https://doi.org/10.1016/S0140-6736(18)30142-9.

    Article  Google Scholar 

  • Cacioppo, J. T., Cacioppo, S., & Boomsma, D. I. (2014). Evolutionary mechanisms for loneliness. Cognition & Emotion, 28(1), 3–21. https://doi.org/10.1080/02699931.2013.837379.

    Article  Google Scholar 

  • Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186. https://doi.org/10.1126/science.aal4230.

    Article  PubMed  Google Scholar 

  • Chamorro-Premuzic, T., Winsborough, D., Sherman, R. A., & Hogan, R. (2016). New talent signals: Shiny new objects or a brave new world? Industrial and Organizational Psychology, 9(3), 621–640. https://doi.org/10.1017/iop.2016.6.

    Article  Google Scholar 

  • Courtin, E., & Knapp, M. (2017). Social isolation, loneliness and health in old age: A scoping review. Health and social Care in the Community, 25(3), 799–812. https://doi.org/10.1111/hsc.12311.

  • de Cock, C., Milne-Ives, M., van Velthoven, M. H., Alturkistani, A., Lam, C., & Meinert, E. (2020). Effectiveness of conversational agents (virtual assistants) in health care: Protocol for a systematic review. JMIR Research Protocols, 9(3), e16934. https://doi.org/10.2196/16934.

    Article  PubMed  PubMed Central  Google Scholar 

  • De Jong Gierveld, J., & Tesch-Romer, C. (2012). Loneliness in old age in eastern and Western European societies: Theoretical perspectives. European Journal of Aging, 9(4), 285–295. https://doi.org/10.1007/s10433-012-0248-2.

    Article  PubMed  PubMed Central  Google Scholar 

  • De Jong Gierveld, J., & Van Tilburg, T. (2006). A 6-item scale for overall, emotional, and social loneliness: Confirmatory tests on survey data. Research on Aging, 28, 582–598. https://doi.org/10.1177/0164027506289723.

    Article  Google Scholar 

  • De Jong Gierveld, J., & Van Tilburg, T. (2010). The De Jong Gierveld short scales for emotional and social loneliness: Tested on data from 7 countries in the UN generations and gender surveys. European Journal of Ageing, 7(2), 121–130. https://doi.org/10.1007/s10433-010-0144-6.

    Article  PubMed  PubMed Central  Google Scholar 

  • De Jong Gierveld, J., Van Tilburg, T., & Dykstra, P. A. (2006). Loneliness and social isolation. In D. Perlman & A. Vangelisti (Eds.), The Cambridge handbook of personal relationships (pp. 485–500). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Desideri, L., Ottaviani, C., Malavasi, M., di Marzio, R., & Bonifacci, P. (2019). Emotional processes in human-robot interaction during brief cognitive testing. Computers in Human Behavior, 90, 331–342. https://doi.org/10.1016/j.chb.2018.08.013.

    Article  Google Scholar 

  • Ercole, V., & Parr, J. (2020). Problems of loneliness and its impact on health and well- being. In: W. Leal, T. wall, A. Azul, L. Brandli & P. Özuyar (Eds.), Good Health and Well-Being. Encyclopedia of the UN Sustainable Development Goals. Basel, Switzerland: Springer nature Switzerland AG. https://doi.org/10.1007/978-3-319-69627-0_127-1.

  • Fokkema, T., De Jong Gierveld, J., & Dykstra, P. A. (2012). Cross-national differences in older adult loneliness. The Journal of Psychology, 146, 201–228. https://doi.org/10.1080/00223980.2011.631612.

    Article  PubMed  Google Scholar 

  • Hajek A., & König H. (2017). The association of falls with loneliness and social exclusion: evidence from the DEAS German Ageing Survey. BMC Geriatrics 17(1), 204. https://doi.org/10.1186/s12877-017-0602-5.

  • Harari, G. M., Müller, S. R., Aung, M. S., & Rentfrow, P. J. (2017). Smartphone sensing methods for studying behavior in everyday life. Current Opinion in Behavioral Sciences, 18, 83–90. https://doi.org/10.1016/j.cobeha.2017.07.018.

    Article  Google Scholar 

  • Hewson, C. (2014). Conducting research on the internet-a new era. The Psychologist, 27(12), 946–950.

    Google Scholar 

  • Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., & Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6), 82–97.

    Article  Google Scholar 

  • Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118.

    Article  Google Scholar 

  • Hughes, M. E., Waite, L. J., Hawkley, L. C., & Cacioppo, J. T. (2004). A short scale for measuring loneliness in large surveys: Results from two population-based studies. Research on Aging, 26(6), 655–672. https://doi.org/10.1177/0164027504268574.

    Article  PubMed  PubMed Central  Google Scholar 

  • Khosravi, P., Rezvani, A., & Wiewiora, A. (2016). The impact of technology on older adults’ social isolation. Computers in Human Behavior, 63, 594–603. https://doi.org/10.1016/j.chb.2016.05.092.

  • Kim, S., Lee, J., & Gweon, G. (2019). Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-12).

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159.

    Article  Google Scholar 

  • Lara, E., Martín-María, N., De la Torre-Luque, A., Koyanagi, A., Vancampfort, D., Izquierdo, A., & Miret, M. (2019). Does loneliness contribute to mild cognitive impairment and dementia? A systematic review and meta-analysis of longitudinal studies. Ageing Research Reviews, 52, 7–16. https://doi.org/10.1016/j.arr.2019.03.002.

    Article  PubMed  Google Scholar 

  • Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A., & Coiera, E. (2018). Conversational agents in healthcare: A systematic review. Journal of the American Medical Informatics Association, 25(9), 1248–1258. https://doi.org/10.1093/jamia/ocy072.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lee, N., Madotto, A., & Fung, P. (2019). Exploring social Bias in Chatbots using stereotype knowledge. In Proceedings of the 2019 Workshop on Widening NLP (pp. 177-180).

  • Lucas, G. M., Rizzo, A. S., Gratch, J., Scherer, S., Stratou, G., Boberg, J., & Morency, L. P. (2017). Reporting mental health symptoms: Breaking down barriers to care with virtual human interviewers. Frontiers in Robotics and AI, 4, 51. https://doi.org/10.3389/frobt.2017.00051.

    Article  Google Scholar 

  • McDevitt-Murphy, M. E., Luciano, M. T., & Zakarian, R. J. (2018). Use of ecological momentary assessment and intervention in treatment with adults. Focus, 16(4), 370–375. https://doi.org/10.1176/appi.focus.20180017.

    Article  PubMed  PubMed Central  Google Scholar 

  • Miller, G. (2012). The smartphone psychology Manifiesto. Perspectives on Psychology Science, 7(3), 221–237.

    Article  Google Scholar 

  • Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342.

    Article  PubMed  Google Scholar 

  • Okeke, F., Sobolev, M., & Estrin, D. (2018). Towards a framework for Mobile behavior change research. Proceedings of the technology, mind, and society, Washington, DC, USA.

  • Perlman, D., & Peplau, L. (1982). Theoretical approaches to loneliness. In: D. Perlman y L. Peplau (Eds.), Loneliness: A sourcebook of current theory, research and therapy (pp. 123–134). New York: Wiley.

  • Pickard, M. D., Roster, C. A., & Chen, Y. (2016). Revealing sensitive information in personal interviews: Is self-disclosure easier with humans or avatars and under what conditions? Computers in Human Behavior, 65, 23–30. https://doi.org/10.1016/j.chb.2016.08.004.

    Article  Google Scholar 

  • Rabbitt, S. M., Kazdin, A. E., & Scassellati, B. (2015). Integrating socially assistive robotics into mental healthcare interventions: Applications and recommendations for expanded use. Clinical Psychology Review, 35, 35–46. https://doi.org/10.1016/j.cpr.2014.07.001.

    Article  PubMed  Google Scholar 

  • Salas, A. J., Reynolds, A., & Thomas, C. M. (2018). Technology Design for Frailty. In D. Chau & T. F. Osborne (Eds.), Using technology to improve Care of Older Adults (pp. 103–133). New York: Springer Publishing Company.

    Google Scholar 

  • Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., & Torous, J. B. (2019). Chatbots and conversational agents in mental health: A review of the psychiatric landscape. The Canadian Journal of Psychiatry, 64(7), 456–464. https://doi.org/10.1177/0706743719828977.

    Article  PubMed  Google Scholar 

  • Valtorta, N. K., Moore, D. C., Barron, L., Stow, D., & Hanratty, B. (2018). Older adults’ social relationships and health care utilization: A systematic review. American Journal of Public Health, 108(4), e1–e10. https://doi.org/10.2105/AJPH.2017.304256.

  • Victor, C. R., & Yang, K. (2012). The prevalence of loneliness among adults: A case study of the United Kingdom. The Journal of Psychology, 146(1–2), 85–104. https://doi.org/10.1080/00223980.2011.613875.

    Article  PubMed  Google Scholar 

  • Weigold, A., Weigold, I. K., Drakeford, N. M., & Dykema, S. S. (2016). Equivalence of paper-and-pencil and computerized self-report surveys in older adults. Computers in Human Behavior, 54, 407–413. https://doi.org/10.1016/j.chb.2015.08.033.

    Article  Google Scholar 

  • Weigold, A., Weigold, I. K., & Natera, S. N. (2018). Mean scores for self-report surveys completed using paper-and-pencil and computers: A meta-analytic test of equivalence. Computers in Human Behavior, 86, 153–164. https://doi.org/10.1016/j.chb.2018.04.038.

    Article  Google Scholar 

  • Weiss, R. S. (1973). Loneliness: The experience of emotional and social isolation. Cambridge: MIT Press.

    Google Scholar 

  • White, M. K., Maher, S. M., Rizio, A. A., & Bjorner, J. B. (2018). A meta-analytic review of measurement equivalence study findings of the SF-36® and SF-12® health surveys across electronic modes compared to paper administration. Quality of Life Research, 27(7), 1757–1767. https://doi.org/10.1007/s11136-018-1851-2.

    Article  PubMed  PubMed Central  Google Scholar 

  • World Health Organization (WHO) (2016). What are integrated people-centred health services? Retrieved from https://www.who.int/servicedeliverysafety/areas/people-centred-care/en/. Accessed 4 November 2019.

Download references

Acknowledgements

We are grateful to the undergraduate university students from Universitat Jaume I who voluntarily participated in the study.

Funding

This research was supported by the Ministry of Science and Innovation and the Spanish Foundation for Science and Technology (FECYT) with reference number: FCT-18-13677.

Author information

Authors and Affiliations

  1. Department of Developmental Educational and Social Psychology and Methodology, Institute of New Imaging Technologies, Jaume I University, Av. Vicent Sos Baynat, s/n 12071, Castelló de la Plana, Spain

    Antonio Caballer

  2. Department of Computer Languages and Systems, Institute of New Imaging Technologies, Jaume I University, Castelló de la Plana, Spain

    Oscar Belmonte

  3. Institute of New Imaging Technologies, Jaume I University, Castelló de la Plana, Spain

    Andrea Castillo & Arturo Gasco

  4. Department Engineering of Industrial Systems and Design, Institute of New Imaging Technologies, Jaume I University, Castelló de la Plana, Spain

    Emilio Sansano

  5. Department of Computer Science and Engineering, Institute of New Imaging Technologies, Jaume I University, Castelló de la Plana, Spain

    Raul Montoliu

Authors
  1. Antonio Caballer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oscar Belmonte
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Castillo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arturo Gasco
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Emilio Sansano
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Raul Montoliu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonio Caballer.

Ethics declarations

Conflict of Interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Ethical Approval

This project was approved by the Ethical Commission at Universitat Jaume I with reference number: CD/58/2019.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Caballer, A., Belmonte, O., Castillo, A. et al. Equivalence of chatbot and paper-and-pencil versions of the De Jong Gierveld loneliness scale. Curr Psychol 41, 6225–6232 (2022). https://doi.org/10.1007/s12144-020-01117-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12144-020-01117-0

Keywords

Search

Navigation