Abstract
This study examined the convergence between two methods of thought content analysis, manual coding by trained raters and computer-generated word counts, in a sample of clinically depressed participants assessed before and after treatment. Automated word count programs have traditionally used longer narrative texts so their utility for shorter thought samples is uncertain. Aims were to evaluate their direct correspondence and to determine whether the two methods yield similar results in assessing change from pre- to post-treatment. Thirty participants recorded in-the-moment thoughts during random phone-based signaling. Thought samples were analyzed for presence of negative emotion (NE), positive emotion (PE), and self-focus (SF), using hand coded ratings and automated word counts. Correlations between ratings and word counts for each of the three content categories were significant for all but post-treatment NE. Thought samples rated as showing the presence of NE, PE, or SF showed significantly higher NE, PE, and SF (respectively) word counts than those without. Comparisons of pre/post data showed significant decreases in NE and no differences in PE across both methods; increases in SF emerged only for ratings. While limited by a small sample size, these findings suggest that word count analyses may be a reasonable replacement for more laborious hand coding in thought sampling data, but there may be important differences across content categories. These results contribute to knowledge about the methodology of thought sampling analysis in clinical samples.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The dataset analyzed for the current study are not publicly available because they consist of audio samples that cannot be completely de-identified.
References
Alpers, G. W., Winzelberg, A. J., Classen, C., Roberts, H., Dev, P., Koopman, C., & Taylor, B., C (2005). Evaluation of computerized text analysis in an internet breast cancer support group. Computers in Human Behavior, 21(2), 361–376. https://doi.org/10.1016/j.chb.2004.02.008.
Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the Beck Depression Inventory-II. Psychological Corporation.
Ben-Zeev, D., Young, M. A., & Madsen, J. W. (2009). Retrospective recall of affect in clinically depressed individuals and controls. Cognition & Emotion, 23(5), 1021–1040.
Boumparis, N., Karyotaki, E., Kleiboer, A., Hofmann, S. G., & Cuijpers, P. (2016). The effect of psychotherapeutic interventions on positive and negative affect in depression: A systematic review and meta-analysis. Journal of Affective Disorders, 202, 153–162. https://doi.org/10.1016/j.jad.2016.05.019.
Boyd, R. L., Ashokkumar, A., Seraj, S., & Pennebaker, J. W. (2022). The development and psychometric properties of LIWC-22
Chamberlain, J., & Haaga, D. A. F. (1999). Convergent Validity of Cognitive Assessment Methods. Behavior Modification, 23(2), 294–315. https://doi.org/10.1177/0145445599232007.
Christopher, M. S., Jacob, K. L., Neuhaus, E. C., Neary, T. J., & Fiola, L. A. (2009). Cognitive and behavioral changes related to symptom improvement among patients with a mood disorder receiving intensive cognitive-behavioral therapy. Journal of Psychiatric Practice, 15(2), 95–102. https://doi.org/10.1097/01.pra.0000348362.11548.5f. psyh.
Eddington, K. M., Silvia, P. J., Foxworth, T. E., Hoet, A., & Kwapil, T. R. (2015). Motivational deficits differentially predict improvement in a randomized trial of self-system therapy for depression. Journal of Consulting and Clinical Psychology, 83(3), 602–616. https://doi.org/10.1037/a0039058.
Eddington, K. M., Burgin, C. J., Silvia, P. J., Fallah, N., Majestic, C., & Kwapil, T. R. (2017). The Effects of psychotherapy for major depressive disorder on Daily Mood and Functioning: A longitudinal experience sampling study. Cognitive Therapy and Research, 41(2), 266–277. https://doi.org/10.1007/s10608-016-9816-7.
Edwards, T., & Holtzman, N. S. (2017). A meta-analysis of correlations between depression and first person singular pronoun use. Journal of Research in Personality, 68, 63–68. https://doi.org/10.1016/j.jrp.2017.02.005.
First, M. B., Gibbon, M., Spitzer, R. L., Williams, J. B. W., & Benjamin, L. S. (1997). Structured clinical interview for DSM?IV axis II personality disorders (SCID-II). Washington, DC: American Psychiatric Press.
First, M. B., Spitzer, R. L., Gibbon, M., & Williams, J. B. W. (2002). Structured clinical interview for DSM?IV?TR axis I disorders. New York: Biometrics Research, New York State Psychiatric Institute.
Hofheinz, C., Reder, M., & Michalak, J. (2020). How specific is cognitive change? A randomized controlled trial comparing brief cognitive and mindfulness interventions for depression. Psychotherapy Research, 30(5), 675–691. https://doi.org/10.1080/10503307.2019.1685138.
Ingram, R. E., Cruet, D., Johnson, B. R., & Wisnicki, K. S. (1988). Self-focused attention, gender, gender role, and vulnerability to negative affect. Journal of Personality and Social Psychology, 55, 967–978. https://doi.org/10.1037/0022-3514.55.6.967.
Josephson, B. R., Rose, R. D., & Singer, J. A. (1999). Thought sampling after mood induction in depressed vs non-depressed college students. Imagination Cognition and Personality, 19(1), 27–37. https://doi.org/10.2190/UWA1-HF64-B26N-8L8P. APA PsycInfo.
Krejtz, I., Rohnka, N., Holas, P., Rusanowska, M., & Nezlek, J. B. (2020). Manifestations of clinical depression in daily life: A daily diary study of descriptions of naturally occurring events. Cognition & Emotion, 34(8), 1664–1675. https://doi.org/10.1080/02699931.2020.1795627.
McVay, J. C., & Kane, M. J. (2009). Conducting the train of thought: Working memory capacity, goal neglect, and mind wandering in an executive-control task. Journal of Experimental Psychology: Learning Memory and Cognition, 35(1), 196–204. https://doi.org/10.1037/a0014104.
Mor, N., & Winquist, J. (2002). Self-focused attention and negative affect: A meta-analysis. Psychological Bulletin, 128(4), 638–662.
Mor, N., Doane, L. D., Adam, E. K., Mineka, S., Zinbarg, R. E., Griffith, J. W., Craske, M. G., Waters, A., & Nazarian, M. (2010). Within-person variations in self-focused attention and negative affect in depression and anxiety: A diary study. Cognition and Emotion, 24(1), 48–62. https://doi.org/10.1080/02699930802499715.
Mota, N. B., Weissheimer, J., Ribeiro, M., de Paiva, M., Avilla-Souza, J., Simabucuru, G., Chaves, M. F., Cecchi, L., Cirne, J., Cecchi, G., Rodrigues, C., Copelli, M., & Ribeiro, S. (2020). Dreaming during the Covid-19 pandemic: Computational assessment of dream reports reveals mental suffering related to fear of contagion. Plos One, 15(11), https://doi.org/10.1371/journal.pone.0242903.
Parrish, B. P., Cohen, L. H., Gunthert, K. C., Butler, A. C., Laurenceau, J. P., & Beck, J. S. (2009). Effects of cognitive therapy for depression on daily stress-related variables. Behaviour Research and Therapy, 47(5), 444–448. https://doi.org/10.1016/j.brat.2009.02.005. psyh.
Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. Austin, TX: University of Texas at Austin.
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and Computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54. https://doi.org/10.1177/0261927X09351676.
Tov, W., Ng, K. L., Lin, H., & Qiu, L. (2013). Detecting well-being via computerized content analysis of brief diary entries. Psychological Assessment, 25(4), 1069–1078. https://doi.org/10.1037/a0033007.
van der Velden, A. M., Kuyken, W., Wattar, U., Crane, C., Pallesen, K. J., Dahlgaard, J., Fjorback, L. O., & Piet, J. (2015). A systematic review of mechanisms of change in mindfulness-based cognitive therapy in the treatment of recurrent major depressive disorder. Clinical Psychology Review, 37, 26–39. https://doi.org/10.1016/j.cpr.2015.02.001.
Wood, J. V., Saltzberg, J. A., Neale, J. M., Stone, A. A., & Rachmiel, T. B. (1990). Self-focused attention, coping responses, and distressed mood in everyday life. Journal of Personality and Social Psychology, 58(6), 1027–1036. https://doi.org/10.1037//0022-3514.58.6.1027.
Zelenski, J. M., & Larsen, R. J. (2000). The distribution of Basic Emotions in Everyday Life: A state and trait perspective from experience Sampling Data. Journal of Research in Personality, 34(2), 178–197. https://doi.org/10.1006/jrpe.1999.2275.
Zheng, X., & Schweickert, R. (2021). Comparing hall Van de Castle coding and linguistic Inquiry and Word Count using canonical correlation analysis. Dreaming, 31(3), 207–224. https://doi.org/10.1037/drm0000173.
Ziemer, K. S., & Korkmaz, G. (2017). Using text to predict psychological and physical health: A comparison of human raters and computerized text analysis. Computers in Human Behavior, 76, 122–127. https://doi.org/10.1016/j.chb.2017.06.038.
American Psychiatric Association (2000). Diagnostic and statistical manual of mental disorders (4th ed - Text revision). American Psychiatric Publishing, Inc. http://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=1994-97698-000&site=ehost-live
Funding
This research was supported by the National Institute of Mental Health of the National Institutes of Health under award number R21MH090414 (Trial Registration Number NCT02134678) and by grants provided by the Undergraduate Research, Scholarship and Creativity office at the University of North Carolina at Greensboro. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This research was conducted in accordance with the principles embodied in the Declaration of Helsinki and was approved by the UNCG IRB; all participants gave informed consent to participate.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors have no conflicts to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Stiles, L., Frazier, A. & Eddington, K.M. What were you Thinking? A Comparison of Rater Coding and word Counts for Content Analysis of Thought Samples in Depression. J Rat-Emo Cognitive-Behav Ther (2023). https://doi.org/10.1007/s10942-023-00507-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s10942-023-00507-0