Skip to main content
SpringerLink
Log in

Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes

  • Original, Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Purpose: Chart notes provide a low-cost data source that could help characterize what occurs in treatment with sufficient precision to improve management of care. This study assessed the interrater reliability of treatment content coded from chart notes and evaluated its concordance with content coded from transcribed treatment sessions. Method: Fifty randomly selected and digitally recorded treatment events were transcribed and coded for practice content. Independent coders then applied the same code system to chart notes for these same treatment events. Analysis: We measured reliability and concordance of practice occurrence and extensiveness at two levels of specificity: practices (full procedures) and steps (subcomponents of those procedures). Results: For chart notes, practices had moderate interrater reliability (M k = 0.50, M ICC = 0.56) and steps had moderate (M ICC = 0.74) to substantial interrater reliability (M k = 0.78). On average, 2.54 practices and 5.64 steps were coded per chart note and 4.53 practices and 13.10 steps per transcript. Across sources, ratings for 64% of practices and 41% of steps correlated significantly, with those with significant correlations generally demonstrating moderate concordance (practice M r = 0.48; step M r = 0.47). Forty one percent of practices and 34% of steps from transcripts were also identified in the corresponding chart notes. Conclusion: Chart notes provide an accessible data source for evaluating treatment content, with different levels of specificity posing tradeoffs for validity and reliability, which in turn may have implications for chart note interfaces, training, and new metrics to support accurate, reliable, and efficient measurement of clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aguirre, R.R., Suarez, O., Fuentes, M., & Sanchez-Gonzalez, M.A. (2019) Electronic health record implementation: A review of resources and tools. Cureus, 11(9), e5649 https://doi.org/10.7759/cureus.5649

  • Akoglu, H. (2018). User’s guide to correlation coefficients. Turkish Journal of Emergency Medicine, 18(3), 91–93. https://doi.org/10.1016/j.tjem.2018.08.001

    Article  PubMed  PubMed Central  Google Scholar 

  • Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W. J., Sinsky, C. A., & Gilchrist, V. J. (2017). Tethered to the EHR: Primary care physician workload assessment using EHR event log data and time-motion observations. Annals of Family Medicine, 15(5), 419–426. https://doi.org/10.1370/afm.2121

    Article  PubMed  PubMed Central  Google Scholar 

  • Asim, M. N., Wasim, M., Khan, M. U. G., Mahmood, W., & Abbasi, H. M. (2018). A survey of ontology learning techniques and applications. The Journal of Biological Database and Curation, 2018, 1–24. https://doi.org/10.1093/database/bay101

  • Bailey, J. (2008). First steps in qualitative data analysis: Transcribing. Family Practice, 25(2), 127–131. https://doi.org/10.1093/fampra/cmn003

    Article  PubMed  Google Scholar 

  • Bearman, S. K., Schneiderman, R. L., & Zoloth, E. (2017). Building an evidence base for effective supervision practices: An analogue experiment of supervision to increase EBT fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 44(2), 293–307. https://doi.org/10.1007/s10488-016-0723-8

    Article  PubMed  Google Scholar 

  • Becker, K. D., Boustani, M., Gellatly, R., & Chorpita, B. F. (2018). Forty years of engagement research in children’s mental health services: Multidimensional measurement and practice elements. Journal of Clinical Child and Adolescent Psychology, 47(1), 1–23. https://doi.org/10.1080/15374416.2017.1326121

    Article  PubMed  Google Scholar 

  • Becker, K. D., Buckingham, S., Rith-Najarian, L., & Kline, E. (2015a). The common elements of treatment engagement for clinically high-risk youth and youth with first-episode psychosis. Early Intervention Psychiatry, 10(6), 455–467https://doi.org/10.1111/eip.12283

  • Becker, K. D., Lee, B. R., Daleiden, E. L., Lindsey, M., Brandt, N. E., & Chorpita, B. F. (2015b). The common elements of engagement in children’s mental health services: Which elements for which outcomes? Journal of Clinical Child and Adolescent Psychology, 44(1), 30–43. https://doi.org/10.1080/15374416.2013.814543

    Article  PubMed  Google Scholar 

  • Borntrager, C. F., Chorpita, B. F., Orimoto, T., Allison Love, M., Mueller, C., Love, M. W., A., & Mueller, C. W. (2013). Validity of clinican’s self-reported practice elements on monthly treatment and progress summary. Journal of Behavioral Health Services & Research, 42(3), 376382. https://doi.org/10.1007/s11414-013-9363-x

    Google Scholar 

  • Briggie, A. M., Hilsenroth, M. J., Conway, F., Muran, C., J., & Jackson, J. M. (2016). Patient comfort with audio or video recording of their psychotherapy sessions: Relation to symptomatology, treatment refusal, duration, and outcome. Professional Psychology: Research and Practice, 47(1), 66–76. https://doi.org/10.1037/a0040063

    Article  Google Scholar 

  • Brookman-Frazee, L., Stadnick, N. A., Lind, T., Roesch, S., Terrones, L., Barnett, M. L., Regan, J., Kennedy, C. A., Garland, F., A., & Lau, A. S. (2021). Therapist-observer concordance in ratings of EBP strategy delivery: Challenges and targeted directions in pursuing pragmatic measurement in children’s mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 48(1), 155–170. https://doi.org/10.1007/s10488-020-01054-x

    Article  PubMed  Google Scholar 

  • Calvo, R. A., Milne, D. N., Hussain, M. S., & Christensen, H. (2017). Natural language processing in mental health applications using non-clinical texts. Natural Language Engineering, 23(5), 649–685. https://doi.org/10.1017/S1351324916000383

    Article  Google Scholar 

  • Carrell, D. S., Halgrim, S., Tran, D. T., Buist, D. S. M., Chubak, J., Chapman, W. W., & Savova, G. (2014). Using natural language processing to improve efficiency of manual chart abstraction in research: The case of Breast cancer recurrence. American Journal of Epidemiology, 179(6), 749–758. https://doi.org/10.1093/aje/kwt441

    Article  PubMed  PubMed Central  Google Scholar 

  • Chapman, J. E., Sheidow, A. J., Henggeler, S. W., Halliday-Boykins, C. A., & Cunningham, P. B. (2008). Developing a measure of therapist adherence to contingency management: An application of the many-facet rasch model. Journal of Child and Adolescent Substance Abuse, 17(3), 47–68. https://doi.org/10.1080/15470650802071655

    Article  PubMed  Google Scholar 

  • Chapman, C. L., Burlingame, G. M., Gleave, R., Rees, F., Beecher, M., & Porter, G. S. (2012). Clinical prediction in group psychotherapy. Psychotherapy Research, 22(6), 673–681. https://doi.org/10.1080/10503307.2012.702512

    Article  PubMed  Google Scholar 

  • Chapman, J. E., Mccart, M. R., Letourneau, E. J., & Sheidow, A. J. (2013). Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting and Clinical Psychology, 81(4), 674–680. https://doi.org/10.1037/a0033021

    Article  PubMed  PubMed Central  Google Scholar 

  • Child and Adolescent Mental Health Division (2003). Service provider monthly treatment and progress summary. Hawaii Department of Health Child and Adolescent Mental Health Division.

  • Chorpita, B. F., & Becker, K. D. (2017–2022). Coordinated knowledge systems: Connecting evidence to action to engage students in school-based mental health (Project No. 187173) William T. Grant Foundation. https://wtgrantfoundation.org/browse-grants#/grant/187173

  • Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566–579. https://doi.org/10.1037/a0014565

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Bernstein, A., Daleiden, E. L., Weisz, J., Gibbons, R., Glisson, C., Green, E. P., Hoagwood, K., Jensen, P. S., Kelleher, K., Landsverk, J., Mayberg, S., Miranda, J., Palinkas, L., & Schoenwald, S. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 114–123. https://doi.org/10.1007/s10488-007-0151-x

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Bernstein, A. D. (2016). At the intersection of health information technology and decision support: Measurement feedback systems… and beyond. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 471–477. https://doi.org/10.1007/s10488-015-0702-5

    Article  PubMed  Google Scholar 

  • Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. https://doi.org/10.1016/0895-4356(90)90159-m

    Article  PubMed  CAS  Google Scholar 

  • Cohen, Z., Delgadillo, J., & DeRubeis, R. (2021). Personalized treatment approaches. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and behavior change (50th ed., pp. 673–703). John Wiley & Sons, Inc.

  • Creed, T. A., Salama, L., Slevin, R., Tanana, M., Imel, Z., Narayanan, S., & Atkins, D. C. (2022). Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): A study protocol. BMC Health Services Research, 22(1), 1177 https://doi.org/10.1186/s12913-022-08519-9

  • Dilley, J. A., Reuer, J. R., Colman, V., & Norman, R. K. (2009). From making pamphlets to making policies: Results from a collaborative training to increase knowledge, motivation, and self-efficacy for achieving public health policy and systems change. Health Promotion Practice, 10(2 Suppl), 138S–145S https://doi.org/10.1177/1524839909332601

  • Garg, A. X., Adhikari, N. K. J., Mcdonald, H., Rosas-Arellano, P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, B. R. B. (2005). Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: A systematic review. American Medical Association, 293(10), 1223–1238. https://doi.org/10.1001/jama.293.10.1223

  • Garland, A., Brookman-Frazee, L., Hurlburt, M., Accurso, E., Zoffness, R., Haine-Schlagel, R., & Ganger, W. (2010). Mental health care for children with disruptive behavior problems: A view inside therapists’ offices. Psychiatric Services, 61(8), 788–795 https://doi.org/10.1176/appi.ps.61.8.788

  • Gearing, R. E., Mian, I. A., Barber, J., Ickowicz, & Abel (2006). A methodology for conducting retrospective chart review research in child and adolescent psychiatry. Journal of the Canadian Academy of Child Adolescent Psychiatry, 15(3), 126–134. PMID: 18392182

    PubMed  PubMed Central  Google Scholar 

  • Gruber, T. (1995). Toward priniciples for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies, 43, 907–928. https://doi.org/10.1006/ijhc.1995.1081

    Article  Google Scholar 

  • He, Q., Veldkamp, B. P., Glas, C. A. W., & de Vries, T. (2017). Automated assessment of patients’ self-narratives for posttraumatic stress disorder screening using natural language processing and text mining. Assessment, 24(2), 157–172. https://doi.org/10.1177/1073191115602551

    Article  PubMed  Google Scholar 

  • Herschell, A. D., Quetsch, L. B., & Kolko, D. J. (2020). Measuring adherence to key teaching techniques in an evidence-based treatment: A comparison of caregiver, therapist, and behavior observation ratings. Journal of Emotional and Behavioral Disorders, 28(2), 92–103. https://doi.org/10.1177/1063426618821901

    Article  Google Scholar 

  • Hogue, A., Turner, R. M., Liddle, H. A., Rowe, C., Dakof, G. A., & Lapann, K. (1998). Treatment adherence and differentiation in individual versus family therapy for adolescent substance abuse. Journal of Counseling Psychology, 45(1), 104–114. https://doi.org/10.1037/0022-0167.45.1.104

    Article  Google Scholar 

  • Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2

    Article  PubMed  Google Scholar 

  • Hogue, A., Bobek, M., Porter, N., MacLean, A., Bruynesteyn, L., Jensen-Doss, A., & Henderson, C. E. (2022). Therapist self-report of fidelity to core elements of family therapy for adolescent behavior problems: Psychometrics of a pragmatic quality indicator tool. Administration and Policy in Mental Health and Mental Health Services Research, 49(2), 298–311. https://doi.org/10.1007/s10488-021-01164-0

    Article  PubMed  Google Scholar 

  • Houlding, D. (2019). June 25). Using natural language processing to manage healthcare records. Azure.

  • Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Researcha, 37(3), 230–244. https://doi.org/10.1007/s10488-009-0251-x

    Article  Google Scholar 

  • JaKa, M. M., French, S. A., Wolfson, J., Jeffery, R. W., Lorencatto, F., Michie, S., Langer, S. L., Levy, R. L., & Sherwood, N. E. (2017). Feasibility of standardized methods to specify behavioral pediatric obesity prevention interventions. Journal of Behavioral Medicine, 40(5), 730–739. https://doi.org/10.1007/s10865-017-9845-z

    Article  PubMed  PubMed Central  Google Scholar 

  • Joukes, E., Abu-Hanna, A., Cornet, R., & de Keizer, N. F. (2018). Time spent on dedicated patient care and documentation tasks before and after the introduction of a structured and standardized electronic health record. Applied Clinical Informatics, 9(1), 46–53. https://doi.org/10.1055/s-0037-1615747

    Article  PubMed  PubMed Central  Google Scholar 

  • Kelley, S. D., De Andrade, A. R. V., Sheffer, E., & Bickman, L. (2010). Exploring the black box: Measuring youth treatment process and progress in usual care. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 287–300. https://doi.org/10.1007/s10488-010-0298-8

    Article  PubMed  Google Scholar 

  • Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting untraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

    Article  PubMed  PubMed Central  Google Scholar 

  • Kowalski-Dobson, T., Quinn, M., & Lachance, L. (2018). Rebuilding community-determined partnerships. Health Promotion Practice, 19(1_suppl), 55S–56S. https://doi.org/10.1177/1524839918795893

    Article  PubMed  Google Scholar 

  • Landis, J. R., & Kock, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310

    Article  PubMed  CAS  Google Scholar 

  • Larsen, K. R., Michie, S., Hekler, E. B., Gibson, B., Spruijt-Metz, D., Ahern, D., Cole-Lewis, H., Ellis, R. J. B., Hesse, B., Moser, R. P., & Yi, J. (2017). Behavior change interventions: The potential of ontologies for advancing science and practice. Journal of Behavioral Medicine, 40(1), 6–22. https://doi.org/10.1007/s10865-016-9768-0

    Article  PubMed  Google Scholar 

  • Last, B. S., Buttenheim, A. M., Timon, C. E., Mitra, N., & Beidas, R. S. (2021). Systematic review of clinician-directed nudges in healthcare contexts. British Medical Journal Open, 11(7), https://doi.org/10.1136/bmjopen-2021-048801

  • Liber, J. M., McLeod, B. D., Van Widenfelt, B. M., Goedhart, A. W., van der Leeden, A. J. M., Utens, E. M. W. J., & Treffers, P. D. A. (2010). Examining the relation between the therapeutic alliance, treatment adherence, and outcome of cognitive behavioral therapy for children with anxiety disorders. Behavior Therapy, 41(2), 172–186. https://doi.org/10.1016/j.beth.2009.02.003

    Article  PubMed  Google Scholar 

  • Lindsey, M. A., Brandt, N. E., Becker, K. D., Lee, B. R., Barth, R. P., Daleiden, E. L., & Chorpita, B. F. (2014) Identifying the common elements of treatment engagement interventions in children’s mental health services. Clinical Child and Family Psychology Review 17(3), 283–298. https://doi.org/10.1007/s10567-013-0163-x

  • Madson, M. B., Mohn, R. S., Zuckoff, A., Schumacher, J. A., Kogan, J., Hutchison, S., Magee, E., & Stein, B. (2013). Measuring client perceptions of motivational interviewing: Factor analysis of the client evaluation of motivational interviewing scale. Journal of Substance Abuse Treatment, 44(3), 330–335. https://doi.org/10.1016/j.jsat.2012.08.015

    Article  PubMed  Google Scholar 

  • Madson, M. B., Mohn, R. S., Schumacher, J. A., & Landry, A. S. (2015). Measuring client experiences of motivational interviewing during a lifestyle intervention. Measurement and Evaluation in Counseling and Development, 48(2), 140–151. https://doi.org/10.1177/0748175614544687

    Article  PubMed  Google Scholar 

  • Marcus, M., Westra, H., Angus, L., & Kertes, A. (2011). Client experiences of motivational interviewing for generalized anxiety disorder: A qualitative analysis. Psychotherapy Research, 21(4), 447–461. https://doi.org/10.1080/10503307.2011.578265

    Article  PubMed  Google Scholar 

  • Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19(2), 181–193. https://doi.org/10.1080/10503300802688460

    Article  PubMed  PubMed Central  Google Scholar 

  • McLeod, B.D., & Weisz, J.R. (2010) The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39(3), 436–443 https://doi.org/10.1080/15374411003691750

  • McNeely, J., Adam, A., Rotrosen, J., Wakeman, S. E., Wilens, T. E., Kannry, J., Rosenthal, R. N., Wahle, A., Pitts, S., Farkas, S., Rosa, C., Peccoralo, L., Waite, E., Vega, A., Kent, J., Craven, C. K., Kaminski, T. A., Firmin, E., Isenberg, B., & Hamilton, L. (2021). Comparison of methods for alcohol and drug screening in primary care clinics. JAMA Network Open, 4(5), https://doi.org/10.1001/jamanetworkopen.2021.10721

  • Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., Eccles, M. P., Cane, J., & Wood, C. E. (2013). The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: Building an international consensus for the reporting of behavior change interventions. Annals of Behavioral Medicine, 46(1), 81–95. https://doi.org/10.1007/s12160-013-9486-6

    Article  PubMed  Google Scholar 

  • Michie, S., West, R., Finnerty, A. N., Norris, E., Wright, A. J., Marques, M. M., Johnston, M., Kelly, M. P., Thomas, J., & Hastings, J. (2020). Representation of behaviour change interventions and their evaluation: Development of the upper level of the behaviour change intervention ontology. Wellcome Open Research, 5, 1–27. https://doi.org/10.12688/WELLCOMEOPENRES.15902.1

    Article  Google Scholar 

  • Miranda, J., Hepner, K. A., Azocar, F., Greenwood, G., Ngo, V., & Burnam, M. A. (2010). Development of a patient-report measure of psychotherapy for depression. Administration and Policy in Mental Health and Mental Health Services Research, 37(3), 245–253. https://doi.org/10.1007/s10488-009-0238-7

    Article  PubMed  Google Scholar 

  • Moyers, T. B., Rowell, L. N., Manuel, J. K., Ernst, D., & Houck, J. M. (2016). The Motivational Interviewing Treatment Integrity Code (MITI 4): Rationale, preliminary reliability and validity. Journal of Substance Abuse Treatment, 65, 36–42. https://doi.org/10.1016/j.jsat.2016.01.001

    Article  PubMed  PubMed Central  Google Scholar 

  • Nakamura, B. J., Daleiden, E. L., & Mueller, C. W. (2007). Validity of treatment target progress ratings as indicators of youth improvement. Journal of Child and Family Studies, 16(5), 729–741. https://doi.org/10.1007/s10826-006-9119-9

    Article  Google Scholar 

  • Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75(6), 829–841. https://doi.org/10.1037/0022-006X.75.6.829

    Article  PubMed  Google Scholar 

  • PracticeWise. (2023). PracticeWise. Evidence-based services (PWEBS) database. Retrieved from https://welcome.practicewise.com/

  • Regan, J., Park, A. L., & Chorpita, B. F. (2019). Choices in Treatment Integrity: Considering the protocol and consultant recommendations in child and adolescent therapy. Journal of Clinical Child and Adolescent Psychology, 48(1), 79–89. https://doi.org/10.1080/15374416.2016.1212360

    Article  Google Scholar 

  • Revelle, W. (2022). Psych: Procedures for psychological, psychometric, and Personality Research (2.2.9). Northwestern University.

  • Rowley, J. (2006). What do we need to know about wisdom? Management Decision, 44(9), 1246–1257. https://doi.org/10.1108/00251740610707712

    Article  Google Scholar 

  • Schoenwald, S. K. (2008). Toward evidence-based transport of evidence-based treatments: MST as an example. Journal of Child and Adolescent Substance Abuse, 17(3), 69–91. https://doi.org/10.1080/15470650802071671

    Article  Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. https://doi.org/10.1007/s10488-010-0321-0

    Article  PubMed  Google Scholar 

  • Shiner, B., D’Avolio, L., Nguyen, T., Zayed, M., Young-Xu, Y., Desai, R., Schnurr, P., Fiore, L., & Watts, B. (2013). Measuring use of evidence based psychotherapy for posttraumatic stress disorder. Administration and Policy in Mental Health and Mental Health Services Research, 40(4), 311–318. https://doi.org/10.1007/s10488-012-0421-0

    Article  PubMed  Google Scholar 

  • Shiner, B., Levis, M., Dufort, V. M., Patterson, O. V., Watts, B. V., DuVall, S. L., Russ, C. J., & Maguen, S. (2021). Improvements to PTSD quality metrics with natural language processing. Journal of Evaluation in Clinical Practice, 28(4), 520–530. https://doi.org/10.1111/jep.13587

    Article  PubMed  Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420

    Article  PubMed  CAS  Google Scholar 

  • Simon, H.A., Deutsch, K.D., & Shubik, M. (1971). Designing organizations for an information-rich world. In M. Greenberger (Ed.), Computers, Communications, and the Public Interest. Johns Hopkins Press.

  • Southam-Gerow, M. A., McLeod, B. D., Arnold, C. C., Rodríguez, A., Cox, J. R., Reise, S. P., Bonifay, W. E., Weisz, J. R., & Kendall, P. C. (2016). Initial development of a treatment adherence measure for cognitive-behavioral therapy for child anxiety. Psychological Assessment, 28(1), 70–80. https://doi.org/10.1037/pas0000141

    Article  PubMed  Google Scholar 

  • Stengel, D., Bauwens, K., Walter, M., Köpfer, T., & Ekkernkamp, A. (2004). Comparison of handheld computer-assisted and conventional paper chart documentation of medical records: A randomized controlled trial. The Journal of Bone and Joint Surgery, 86(3), 553–560. https://doi.org/10.2106/00004623-200403000-00014

    Article  PubMed  Google Scholar 

  • Trochim, W. M. K., Milstein, B., Wood, B. J., Jackson, S., & Pressler, V. (2016). Setting objectives for community and systems change: An application of concept mapping for planning a statewide health improvement initiative. Http://Dx Doi Org/, 5(1), https://doi.org/10.1177/1524839903258020

  • Viera, A., & Garrett, J. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37(5), 360–363.

    PubMed  Google Scholar 

  • Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., Daleiden, E. L., Bearman, S. K., & Weisz, J. R. (2013). Tracking evidence based practice with touth: Validity of the MATCH and standard manual consultation records. Journal of Clinical Child and Adolescent Psychology, 42(1), 44–55. https://doi.org/10.1080/15374416.2012.700505

    Article  PubMed  Google Scholar 

  • Weisz, J., Ng, M.m & Bearman, S. (2014) Odd couple? Reenvisioning the relation between science and practice in the dissemination-implementation era. Clinical Psychological Science, 2(1), 58–74 https://doi.org/10.1177/2167702613501307

  • Weersing, V. R., Weisz, J. R., & Donenberg, G. R. (2002). Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child and Adolescent Psychology, 31(2), 168–180. https://doi.org/10.1207/S15374424JCCP3102_03

    Article  PubMed  Google Scholar 

  • Wu, E. G., Becker, K. D., Kim, R. E., Martinez, J. I., Gamarra, J. M., & Chorpita, B. F. (2022). How do treatment protocols affect the use of engagement practices in youth mental health services? Administration and Policy in Mental Health and Mental Health Services Research, 49(6), 943–961. https://doi.org/10.1007/s10488-022-01210-5

    Article  PubMed  Google Scholar 

  • Yehia, E., Boshnak, H., AbdelGaber, S., Abdo, A., & Elzanfaly, D.S. (2019) Ontology-based clinical information extraction from physician’s free-text notes. Journal of Biomedical Informatics, 98, 103276. https://doi.org/10.1016/j.jbi.2019.103276

    Article  PubMed  Google Scholar 

  • Zhang, T., Schoene, A. M., Ji, S., & Ananiadou, S. (2022). Natural language processing applied to mental Illness detection: A narrative review. Digital Medicine, 5(1), https://doi.org/10.1038/s41746-022-00589-7

Download references

Funding

Partial financial support was received from the William T. Grant Foundation (Award #187173; PI: Chorpita; Co-PI: Becker), who did not participate in the design or conduct of the study; collection, management, analysis, or interpretation of the data; or preparation, review, or approval of the manuscript.

Author information

Authors and Affiliations

  1. Department of Psychology, University of California Los Angeles, Franz Hall, Los Angeles, CA, 90095, United States of America

    Benjamin M. Isenberg, Hyun Seon Park, Danielle Keenan-Miller & Bruce F. Chorpita

  2. Department of Psychology, University of South Carolina, 1512 Pendleton Street, Columbia, SC, 29208, United States of America

    Kimberly D. Becker, Eleanor Wu & Wendy Chu

Authors
  1. Benjamin M. Isenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kimberly D. Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eleanor Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hyun Seon Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wendy Chu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Danielle Keenan-Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bruce F. Chorpita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruce F. Chorpita.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

Approval by the institutional review board of the University of California, Los Angeles, as well as those institutional review boards of participating service agencies that requested independent reviews, was obtained for this research. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Isenberg, B.M., Becker, K.D., Wu, E. et al. Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes. Adm Policy Ment Health 51, 103–122 (2024). https://doi.org/10.1007/s10488-023-01316-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-023-01316-4

Keywords

Search

Navigation