Advertisement

Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems

Abstract

Implementation of measurement-based care (MBC) by child-serving community mental health providers, particularly school-based providers, is low. To inform user-centered design of measurement feedback systems (MFSs) and MBC implementation more broadly, semi-structured interviews were conducted with 61 middle and high school students. Interviews explored student preferences for and perceived helpfulness of different assessment methods and use of MFS in counseling. Results indicate that student preference for digitally-administered assessment is equivocal, with preferences being influenced by student perceptions of the ease of use, impersonalization, and confidentiality. Students with exposure to the MFS found it helpful when used by their provider to share assessment feedback.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

References

  1. Alpert, J. S. (2016). The electronic medical record in 2016: Advantages and disadvantages. Digital Medicine,2, 48–51. https://doi.org/10.4103/2226-8561.189504.

  2. American Psychological Association. (2008). Task Force on Evidence-Based Practice with Children and Adolescents.

  3. Bergman, H., Kornor, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., Tollefesen, T. K., & Bjorndal, A. (2018). Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database of Systematic Reviews 2018, Issue 8. Art. No.: CD011729. https://doi.org/10.1002/14651858.CD011729.pub2.

  4. Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry,47(10), 1114. https://doi.org/10.1097/CHI.0b013e3181825af8.

  5. Bickman, L., Douglas, S., & R.,…, Hoagwood, K., (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health,43, 410–425. https://doi.org/10.1007/s10488-015-0647-8.

  6. Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services,62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011.

  7. Boo, J., & Vispoel, W. (2012). Computer versus paper-and-pencil assessment of educational development: A comparison of psychometric features and examinee preferences. Psychological Reports,111(2), 443–460. https://doi.org/10.2466/10.03.PRO.111.5.443-460.

  8. Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health,27(3), 281–291. https://doi.org/10.1093/pubmed/fdi031.

  9. Chorpita, B. F., & Weisz, J. R. (2009). Modular approach to therapy for children with anxiety, depression, trauma, or conduct problems (MATCH-ADTC). Satellite Beach, FL: PracticeWise.

  10. Connors, E. H., Arora, P., Curtis, L. N., & Stephan, S. H. (2015). Evidence-based assessment in school mental health. Cognitive and Behavioral Practice,22(1), 60–73. https://doi.org/10.1016/j.cbpra.2014.03.008.

  11. Courage, C., & Baxter, K. (2005). Understanding your users: A practical guide to user requirements methods, tools, and techniques. San Francisco: Morgan Kaufmann Publishers.

  12. Duong, M. T., Lyon, A. R., Ludwig, K., Wasse, J. K., & McCauley, E. (2016). Student perceptions of the acceptability and utility of standardized and idiographic assessment in school mental health. International Journal of Mental Health Promotion,18(1), 19–63. https://doi.org/10.1080/14623730.2015.1079429.

  13. Dyrborg, J., Larsen, F. W., Nielson, S., Byman, J., Nielson, B. B., & Gautre-Delay, F. (2000). The Children’s Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice: Substance and reliability as judged by intraclass correlations. European Child and Adolescent Psychiatry,9(3), 195–201. https://doi.org/10.1007/s007870070043.

  14. Eisen, S. V., Dickey, B., & Sederer, L. I. (2000). A self-report symptom and problem rating scale to increase inpatients’ involvement in treatment. Psychiatric Services,51(3), 349–353. https://doi.org/10.1176/appi.ps.51.3.349.

  15. Fleming, M. (1997). Evaluating adolescent responses to a computer health information system. Journal of Adolescent Health,20, 166. https://doi.org/10.1019/S1054-139X(97)87625-2.

  16. Fouladi, R. T., McCarthy, C. J., & Moller, N. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment,9, 204–215. https://doi.org/10.1177/10791102009002011.

  17. Gleacher, A. A., Olin, S. S., …, Hoagwood, K. (2016). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health,43, 426–440. https://doi.org/10.1007/s10488-015-0642-0.

  18. Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health,11(2), 322–333. https://doi.org/10.1111/j.1524-4733.2007.00231.x.

  19. Haynes, S. N., Mumma, G. H., & Pinson, C. (2009). Idiographic assessment: Conceptual and psychometric foundations of individualized behavioral assessment. Clinical Psychology Review,29, 179–191. https://doi.org/10.1016/j.cpr.2008.12.003.

  20. Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research,15(9), 1277–1288. https://doi.org/10.1177/1049732305276687.

  21. Jensen-Doss, A., Becker, E. M., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., et al. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research,45, 48–61. https://doi.org/10.1007/s10488-016-0763-0.

  22. Jenson-Doss, A., & Hawley, K. M. (2011). Understanding clinicians’ diagnostic practices: Attitudes toward the utility of diagnosis and standardized diagnostic tools. Administration and Policy in Mental Health,38(6), 476–485. https://doi.org/10.1007/s10488-011-0334-3.

  23. Jensen-Doss, A., & Weisz, J. (2008). Diagnostic agreement predicts treatment process and outcomes in youth mental health clinics. Journal of Consulting and Clinical Psychology,76(5), 711–722. https://doi.org/10.1037/0022-006X.76.5.711.

  24. Kazak, A. E., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T. R., Vargas, L. A., et al. (2010). A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist,65(2), 85–97. https://doi.org/10.1037/a0017784.

  25. Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L.,…, Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews, Issue 7, Art. No.: CD011119. https://doi.org/10.1002/14651858.CD011119.pub2.

  26. Kokkonen, E. W. J., Davis, S. A., Lin, H.-C., Dabade, T. S., Feldeman, S. R., & Fleischer, A. B. (2013). Use of electronic medical records differs by specialty and office settings. Journal of American Medical Informatics Association,20, 33–38. https://doi.org/10.1136/amiajnl-2012-001609.

  27. Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., et al. (2002). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology and Psychotherapy,9(2), 91–103. https://doi.org/10.1002/cpp.324.

  28. Lavin, P., Barry, L., & Williams, S. (2017). Measurement-based care in behavioral health [power point slides]. https://www.jointcommission.org/assets/1/6/bhc_Joint_Commission_measures_webinar_041117.pdf.

  29. Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., …, Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi-org.flagship.luc.edu/10.1001/jamapsychiatry.2018.3329.

  30. Lindhiem, O., Bennett, C. B., Orimoto, T. E., & Kolko, D. J. (2016). A meta-analysis of personalized treatment goals in psychotherapy: A preliminary report and call for more studies. Clinical Psychology: Science and Practice,23(2), 165–176.

  31. Lyon, A. R., Dorsey, S., Pullmann, M., Silbaugh-Cowdin, J., & Berliner, L. (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health and Mental Health Services Research,42(1), 47–60.

  32. Lyon, A. R., Knaster Wasse, J., Ludwig, K., Zachry, M., Bruns, E. J., Unutzer, J., et al. (2016a). The Contextualized Technology Adaptation Process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research,43(3), 394–409. https://doi.org/10.1007/s10488-015-0637-x.

  33. Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Administration and Policy in Mental Health,43, 344–349. https://doi.org/10.1007/s10488-015-0704-3.

  34. Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016b). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research,43, 441–466. https://doi.org/10.1007/s10488-016-0719-4.

  35. Lyon, A. R., Lewis, C. C., Melvin, A., Boyd, M., Nicodimos, S., Liu, F. F., et al. (2016c). Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: Description and application to clinical feedback systems. Implementation Science,11(1), 1–18. https://doi.org/10.1186/s13012-016-0495-2.

  36. Lyon, A. R., Ludwig, K., Knaster Wasse, J., Bergstrom, A., Hendrix, E., & McCauley, E. (2016d). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research,43, 122–134. https://doi.org/10.1007/s10488-015-0626-0.

  37. Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Knaster Wasse, J., & McCauley, E. (2019). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child and Adolescent Psychology,48, 168–179. https://doi.org/10.1080/15374416.2017.1280808.

  38. Mangunkusomo, R. T., Duisterhout, J. S., De Graaff, N., Maarsingh, E. J., de Koning, H. J., & Raat, H. (2006). Internet versus paper mode of health and health behavior questionnaires in elementary schools: Asthma and fruit as examples. Journal of School Health,76, 80–86.

  39. Mangunkusomo, R. T., Moorman, P. W., Berg-de Ruiter, A. E., & Van Der Lei, J. (2005). Internet-administered adolescent questionnaires compared with a paper version in a randomized study. Journal of Adolescent Health,36(1), 71–76. https://doi.org/10.1016/j.jadohealth.2004.02.020.

  40. Menachemi, N., & Collum, T. H. (2011). Benefits and drawbacks of electronic record systems. Risk Management and Healthcare Policy,4, 47–55. https://doi.org/10.2147/RMHP.S12985.

  41. Merikangas, K. R., He, J., Burstein, M., Swendsen, J., Avenevoli, S., Case, B., …, Olfson, M. (2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the national comorbidity survey-adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 50(1), 32–45. https://doi.org/10.1016/j.jaac.2010.10.006.

  42. Muhr, T. (2004). User’s manual for ALTAS.ti 5.0. Berlin: ATLAS.ti. Scientific Software Development GmbH.

  43. Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health,43, 379–393.

  44. Norman, D. A., & Draper, S. W. (Eds.). (1986). User-centered system design: New perspectives on human–computer interaction. Hillsdale, NJ: Lawrence Earlbaum Associates.

  45. Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed methods designs in implementation research. Administration and Policy in Mental Health,38, 44–53. https://doi.org/10.1007/s10488-010-0314-z.

  46. Poston, J. M., & Hanson, W. E. (2010). Meta-analysis of psychological assessment as a therapeutic intervention. Psychological Assessment,22, 203–212. https://doi.org/10.1037/a0018679.

  47. Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., et al. (2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services and Research,44(2), 177–194.

  48. Rakovshik, S. G., & McManus, F. (2010). Establishing evidence-based training in cognitive behavioral therapy: A review of current empirical findings and theoretical guidance. Clinical Psychology Review,30(5), 496–516. https://doi.org/10.1016/j.cpr.2010.03.004.

  49. Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice,22, 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010.

  50. Seidman, E., Chorpita, B. F., Reay, W. E., Stelk, W., Garland, A. F., Jutash, K., …, Ringeisen, H. (2010). A framework for measurement feedback to improve decision-making in mental health. Administration Policy in Mental Health,37, 128–131. https://doi.org/10.1007/s10488-009-0260-9.

  51. Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., et al. (1983). A children’s global assessment scale (CGAS). Archives of General Psychiatry,40(1), 1223–1231. https://doi.org/10.1001/archpsyc.1983.017901000074010.

  52. Tharinger, D. J., Finn, S. E., Gentry, L., Hamilton, A., Fowler, J., Matson, M., …, Walkowiak, J. (2009). Therapeutic assessment with children: A pilot study of treatment acceptability and outcome. Journal of Personality Assessment,91, 238–244. https://doi.org/10.1080/00223890902794275.

  53. Unützer, J., Chan, Y. F., Hafer, E., Knaster, J., Shields, A., Powers, D., et al. (2012). Quality improvement with pay-for performance incentives in integrated behavioral health care. American Journal of Public Health,102(6), e41–e45.

  54. Unützer, J., Katon, W., Callahan, C. M., Williams, J. W., Jr., Hunkeler, E., Harpole, L., et al. (2002). Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial. JAMA,288(22), 2836–2845.

  55. Vallejo, M. A., Mananes, G., Comeche, M. I., & Diaz, M. I. (2008). Comparison between administration via internet and paper-and-pencil administration of two clinical instruments: SCL-90-R and GHQ-28. Journal of Behavioral Therapy and Experimental Psychiatry,39(3), 201–208. https://doi.org/10.1016/j.jbtep.2007.04.001.

  56. Van de Looij-Jansen, P. M., & Wilde, E. J. D. (2008). Comparison of web-based versus paper-and-pencil self-administered questionnaire: Effects on health indicators in Dutch adolescents. Health Services Research,43(5), 1708–1721. https://doi.org/10.1111/j.1475-6773.2008.00860.

  57. Vispoel, W. P., Boo, J., & Bieiler, T. (2001). Computerized and paper-and-pencil versions of the Rosenberg Self-Esteem Scale: A comparison of psychometric features and respondent preferences. Educational and Psychological Measurement,61(3), 461–474. https://doi.org/10.1177/00131640121971329.

  58. Williams, J. W., Katon, W., Lin, E. H., Nöel, P. H., Worchel, J., Cornell, J., et al. (2004). The effectiveness of depression care management on diabetes-related outcomes in older patients. Annals of Internal Medicine,140(12), 1015–1024.

Download references

Acknowledgements

This publication was made possible in part by funding from Grant K08MH095939, awarded to the last author (Aaron R. Lyon) from the National Institute of Mental Health (NIMH). The authors would also like to thank the participating students, Seattle Children’s Hospital, and Public Health of Seattle and King County for their support of this project. This paper was not previously presented at a meeting.

Author information

Correspondence to Ashley M. Mayworm.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the Ethical Standards of the Institutional Review Board of the University of Washington (43941 UW IRB) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mayworm, A.M., Kelly, B.M., Duong, M.T. et al. Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems. Adm Policy Ment Health (2020). https://doi.org/10.1007/s10488-020-01010-9

Download citation

Keywords

  • Measurement-based care
  • Evidence-based assessment
  • Measurement feedback system
  • Digitally-delivered assessment
  • School mental health