Skip to main content

Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems

Abstract

Implementation of measurement-based care (MBC) by child-serving community mental health providers, particularly school-based providers, is low. To inform user-centered design of measurement feedback systems (MFSs) and MBC implementation more broadly, semi-structured interviews were conducted with 61 middle and high school students. Interviews explored student preferences for and perceived helpfulness of different assessment methods and use of MFS in counseling. Results indicate that student preference for digitally-administered assessment is equivocal, with preferences being influenced by student perceptions of the ease of use, impersonalization, and confidentiality. Students with exposure to the MFS found it helpful when used by their provider to share assessment feedback.

This is a preview of subscription content, access via your institution.

References

  1. Alpert, J. S. (2016). The electronic medical record in 2016: Advantages and disadvantages. Digital Medicine,2, 48–51. https://doi.org/10.4103/2226-8561.189504.

    Article  Google Scholar 

  2. American Psychological Association. (2008). Task Force on Evidence-Based Practice with Children and Adolescents.

  3. Bergman, H., Kornor, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., Tollefesen, T. K., & Bjorndal, A. (2018). Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database of Systematic Reviews 2018, Issue 8. Art. No.: CD011729. https://doi.org/10.1002/14651858.CD011729.pub2.

  4. Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry,47(10), 1114. https://doi.org/10.1097/CHI.0b013e3181825af8.

    Article  PubMed  Google Scholar 

  5. Bickman, L., Douglas, S., & R.,…, Hoagwood, K., (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health,43, 410–425. https://doi.org/10.1007/s10488-015-0647-8.

    Article  Google Scholar 

  6. Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services,62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011.

    Article  PubMed  Google Scholar 

  7. Boo, J., & Vispoel, W. (2012). Computer versus paper-and-pencil assessment of educational development: A comparison of psychometric features and examinee preferences. Psychological Reports,111(2), 443–460. https://doi.org/10.2466/10.03.PRO.111.5.443-460.

    Article  PubMed  Google Scholar 

  8. Bowling, A. (2005). Mode of questionnaire administration can have serious effects on data quality. Journal of Public Health,27(3), 281–291. https://doi.org/10.1093/pubmed/fdi031.

    Article  PubMed  Google Scholar 

  9. Chorpita, B. F., & Weisz, J. R. (2009). Modular approach to therapy for children with anxiety, depression, trauma, or conduct problems (MATCH-ADTC). Satellite Beach, FL: PracticeWise.

    Google Scholar 

  10. Connors, E. H., Arora, P., Curtis, L. N., & Stephan, S. H. (2015). Evidence-based assessment in school mental health. Cognitive and Behavioral Practice,22(1), 60–73. https://doi.org/10.1016/j.cbpra.2014.03.008.

    Article  Google Scholar 

  11. Courage, C., & Baxter, K. (2005). Understanding your users: A practical guide to user requirements methods, tools, and techniques. San Francisco: Morgan Kaufmann Publishers.

    Google Scholar 

  12. Duong, M. T., Lyon, A. R., Ludwig, K., Wasse, J. K., & McCauley, E. (2016). Student perceptions of the acceptability and utility of standardized and idiographic assessment in school mental health. International Journal of Mental Health Promotion,18(1), 19–63. https://doi.org/10.1080/14623730.2015.1079429.

    Article  PubMed  Google Scholar 

  13. Dyrborg, J., Larsen, F. W., Nielson, S., Byman, J., Nielson, B. B., & Gautre-Delay, F. (2000). The Children’s Global Assessment Scale (CGAS) and Global Assessment of Psychosocial Disability (GAPD) in clinical practice: Substance and reliability as judged by intraclass correlations. European Child and Adolescent Psychiatry,9(3), 195–201. https://doi.org/10.1007/s007870070043.

    CAS  Article  PubMed  Google Scholar 

  14. Eisen, S. V., Dickey, B., & Sederer, L. I. (2000). A self-report symptom and problem rating scale to increase inpatients’ involvement in treatment. Psychiatric Services,51(3), 349–353. https://doi.org/10.1176/appi.ps.51.3.349.

    CAS  Article  PubMed  Google Scholar 

  15. Fleming, M. (1997). Evaluating adolescent responses to a computer health information system. Journal of Adolescent Health,20, 166. https://doi.org/10.1019/S1054-139X(97)87625-2.

    Article  Google Scholar 

  16. Fouladi, R. T., McCarthy, C. J., & Moller, N. (2002). Paper-and-pencil or online? Evaluating mode effects on measures of emotional functioning and attachment. Assessment,9, 204–215. https://doi.org/10.1177/10791102009002011.

    Article  PubMed  Google Scholar 

  17. Gleacher, A. A., Olin, S. S., …, Hoagwood, K. (2016). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health,43, 426–440. https://doi.org/10.1007/s10488-015-0642-0.

    Article  Google Scholar 

  18. Gwaltney, C. J., Shields, A. L., & Shiffman, S. (2008). Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: A meta-analytic review. Value in Health,11(2), 322–333. https://doi.org/10.1111/j.1524-4733.2007.00231.x.

    Article  PubMed  Google Scholar 

  19. Haynes, S. N., Mumma, G. H., & Pinson, C. (2009). Idiographic assessment: Conceptual and psychometric foundations of individualized behavioral assessment. Clinical Psychology Review,29, 179–191. https://doi.org/10.1016/j.cpr.2008.12.003.

    Article  PubMed  Google Scholar 

  20. Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research,15(9), 1277–1288. https://doi.org/10.1177/1049732305276687.

    Article  PubMed  Google Scholar 

  21. Jensen-Doss, A., Becker, E. M., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., et al. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research,45, 48–61. https://doi.org/10.1007/s10488-016-0763-0.

    Article  PubMed  Google Scholar 

  22. Jenson-Doss, A., & Hawley, K. M. (2011). Understanding clinicians’ diagnostic practices: Attitudes toward the utility of diagnosis and standardized diagnostic tools. Administration and Policy in Mental Health,38(6), 476–485. https://doi.org/10.1007/s10488-011-0334-3.

    Article  Google Scholar 

  23. Jensen-Doss, A., & Weisz, J. (2008). Diagnostic agreement predicts treatment process and outcomes in youth mental health clinics. Journal of Consulting and Clinical Psychology,76(5), 711–722. https://doi.org/10.1037/0022-006X.76.5.711.

    Article  PubMed  Google Scholar 

  24. Kazak, A. E., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T. R., Vargas, L. A., et al. (2010). A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist,65(2), 85–97. https://doi.org/10.1037/a0017784.

    Article  PubMed  Google Scholar 

  25. Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L.,…, Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews, Issue 7, Art. No.: CD011119. https://doi.org/10.1002/14651858.CD011119.pub2.

  26. Kokkonen, E. W. J., Davis, S. A., Lin, H.-C., Dabade, T. S., Feldeman, S. R., & Fleischer, A. B. (2013). Use of electronic medical records differs by specialty and office settings. Journal of American Medical Informatics Association,20, 33–38. https://doi.org/10.1136/amiajnl-2012-001609.

    Article  Google Scholar 

  27. Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., et al. (2002). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology and Psychotherapy,9(2), 91–103. https://doi.org/10.1002/cpp.324.

    Article  Google Scholar 

  28. Lavin, P., Barry, L., & Williams, S. (2017). Measurement-based care in behavioral health [power point slides]. https://www.jointcommission.org/assets/1/6/bhc_Joint_Commission_measures_webinar_041117.pdf.

  29. Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., …, Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi-org.flagship.luc.edu/10.1001/jamapsychiatry.2018.3329.

    Article  Google Scholar 

  30. Lindhiem, O., Bennett, C. B., Orimoto, T. E., & Kolko, D. J. (2016). A meta-analysis of personalized treatment goals in psychotherapy: A preliminary report and call for more studies. Clinical Psychology: Science and Practice,23(2), 165–176.

    Google Scholar 

  31. Lyon, A. R., Dorsey, S., Pullmann, M., Silbaugh-Cowdin, J., & Berliner, L. (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health and Mental Health Services Research,42(1), 47–60.

    Article  Google Scholar 

  32. Lyon, A. R., Knaster Wasse, J., Ludwig, K., Zachry, M., Bruns, E. J., Unutzer, J., et al. (2016a). The Contextualized Technology Adaptation Process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research,43(3), 394–409. https://doi.org/10.1007/s10488-015-0637-x.

    Article  PubMed  Google Scholar 

  33. Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery. Administration and Policy in Mental Health,43, 344–349. https://doi.org/10.1007/s10488-015-0704-3.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016b). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research,43, 441–466. https://doi.org/10.1007/s10488-016-0719-4.

    Article  PubMed  Google Scholar 

  35. Lyon, A. R., Lewis, C. C., Melvin, A., Boyd, M., Nicodimos, S., Liu, F. F., et al. (2016c). Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: Description and application to clinical feedback systems. Implementation Science,11(1), 1–18. https://doi.org/10.1186/s13012-016-0495-2.

    Article  Google Scholar 

  36. Lyon, A. R., Ludwig, K., Knaster Wasse, J., Bergstrom, A., Hendrix, E., & McCauley, E. (2016d). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research,43, 122–134. https://doi.org/10.1007/s10488-015-0626-0.

    Article  PubMed  Google Scholar 

  37. Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Knaster Wasse, J., & McCauley, E. (2019). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child and Adolescent Psychology,48, 168–179. https://doi.org/10.1080/15374416.2017.1280808.

    Article  Google Scholar 

  38. Mangunkusomo, R. T., Duisterhout, J. S., De Graaff, N., Maarsingh, E. J., de Koning, H. J., & Raat, H. (2006). Internet versus paper mode of health and health behavior questionnaires in elementary schools: Asthma and fruit as examples. Journal of School Health,76, 80–86.

    Article  Google Scholar 

  39. Mangunkusomo, R. T., Moorman, P. W., Berg-de Ruiter, A. E., & Van Der Lei, J. (2005). Internet-administered adolescent questionnaires compared with a paper version in a randomized study. Journal of Adolescent Health,36(1), 71–76. https://doi.org/10.1016/j.jadohealth.2004.02.020.

    Article  Google Scholar 

  40. Menachemi, N., & Collum, T. H. (2011). Benefits and drawbacks of electronic record systems. Risk Management and Healthcare Policy,4, 47–55. https://doi.org/10.2147/RMHP.S12985.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Merikangas, K. R., He, J., Burstein, M., Swendsen, J., Avenevoli, S., Case, B., …, Olfson, M. (2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the national comorbidity survey-adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 50(1), 32–45. https://doi.org/10.1016/j.jaac.2010.10.006.

    Article  Google Scholar 

  42. Muhr, T. (2004). User’s manual for ALTAS.ti 5.0. Berlin: ATLAS.ti. Scientific Software Development GmbH.

  43. Nadeem, E., Cappella, E., Holland, S., Coccaro, C., & Crisonino, G. (2016). Development and piloting of a classroom-focused measurement feedback system. Administration and Policy in Mental Health,43, 379–393.

    Article  Google Scholar 

  44. Norman, D. A., & Draper, S. W. (Eds.). (1986). User-centered system design: New perspectives on human–computer interaction. Hillsdale, NJ: Lawrence Earlbaum Associates.

    Google Scholar 

  45. Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed methods designs in implementation research. Administration and Policy in Mental Health,38, 44–53. https://doi.org/10.1007/s10488-010-0314-z.

    Article  PubMed  Google Scholar 

  46. Poston, J. M., & Hanson, W. E. (2010). Meta-analysis of psychological assessment as a therapeutic intervention. Psychological Assessment,22, 203–212. https://doi.org/10.1037/a0018679.

    Article  PubMed  Google Scholar 

  47. Powell, B. J., Beidas, R. S., Lewis, C. C., Aarons, G. A., McMillen, J. C., Proctor, E. K., et al. (2017). Methods to improve the selection and tailoring of implementation strategies. The Journal of Behavioral Health Services and Research,44(2), 177–194.

    Article  Google Scholar 

  48. Rakovshik, S. G., & McManus, F. (2010). Establishing evidence-based training in cognitive behavioral therapy: A review of current empirical findings and theoretical guidance. Clinical Psychology Review,30(5), 496–516. https://doi.org/10.1016/j.cpr.2010.03.004.

    Article  PubMed  Google Scholar 

  49. Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice,22, 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Seidman, E., Chorpita, B. F., Reay, W. E., Stelk, W., Garland, A. F., Jutash, K., …, Ringeisen, H. (2010). A framework for measurement feedback to improve decision-making in mental health. Administration Policy in Mental Health,37, 128–131. https://doi.org/10.1007/s10488-009-0260-9.

    Article  Google Scholar 

  51. Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H., et al. (1983). A children’s global assessment scale (CGAS). Archives of General Psychiatry,40(1), 1223–1231. https://doi.org/10.1001/archpsyc.1983.017901000074010.

    Article  Google Scholar 

  52. Tharinger, D. J., Finn, S. E., Gentry, L., Hamilton, A., Fowler, J., Matson, M., …, Walkowiak, J. (2009). Therapeutic assessment with children: A pilot study of treatment acceptability and outcome. Journal of Personality Assessment,91, 238–244. https://doi.org/10.1080/00223890902794275.

    Article  Google Scholar 

  53. Unützer, J., Chan, Y. F., Hafer, E., Knaster, J., Shields, A., Powers, D., et al. (2012). Quality improvement with pay-for performance incentives in integrated behavioral health care. American Journal of Public Health,102(6), e41–e45.

    Article  Google Scholar 

  54. Unützer, J., Katon, W., Callahan, C. M., Williams, J. W., Jr., Hunkeler, E., Harpole, L., et al. (2002). Collaborative care management of late-life depression in the primary care setting: A randomized controlled trial. JAMA,288(22), 2836–2845.

    Article  Google Scholar 

  55. Vallejo, M. A., Mananes, G., Comeche, M. I., & Diaz, M. I. (2008). Comparison between administration via internet and paper-and-pencil administration of two clinical instruments: SCL-90-R and GHQ-28. Journal of Behavioral Therapy and Experimental Psychiatry,39(3), 201–208. https://doi.org/10.1016/j.jbtep.2007.04.001.

    Article  Google Scholar 

  56. Van de Looij-Jansen, P. M., & Wilde, E. J. D. (2008). Comparison of web-based versus paper-and-pencil self-administered questionnaire: Effects on health indicators in Dutch adolescents. Health Services Research,43(5), 1708–1721. https://doi.org/10.1111/j.1475-6773.2008.00860.

    Article  PubMed Central  Google Scholar 

  57. Vispoel, W. P., Boo, J., & Bieiler, T. (2001). Computerized and paper-and-pencil versions of the Rosenberg Self-Esteem Scale: A comparison of psychometric features and respondent preferences. Educational and Psychological Measurement,61(3), 461–474. https://doi.org/10.1177/00131640121971329.

    Article  Google Scholar 

  58. Williams, J. W., Katon, W., Lin, E. H., Nöel, P. H., Worchel, J., Cornell, J., et al. (2004). The effectiveness of depression care management on diabetes-related outcomes in older patients. Annals of Internal Medicine,140(12), 1015–1024.

    Article  Google Scholar 

Download references

Acknowledgements

This publication was made possible in part by funding from Grant K08MH095939, awarded to the last author (Aaron R. Lyon) from the National Institute of Mental Health (NIMH). The authors would also like to thank the participating students, Seattle Children’s Hospital, and Public Health of Seattle and King County for their support of this project. This paper was not previously presented at a meeting.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ashley M. Mayworm.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the Ethical Standards of the Institutional Review Board of the University of Washington (43941 UW IRB) and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mayworm, A.M., Kelly, B.M., Duong, M.T. et al. Middle and High School Student Perspectives on Digitally-Delivered Mental Health Assessments and Measurement Feedback Systems. Adm Policy Ment Health 47, 531–544 (2020). https://doi.org/10.1007/s10488-020-01010-9

Download citation

Keywords

  • Measurement-based care
  • Evidence-based assessment
  • Measurement feedback system
  • Digitally-delivered assessment
  • School mental health