Abstract
Intended for researchers and clinical leaders, this article suggests that embedded program evaluation is a good fit with the desired features of practice-oriented research. The systematic nature of evaluation that is built into the operational workflow of a practice setting may increase the diversity of methods available to explore processes and outcomes of interest. We propose a novel conceptual framework that uses a human-centered systems lens to foster such embedded evaluation in clinical routine. This approach emphasizes the evaluator-practitioner partnership to build confidence in the bi-directional learning of practice-based evidence with evidence-based practice. The iterative cycles inherent to design thinking are aimed at developing better evaluation questions. The attention to structure and context inherent to systems thinking is intended to support meaningful perspectives in the naturally complex world of health care. Importantly, the combined human-centered systems lens can create greater awareness of the influence of individual and systemic biases that exist in any endeavor or institution that involves people. Recommended tools and strategies include systems mapping, program theory development, and visual facilitation using a logic model to represent the complexity of mental health treatment for communication, shared understanding, and connection to the broader evidence base. To illustrate elements of the proposed conceptual framework, two case examples are drawn from routine outcome monitoring (ROM) and progress feedback. We conclude with questions for future collaboration and research that may strengthen the partnership of evaluators and practitioners as a community of learners in service of local and system-level improvement.
This is a preview of subscription content, log in via an institution to check access.
Reprinted with permission
Similar content being viewed by others
Notes
To use bias-free language (Robinson, 2021), we have replaced the word “stakeholder” with other words aimed to represent the individuals and groups intended, such as program collaborators, program actors, participants, etc.
References
Adams, C., & Nash, J. B. (2016). Exploring design thinking practices in evaluation. Journal of MultiDisciplinary Evaluation, 12(26), 12–17. http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/434.
Adkins, J. A., Douglas, S., Voorhies, P., & Bickman, L. (in press). Program evaluation: The bottom line in organizational health. In L. E. Tetrick, G. G. Fischer, M. T. Ford, & J. C. Quick (Eds.), Handbook of Occupational Health psychology (3rd ed., p. 566). American Psychological Association. https://doi.org/10.1037/0000331-028.
Andrews, G., & Page, A. C. (2005). Outcome measurement, outcome management and monitoring. Australian & New Zealand Journal of Psychiatry, 39(8), 649–651. https://doi.org/10.1080/j.1440-1614.2005.01648.x.
Barkham, M., De Jong, K., Delgadillo, J., & Lutz, W. (2023). Routine outcome monitoring (ROM) and feedback: Research review and recommendations. Psychotherapy Research, 0(0), 1–15. https://doi.org/10.1080/10503307.2023.2181114.
Barry, D., Kimble, L. E., Nambiar, B., Parry, G., Jha, A., Chattu, V. K., Massoud, M. R., & Goldmann, D. (2018). A framework for learning about improvement: Embedded implementation and evaluation design to optimize learning. International Journal for Quality in Health Care, 30(1), 10–14. https://doi.org/10.1093/intqhc/mzy008.
Berntsen, E., Starling, J., Durheim, E., Hainsworth, C., De Kloet, L., Chapman, L., & Hancock, K. (2011). Temporal trends in self harm and aggression on a paediatric mental health ward. Australasian Psychiatry, 19(1), 64–69. https://doi.org/10.3109/10398562.2010.526212.
Bickman, L. (1987). Using program theory in evaluation. New Directions for Program Evaluation, 1987(33), 5–18. https://doi.org/10.1002/ev.1443.
Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child & Adolescent Psychiatry, 47(10), 1114–1119. https://doi.org/10.1097/CHI.0b013e3181825af8. http://dx.doi.org.proxy.library.vanderbilt.edu/.
Bickman, L. (2013). Facing reality and jumping the chasm. Administration and Policy in Mental Health and Mental Health Services Research, 40(1), 1–5. https://doi.org/10.1007/s10488-012-0460-6. http://dx.doi.org.proxy.library.vanderbilt.edu/.
Bickman, L., & Rog, D. (2009). The SAGE handbook of applied social research methods (2nd ed.). https://doi.org/10.4135/9781483348858.
Bickman, L., Rosof-Williams, J., Salzer, M. S., Summerfelt, W. T., Noser, K., Wilson, S. J., & Karver, M. S. (2000). What information do clinicians value for monitoring adolescent client progress and outcomes? Professional Psychology: Research and Practice, 31(1), 70–74. https://doi.org/10.1037/0735-7028.31.1.70.
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. https://doi.org/10.1176/appi.ps.002052011.
Bickman, L., Douglas, S., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 410–425. https://doi.org/10.1007/s10488-015-0647-8.
Binder, P. E., Holgersen, H., & Nielsen, G. H. (2010). What is a good outcome in psychotherapy? A qualitative exploration of former patients’ point of view. Psychotherapy Research, 20(3), 285–294. https://doi.org/10.1080/10503300903376338.
Boswell, J. F. (2020). Monitoring processes and outcomes in routine clinical practice: A promising approach to plugging the holes of the practice-based evidence colander. Psychotherapy Research, 30(7), 829–842. https://doi.org/10.1080/10503307.2019.1686192.
Both, T. (2018, March 9). Human-centered, systems-minded design. Stanford Social Innovation Review. https://ssir.org/articles/entry/human_centered_systems_minded_design.
Bourgeois, I., & Cousins, J. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299–319. https://doi.org/10.1177/1098214013477235.
Brooks Holliday, S., Hepner, K. A., Farmer, C. M., Mahmud, A., Kimerling, R., Smith, B. N., & Rosen, C. (2021). Discussing measurement-based care with patients: An analysis of clinician-patient dyads. Psychotherapy Research, 31(2), 211–223. https://doi.org/10.1080/10503307.2020.1776413.
Brown, J., Scholle, S. H., & Azur, M. (2014). Strategies for measuring the quality of psychotherapy: A white paper to inform measure development and implementation. Office of the Assistant Secretary for planning and evaluation. U.S. Department of Health and Human Service. http://aspe.hhs.gov/daltcp/reports/2014/QualPsy.cfm.
Bryson, J. M., & Patton, M. Q. (2015). Analyzing and engaging stakeholders. Handbook of practical program evaluation (4th ed., pp. 36–61). Jossey-Bass.
Buchanan, R. (2019). Systems thinking and design thinking: The search for principles in the world we are making. She Ji: The Journal of Design Economics and Innovation, 5(2), 85–104. https://doi.org/10.1016/j.sheji.2019.04.001.
Camacho, K. S., Page, A. C., & Hooke, G. R. (2021). An exploration of the relationships between patient application of CBT skills and therapeutic outcomes during a two-week CBT treatment. Psychotherapy Research, 31(6), 778–788. https://doi.org/10.1080/10503307.2020.1845414.
Castonguay, L. G., Barkham, M., Youn, S. J., & Page, A. C. (2021). Practice-based evidence – Findings from routine clinical settings. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and behavior change (50th Anniversary Edition) (7th ed., pp. 191–222). John Wiley and Sons.
Centers for Disease Control and Prevention (1999). Framework for program evaluation in public health (MMWR 48(RR11)). https://www.cdc.gov/eval/framework/index.htm.
Chen, H. T. (1990). Theory-driven evaluations. Sage.
Chen, H. T. (2014). Practical program evaluation. Sage.
Chernoff, N. N. (2003). What is translational research? APS Observer, 16. https://www.psychologicalscience.org/observer/what-is-translational-research.
Christie, C. A., & Alkin, M. C. (2013). An evaluation theory tree. In M. C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed., pp. 11–57). Sage.
Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397–418. https://doi.org/10.3102/01623737014004397.
Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. New Directions for Evaluation, 141, 7–23. https://doi.org/10.1002/ev.20076.
Davis, S., Hooke, G. R., & Page, A. C. (2006). Identifying and targeting predictors of drop-out from group cognitive behaviour therapy. Australian Journal of Psychology, 58(1), 48–56. https://doi.org/10.1080/00049530500504096.
De Jong, K., & De Goede, M. (2015). Why do some therapists not deal with outcome monitoring feedback? A feasibility study on the effect of regulatory focus and person-organization fit on attitude and outcome. Psychotherapy Research, 25(6), 661–668. https://doi.org/10.1080/10503307.2015.1076198.
De Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22(4), 464–474. https://doi.org/10.1080/10503307.2012.673023.
De Jong, K., Conijn, J. M., Gallagher, R. A. V., Reshetnikova, A. S., Heij, M., & Lutz, M. C. (2021). Using progress feedback to improve outcomes and reduce drop-out, treatment duration, and deterioration: A multilevel meta-analysis. Clinical Psychology Review, 85, 102002. https://doi.org/10.1016/j.cpr.2021.102002.
de Savigny, D., Blanchet, K., & Adam, T. (2017). Applied systems thinking for health systems research: A methodological handbook. McGraw-Hill Education. https://researchonline.lshtm.ac.uk/id/eprint/4646083.
Dean, A. J., Gibbon, P., McDermott, B. M., Davidson, T., & Scott, J. (2010). Exposure to aggression and the impact on staff in a child and adolescent inpatient unit. Archives of Psychiatric Nursing, 24(1), 15–26. https://doi.org/10.1016/j.apnu.2009.01.002.
Deming, W. E. (2018). The new economics: For industry, government, and education (3rd ed.). MIT Press.
Design Council. (n.d.). History of the double diamond: A universally accepted depiction of the design process. https://www.designcouncil.org.uk/our-resources/the-double-diamond/history-of-the-double-diamond/.
Douglas, S., Button, S., & Casey, S. E. (2016). Implementing for sustainability: Promoting use of a measurement feedback system for innovation and quality improvement. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 286–291. https://doi.org/10.1007/s10488-014-0607-8.
Douglas, S., Bovendeerd, B., van Sonsbeek, M., Manns, M., Milling, X. P., Tyler, K., Bala, N., Satterthwaite, T., Hovland, R. T., Amble, I., Atzil-Slonim, D., Barkham, M., De Jong, K., Kendrick, T., Nordberg, S. S., Lutz, W., Rubel, J. A., Skjulsvik, T., & Moltu, C. (in press). A clinical leadership lens on implementing progress feedback in three countries: Development of a multidimensional qualitative coding scheme. Administration and Policy in Mental Health and Mental Health Services Research.
Douglas Kelley, S., Nixon, C. T., & Bickman, L. (2000). Evaluating mental health services for children and adolescents. In D. Drotar (Ed.), Handbook of Research in Pediatric and Clinical Child Psychology: Practical Strategies and Methods (pp. 463–489). Springer US. https://doi.org/10.1007/978-1-4615-4165-3_22.
Douglas Kelley, S., Bickman, L., & Norwood, E. (2010). Evidence-based treatments and common factors in youth psychotherapy. In B. L. Duncan, S. D. Miller, B. E. Wampold, & M. A. Hubble (Eds.), The Heart and Soul of Change: Delivering What Works in Therapy (2nd ed., pp. 325–355). American Psychological Association. http://search.proquest.com.proxy.library.vanderbilt.edu/psycinfo/docview/622057072/DD7D566E349645B9PQ/1.
Dubois, N., Lloyd, S., Houle, J., Mercier, C., Brousselle, A., & Rey, L. (2011). Discussion: Practice-based evaluation as a response to address intervention complexity. The Canadian Journal of Program Evaluation = La Revue Canadienne D’evaluation de Programme, 26(3), 105–113. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5036968/.
Duncan, B. L., Miller, S. D., Wampold, B. E., & Hubble, M. A. (Eds.). (2010). The heart and soul of change: Delivering what works in therapy (2nd ed.). American Psychological Association. http://search.proquest.com.proxy.library.vanderbilt.edu/psycinfo/docview/622057072/DD7D566E349645B9PQ/1.
Edmondson, A. C. (2019). The fearless organization: Creating psychological safety in the workplace for learning, innovation, and growth (1st edition). Wiley.
Finn, S. E., & Tonsager, M. E. (1997). Information-gathering and therapeutic models of assessment: Complementary paradigms. Psychological Assessment, 9(4), 374–385. https://doi.org/10.1037/1040-3590.9.4.374.
Forbes, C. (2022). Exploring barriers and solutions to encouraging evidence-into-use within an embedded evaluation approach: Reflections from the field. Review of Education, 10(2), e3351. https://doi.org/10.1002/rev3.3351.
Gates, E. F., Madres, J., Hall, J. N., Alvarez, K. B., & U.S. (2022). It takes an ecosystem: Socioecological factors influencing equity-oriented evaluation in New England, 2021. Evaluation and Program Planning, 92, 102068. https://doi.org/10.1016/j.evalprogplan.2022.102068.
Giancola, S. P. (2020). Program evaluation: Embedding evaluation into program design and development (1st Edition). SAGE Publications Inc.
Gill, S., Kuwahara, R., & Wilce, M. (2016). Through a culturally competent lens: Why the program evaluation standards matter. Health Promotion Practice, 17(1), 5–8. https://doi.org/10.1177/1524839915616364.
Gordon, L. T., Gordon, J. D., & Gardner, D. (1996). A study of assaultive/ aggressive behaviour on staff by patients/ clients in psychiatric facilities. Journal of Occupational Health and Safety Australia and New Zealand, 12(2), 169–177. https://doi.org/10.3316/ielapa.961009925.
Greenberg, L. S. (2019). Theory of functioning in emotion-focused therapy. In L. S. Greenberg & R. N. Goldman (Eds.), Clinical handbook of emotion-focused therapy (pp. 37–59). American Psychological Association. https://doi.org/10.1037/0000112-002.
Hall, M. E. (2018). Evaluation’s race problem in the United States: A call to action for the profession and the American Journal of evaluation. American Journal of Evaluation, 39(4), 569–583. https://doi.org/10.1177/1098214018792624.
Hall, J. N. (2020). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20–33. https://doi.org/10.1177/1098214019828485.
Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology, 61(2), 155–163. https://doi.org/10.1002/jclp.20108.
Hawkins, P., & Smith, N. (2013). Coaching, mentoring and organizational consultancy: Supervision, skills and development (2nd ed.). Open University Press.
Hooke, G. R., Savani, P., Stewart, B., Araujo, S., & Page, A. C. (2022). Illustrating routine outcomes monitoring at different points in a patient’s journey: Inpatient then daypatient treatment of a patient with depressive and borderline symptoms. Journal of Clinical Psychology, 78(10), 2041–2053. https://doi.org/10.1002/jclp.23426.
House, E. R. (2017). Evaluation and the framing of race. American Journal of Evaluation, 38(2), 167–189. https://doi.org/10.1177/1098214017694963.
Hovland, R. T., Ytrehus, S., Mellor-Clark, J., & Moltu, C. (2023). How patients and clinicians experience the utility of a personalized clinical feedback system in routine practice. Journal of Clinical Psychology, 79(3), 711–728. https://doi.org/10.1002/jclp.22992.
Institute of Medicine (US) Committee on Quality of Health Care in America. (2001). Crossing the quality chasm: A new health system for the 21st century. National Academies Press (US. http://www.ncbi.nlm.nih.gov/books/NBK222274/.
Jensen-Doss, A., Douglas, S., Phillips, D. A., Gencdur, O., Zalman, A., & Gomez, N. E. (2020). Measurement-based care as a practice improvement tool: Clinical and organizational applications in youth mental health. Evidence-Based Practice in Child and Adolescent Mental Health, 5(3), 233–250. https://doi.org/10.1080/23794925.2020.1784062.
Johnson, B. (2014). Polarity management: Identifying and managing unsolvable problems (4th ed.). HRD Press. 978–0874251760.
Jones, N. D., Azzam, T., Wanzer, D. L., Skousen, D., Knight, C., & Sabarre, N. (2020). Enhancing the effectiveness of logic models. American Journal of Evaluation, 41(3), 452–470. https://doi.org/10.1177/1098214018824417.
Karim, A., Cobos Munoz, D., Mäusezahl, D., & de Savigny, D. (2020). A systems approach to assessing complexity in health interventions: An effectiveness decay model for integrated community case management. Global Health Action, 13(1), 1794106–1794106. https://doi.org/10.1080/16549716.2020.1794106.
Kashyap, S., Hooke, G. R., & Page, A. C. (2015). Identifying risk of deliberate self-harm through longitudinal monitoring of psychological distress in an inpatient psychiatric population. Bmc Psychiatry, 15(1), 81. https://doi.org/10.1186/s12888-015-0464-3.
Kazdin, A. E. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. The American Psychologist, 63(3), 146–159. https://doi.org/10.1037/0003-066X.63.3.146.
Kipoulas, E., Berzengi, A., & Kyriakopoulos, M. (2021). Prevalence and clinical correlates of self-harm and suicidality during admission of children in a mental health inpatient unit. European Psychiatry, 64(1), e1. https://doi.org/10.1192/j.eurpsy.2020.108.
Kivlighan, D. M., Narvaez, R. C., & Carter, W. (2019). Different strokes for different folks: Examining additive and complimentary effects of the advisory working alliance and research team cohesion on trainees’ research outcomes. Journal of Counseling Psychology, 66(1), 104–113. https://doi.org/10.1037/cou0000314.
Kyron, M. J., Hooke, G. R., & Page, A. C. (2018). Daily assessment of interpersonal factors to predict suicidal ideation and non-suicidal self-injury in psychiatric inpatients. Journal of Consulting and Clinical Psychology, 86(6), 556–567. https://doi.org/10.1037/ccp0000305.
Kyron, M. J., Hooke, G. R., & Page, A. C. (2021). Prediction and network modelling of self-harm through daily self-report and history of self-injury. Psychological Medicine, 51(12), 1992–2002. https://doi.org/10.1017/S0033291720000744.
Kyron, M. J., Hooke, G. R., Bryan, C. J., & Page, A. C. (2022). Distress tolerance as a moderator of the dynamic associations between interpersonal needs and suicidal thoughts. Suicide and Life-Threatening Behavior, 52(1), 159–170. https://doi.org/10.1111/sltb.12814.
Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions. Journal of Clinical Psychology, 61(2), 165–174. https://doi.org/10.1002/jclp.20113.
LaVelle, J. M., & Galport, N. (2020). Using the 2018 AEA Evaluator Competencies for Evaluator Education and Professional Development. New Directions for Evaluation, 2020(168), 99–116. https://doi.org/10.1002/ev.20437.
Låver, J., McAleavey, A. A., Castonguay, L. G., & Moltu, C. (2023). Therapists’ and patients’ experiences of using patients’ quantitative self-report data in ongoing psychotherapy processes – a systematic review and meta-analysis of qualitative studies. Psychotherapy Research, 1-18. https://doi.org/10.1080/10503307.2023.2222896
Leeuw, F. L., & Donaldson, S. I. (2015). Theory in evaluation: Reducing confusion and encouraging debate. Evaluation, 21(4), 467–480. https://doi.org/10.1177/1356389015607712.
Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., Douglas, S., Simon, G., & Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324. https://doi.org/10.1001/jamapsychiatry.2018.3329.
Liedtka, J. (2020). Putting technology in its place: Design thinking’s social technology at work. California Management Review, 62(2), 53–83. https://doi.org/10.1177/0008125619897391.
Liedtka, J. (2018, September 1). Why design thinking works. Harvard Business Review. https://hbr.org/2018/09/why-design-thinking-works.
Lipsey, M. W. (1993). Theory as method: Small theories of treatments. New Directions for Program Evaluation, 1993(57), 5–38. https://doi.org/10.1002/ev.1637.
Littlewood, D., Kyle, S. D., Pratt, D., Peters, S., & Gooding, P. (2017). Examining the role of psychological factors in the relationship between sleep problems and suicide. Clinical Psychology Review, 54, 1–16. https://doi.org/10.1016/j.cpr.2017.03.009.
Littlewood, D. L., Kyle, S. D., Carter, L. A., Peters, S., Pratt, D., & Gooding, P. (2019). Short sleep duration and poor sleep quality predict next-day suicidal ideation: An ecological momentary assessment study. Psychological Medicine, 49(3), 403–411. https://doi.org/10.1017/S0033291718001009.
Lutz, W., Ehrlich, T., Rubel, J., Hallwachs, N., Röttger, M. A., Jorasz, C., Mocanu, S., Vocks, S., Schulte, D., & Tschitsaz-Stucki, A. (2013). The ups and downs of psychotherapy: Sudden gains and sudden losses identified with session reports. Psychotherapy Research: Journal of the Society for Psychotherapy Research, 23(1), 14–24. https://doi.org/10.1080/10503307.2012.693837.
Lutz, W., Schwartz, B., Hofmann, S. G., Fisher, A. J., Husen, K., & Rubel, J. A. (2018). Using network analysis for the prediction of treatment dropout in patients with mood and anxiety disorders: A methodological proof-of-concept study. Scientific Reports, 8(1), 7819. https://doi.org/10.1038/s41598-018-25953-0.
Lyon, A. R., Brewer, S. K., & Areán, P. A. (2020). Leveraging human-centered design to implement modern psychological science: Return on an early investment. American Psychologist, 75(8), 1067–1079. https://doi.org/10.1037/amp0000652.
Mark, M. M., & Henry, G. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10(1), 35–57.
McAleavey, A. A., Nordberg, S. S., & Moltu, C. (2021). Initial quantitative development of the norse feedback system: A novel clinical feedback system for routine mental healthcare. Quality of Life Research, 30(11), 3097–3115. https://doi.org/10.1007/s11136-021-02825-1.
Miller, S. D., Hubble, M. A., Chow, D., & Seidel, J. (2015). Beyond measures and monitoring: Realizing the potential of feedback-informed treatment. Psychotherapy, 52(4), 449–457. https://doi.org/10.1037/pst0000031.
Moltu, C., Stefansen, J., Nøtnes, J. C., Skjølberg, Å., & Veseth, M. (2017). What are good outcomes in public mental health settings? A qualitative exploration of clients’ and therapists’ experiences. International Journal of Mental Health Systems, 11(1), 12. https://doi.org/10.1186/s13033-017-0119-5.
Moltu, C., Veseth, M., Stefansen, J., Nøtnes, J. C., Skjølberg, Å., Binder, P. E., Castonguay, L. G., & Nordberg, S. S. (2018). This is what I need a clinical feedback system to do for me: A qualitative inquiry into therapists’ and patients’ perspectives. Psychotherapy Research, 28(2), 250–263. https://doi.org/10.1080/10503307.2016.1189619.
Moltu, C., McAleavey, A. A., Helleseth, M. M., Møller, G. H., & Nordberg, S. S. (2021). How therapists and patients need to develop a clinical feedback system after 18 months of use in a practice-research network: A qualitative study. International Journal of Mental Health Systems, 15(1), 43. https://doi.org/10.1186/s13033-021-00465-z.
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. Bmj, 350, h1258. https://doi.org/10.1136/bmj.h1258.
Morris, Z. S., Wooding, S., & Grant, J. (2011). The answer is 17 years, what is the question: Understanding time lags in translational research. Journal of the Royal Society of Medicine, 104(12), 510–520. https://doi.org/10.1258/jrsm.2011.110180.
Mütze, K., Witthöft, M., Lutz, W., & Bräscher, A. K. (2022). Matching research and practice: Prediction of individual patient progress and dropout risk for basic routine outcome monitoring. Psychotherapy Research, 32(3), 358–371. https://doi.org/10.1080/10503307.2021.1930244.
Newnham, E. A., Hooke, G. R., & Page, A. C. (2010). Progress monitoring and feedback in psychiatric care reduces depressive symptoms. Journal of Affective Disorders, 127(1–3), 139–146. https://doi.org/10.1016/j.jad.2010.05.003.
Nordberg, S. S., McAleavey, A. A., & Moltu, C. (2021). Continuous quality improvement in measure development: Lessons from building a novel clinical feedback system. Quality of Life Research, 30(11), 3085–3096. https://doi.org/10.1007/s11136-021-02768-7.
Norman, C. D. (2021). Supporting systems transformation through design-driven evaluation. New Directions for Evaluation, 2021(170), 149–158. https://doi.org/10.1002/ev.20464.
Olthof, M., Hasselman, F., Strunk, G., van Rooij, M., Aas, B., Helmich, M. A., Schiepek, G., & Lichtwarck-Aschoff, A. (2020). Critical fluctuations as an early-warning signal for sudden gains and losses in patients receiving psychotherapy for mood disorders. Clinical Psychological Science, 8(1), 25–35. https://doi.org/10.1177/2167702619865969.
Orchard, F., Pass, L., Chessell, C., Moody, A., Ellis, J., & Reynolds, S. (2020). Adapting brief CBT-I for depressed adolescents: A case illustration of the sleeping better program. Cognitive and Behavioral Practice, 27(3), 336–346. https://doi.org/10.1016/j.cbpra.2019.07.010.
Patton, M. Q. (2013). The roots of utilization-focused evaluation. In M. C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed., pp. 293–303). Sage.
Patton, M. Q. (2016). What is essential in developmental evaluation? On integrity, fidelity, adultery, abstinence, impotence, long-term commitment, integrity, and sensitivity in implementing evaluation models. American Journal of Evaluation, 37(2), 250–265. https://doi.org/10.1177/1098214015626295.
Patton, M. Q. (2018). Facilitating evaluation: Principles in practice. Sage.
Riemer, M., & Bickman, L. (2011). Using program theory to link social psychology and program evaluation. In M. M. Mark, S. I. Donaldson, & B. Campbell (Eds.), Social psychology and evaluation (pp. 102–139). Guilford Press.
Riemer, M., Rosof-Williams, J., & Bickman, L. (2005). Theories related to changing clinician practice. Child and Adolescent Psychiatric Clinics of North America, 14(2), 241–254. https://doi.org/10.1016/j.chc.2004.05.002.
Riemer, M., Douglas Kelley, S., Casey, S., & Haynes, K. T. (2012). Developing effective research-practice partnerships for creating a culture of evidence-based decision making. Administration and Policy in Mental Health and Mental Health Services Research, 39(4), 248–257. https://doi.org/10.1007/s10488-011-0368-6. http://dx.doi.org.proxy.library.vanderbilt.edu/.
Robinson, S. (August 3, 2021). As an evaluator, do I use words (e.g., stakeholder) that can be harmful to others? By Goldie MacDonald & Anita McLees. AEA 365. https://aea365.org/blog/as-an-evaluator-do-i-use-words-e-g-stakeholder-that-can-be-harmful-to-others-by-goldie-macdonald-anita-mclees/.
Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2018). Evaluation: A systematic approach. SAGE Publications.
Rudd, M. D. (2006). Fluid vulnerability theory: A cognitive approach to understanding the process of acute and chronic suicide risk. In T. E. Ellis (Ed.), Cognition and suicide: Theory, research, and therapy (pp. 355–368). American Psychological Association. https://doi.org/10.1037/11377-016.
Sales, C. M. D., & Alves, P. C. G. (2012). Individualized patient-progress systems: Why we need to move towards a personalized evaluation of psychological treatments. Canadian Psychology / Psychologie Canadienne, 53(2), 115–121. https://doi.org/10.1037/a0028053.
Sales, C. M., Neves, I. T., Alves, P. G., & Ashworth, M. (2018). Capturing and missing the patient’s story through outcome measures: A thematic comparison of patient-generated items in PSYCHLOPS with CORE-OM and PHQ-9. Health Expectations, 21(3), 615–619. https://doi.org/10.1111/hex.12652.
Schuringa, E., Spreen, M., & Bogaerts, S. (2019). Inpatient violence in forensic psychiatry: Does change in dynamic risk indicators of the IFTE help predict short term inpatient violence? International Journal of Law and Psychiatry, 66, 101448. https://doi.org/10.1016/j.ijlp.2019.05.002.
Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice, 22(1), 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010.
Scott, V. C., Alia, K., Scaccia, J., Ramaswamy, R., Saha, S., Leviton, L., & Wandersman, A. (2020). Formative evaluation and complex health improvement initiatives: A learning system to improve theory, implementation, support, and evaluation. American Journal of Evaluation, 41(1), 89–106. https://doi.org/10.1177/1098214019868022.
Scriven, M. (2013). Conceptual revolutions in evaluation: Past, present, and future. In M. C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed., pp. 167–179). Sage.
Senge, P. M. (2006). The fifth discipline: The art & practice of the learning organization. Revised & Updated edition). Doubleday.
Seow, L. L. Y., Collins, K. R. L., Page, A. C., & Hooke, G. R. (2022). Outcomes of brief versions of Dialectical Behaviour Therapy for diagnostically heterogeneous groups in a routine care setting. Psychotherapy Research, 32(2), 179–194. https://doi.org/10.1080/10503307.2021.1933240.
Shadish, W. R. Jr. (1987). Program micro- and macrotheories: A guide for social change. New Directions for Program Evaluation, 1987(33), 93–109. https://doi.org/10.1002/ev.1448.
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation: Theories of practice. Sage Publications.
Sheikh, K., Gilson, L., Agyepong, I. A., Hanson, K., Ssengooba, F., & Bennett, S. (2011). Building the field of health policy and systems research: Framing the questions. PLoS Medicine, 8(8), https://doi.org/10.1371/journal.pmed.1001073.
Shrier, L. A., Burke, P. J., Jonestrask, C., & Katz-Wise, S. L. (2020). Applying Systems thinking and human-centered design to development of intervention implementation strategies: An example from adolescent health research. Journal of Public Health Research, 9(4), jphr.2020.1746.
Solstad, S. M., Castonguay, L. G., & Moltu, C. (2019). Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research, 29(2), 157–170. https://doi.org/10.1080/10503307.2017.1326645.
Solstad, S. M., Kleiven, G. S., & Moltu, C. (2021). Complexity and potentials of clinical feedback in mental health: An in-depth study of patient processes. Quality of Life Research, 30(11), 3117–3125. https://doi.org/10.1007/s11136-020-02550-1.
Stevahn, L., Berger, D. E., Tucker, S. A., & Rodell, A. (2020). Using the 2018 AEA Evaluator Competencies for effective program evaluation practice. New Directions for Evaluation, 2020(168), 75–97. https://doi.org/10.1002/ev.20434.
Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568–1580. https://doi.org/10.1016/j.respol.2013.05.008.
von Schwarz, T., Lundmark, U., R., & Hasson, H. (2016). The dynamic Integrated evaluation model (DIEM): Achieving sustainability in organizational intervention through a participatory evaluation approach. Stress and Health: Journal of the International Society for the Investigation of Stress, 32(4), 285–293. https://doi.org/10.1002/smi.2701.
Wang, S. B., Coppersmith, D. D. L., Kleiman, E. M., Bentley, K. H., Millner, A. J., Fortgang, R., Mair, P., Dempsey, W., Huffman, J. C., & Nock, M. K. (2021). A pilot study using frequent inpatient assessments of suicidal thinking to predict short-term postdischarge suicidal behavior. JAMA Network Open, 4(3), e210591. https://doi.org/10.1001/jamanetworkopen.2021.0591.
Warren, M. R., Hong, Rubin, C. L., & Sychitkokhong, P. U. (2009). Beyond the bake sale: A community-based relational approach to parent engagement in schools. Teachers College Record, 111(9), 2209–2254. https://doi.org/10.1177/016146810911100901.
Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Prentice-Hall.
Weiss, C. H. (1993). Where politics and evaluation research meet. Evaluation Practice, 14(1), 93–106. https://doi.org/10.1016/0886-1633(93)90046-R.
Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. New Approaches to evaluating community initiatives: Concepts, methods, and contexts (pp. 65–92). The Aspen Institute.
Weiss, C. H. (2013). Rooting for evaluation: Digging into beliefs. In M. C. Alkin (Ed.), Evaluation roots: A wider perspective of theorists’ views and influences (2nd ed., pp. 130–143). Sage.
Wucherpfennig, F., Rubel, J. A., Hollon, S. D., & Lutz, W. (2017). Sudden gains in routine care cognitive behavioral therapy for depression: A replication with extensions. Behaviour Research and Therapy, 89, 24–32. https://doi.org/10.1016/j.brat.2016.11.003.
Author information
Authors and Affiliations
Contributions
Susan Douglas, Andrew Page, and Christian Moltu were responsible for the initial conceptualization and outline of the article. All authors contributed to the writing and assisted with references. Susan Douglas organized and formatted the drafted versions, oversaw the revision process, and ensured completion of the final product. All authors reviewed the final draft of the document and have approved the final manuscript.
Corresponding author
Ethics declarations
Statements and Declarations
This is a conceptual paper that is submitted as part of a special issue and does not follow an existing reporting guideline framework. The paper is part of the special issue on: “Practice-Oriented Research.”
Conflict of Interest
Vanderbilt University and Susan Douglas receive compensation related to the Peabody Treatment Progress Battery; and Susan Douglas has a financial relationship with MIRAH, and both are Measurement-Based Care (MBC) tools. The author declares a potential conflict of interest. There is a management plan in place at Vanderbilt University to monitor that this potential conflict does not jeopardize the objectivity of Dr. Douglas’ research. Christian Moltu owns equity in a Norwegian private company that markets and sells a clinical feedback technology based on the Norse Feedback methodology. The author declares a potential conflict of interest. There is a management plan in place at Førde Hospital Trust to monitor that this potential conflict does not jeopardize the objectivity of Dr. Moltu’s research. Andrew Page is a research consultant to Perth Clinic but he does not receive funding from their use or otherwise of clinical feedback tools. The author declares a potential conflict of interest. There is a management plan in place at Perth Clinic to monitor that this potential conflict does not jeopardize the objectivity of Dr. Page’s research.
Compliance with Ethical Standards
This is a conceptual article that does not involve human subjects research, so it did not require formal ethics approval.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Douglas, S., Page, A.C., Moltu, C. et al. The Connections Matter: Bi-Directional Learning in Program Evaluation and Practice-Oriented Research. Adm Policy Ment Health 51, 318–335 (2024). https://doi.org/10.1007/s10488-023-01304-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-023-01304-8