Design Elements in Implementation Research: A Structured Review of Child Welfare and Child Mental Health Studies

  • John Landsverk
  • C. Hendricks Brown
  • Jennifer Rolls Reutz
  • Lawrence Palinkas
  • Sarah McCue Horwitz


Implementation science is an emerging field of research with considerable penetration in physical medicine and less in the fields of mental health and social services. There remains a lack of consensus on methodological approaches to the study of implementation processes and tests of implementation strategies. This paper addresses the need for methods development through a structured review that describes design elements in nine studies testing implementation strategies for evidence-based interventions addressing mental health problems of children in child welfare and child mental health settings. Randomized trial designs were dominant with considerable use of mixed method designs in the nine studies published since 2005. The findings are discussed in reference to the limitations of randomized designs in implementation science and the potential for use of alternative designs.


Child welfare Child mental health Implementation research Research design 



Support for this manuscript was provided by the following grants: P30 MH074678, DA019984, MH040859, DA024370, P30 MH068579, and MH080916.


  1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (in press). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research.Google Scholar
  2. Atkins, M. S., Frazier, S. L., Leathers, S. J., Graczyk, P. A., Talbott, E., Jakobsons, L., et al. (2008). Teacher key opinion leaders and mental health consultation in low-income urban schools. Journal of Consulting and Clinical Psychology, 76(5), 905–908.CrossRefPubMedGoogle Scholar
  3. Barrett-Connor, E., Grady, D., & Stefanick, M. (2004). The rise and fall of menopausal hormone therapy. Annual Review of Public Health, 26, 115–140.CrossRefGoogle Scholar
  4. Barwick, M. A., Peters, J., & Boydell, K. (2009). Getting to uptake: Do communities of practice support the implementation of evidence-based practice? Journal of the Canadian Academy of Child and Adolescent Psychiatry, 18(1), 16–29.PubMedGoogle Scholar
  5. Bert, S. C., Farris, J. R., & Borkowski, J. G. (2008). Parent training: Implementation strategies for adventures in parenting. Journal of Primary Prevention, 29(3), 243–261.CrossRefPubMedGoogle Scholar
  6. Berwick, D. (2008). The science of improvement. JAMA, 299(10), 1182–1184.CrossRefPubMedGoogle Scholar
  7. Brown, C. H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiologic paradigm. American Journal of Community Psychology, 27, 677–714.CrossRefGoogle Scholar
  8. Brown, C., Ten Have, T., Jo, B., Dagne, G., Wyman, P., Muthén, B., et al. (2009). Adaptive designs for randomized trials in public health. Annual Review of Public Health, 30, 1–25.CrossRefPubMedGoogle Scholar
  9. Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., et al. (2008). Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug and Alcohol Dependence, S95, S74–S104.CrossRefGoogle Scholar
  10. Brown, C. H., Wyman, P. A., Guo, J., & Peña, J. (2006). Dynamic wait-listed designs for randomized trials: New designs for prevention of youth suicide. Clinical Trials, 3, 259–271.CrossRefPubMedGoogle Scholar
  11. Cable, G. (2001). Enhancing causal interpretations of quality improvement interventions. Quality in Health Care, 10(3), 179–186.CrossRefPubMedGoogle Scholar
  12. Chamberlain, P., Brown, C. H., Saldana, L., Reid, J., Wang, W., Marsenich, L., et al. (2008). Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy in Mental Health, 35(4), 250–260.CrossRefPubMedGoogle Scholar
  13. Collins, L., Murphy, S., Nair, V., & Strecher, V. (2005). A strategy for optimizing and evaluating behavioral interventions. Annals of Behavioral Medicine, 30(1), 65–73.CrossRefPubMedGoogle Scholar
  14. Concato, J., Shah, N., & Horwitz, R. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. The New England Journal of Medicine, 342(25), 1887–1892.CrossRefPubMedGoogle Scholar
  15. Cook, T., Shadish, W., & Wong, V. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724–750.CrossRefGoogle Scholar
  16. Culliton, B. (2006). Extracting knowledge from science: A conversation with Elias Zerhouni. Health Affairs, 25(3), w94–w103.CrossRefGoogle Scholar
  17. Duan, N., Braslow, J., Weisz, J., & Wells, K. (2001). Fidelity, adherence, and robustness of interventions. Psychiatric Services, 52(4), 413.CrossRefPubMedGoogle Scholar
  18. Duan, N., Wells, K., Braslow, J., & Weisz, J. (2002). Randomized encouragement trial: A paradigm for public health intervention evaluation. Unpublished Manuscript.Google Scholar
  19. Flay, B. (1986). Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Preventive Medicine, 15(5), 451–474.CrossRefPubMedGoogle Scholar
  20. Glasgow, R., Magid, D., Beck, A., Ritzwoller, D., & Estabrooks, P. (2005). Practical clinical trials for translating research to practice: Design and measurement recommendations. Medical Care, 43(6), 551–557.CrossRefPubMedGoogle Scholar
  21. Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research, 7(4), 243–259.CrossRefPubMedGoogle Scholar
  22. Green, P., & Krieger, A. (1993). Conjoint analysis with product-positioning applications. Handbooks in Operations Research and Management Science, 5, 467–515.CrossRefGoogle Scholar
  23. Green, P., Krieger, A., & Wind, Y. (2001). Thirty years of conjoint analysis: Reflections and prospects. Interfaces, 31(3), 56–73.CrossRefGoogle Scholar
  24. Herschell, A. D., McNeil, C. B., Urquiza, A. J., McGrath, J. M., Zebell, N. M., Timmer, S. G., et al. (2009). Evaluation of a treatment manual and workshops for disseminating parent–child interaction therapy. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 63–81.CrossRefPubMedGoogle Scholar
  25. Lavori, P., Dawson, R., & Rush, A. (2000). Flexible treatment strategies in chronic disease: Clinical and research implications. Biological Psychiatry, 48(6), 605–614.CrossRefPubMedGoogle Scholar
  26. Lavori, P., Rush, A., Wisniewski, S., Alpert, J., Fava, M., Kupfer, D., et al. (2001). Strengthening clinical effectiveness trials: Equipoise-stratified randomization. Biological Psychiatry, 50(10), 792–801.CrossRefPubMedGoogle Scholar
  27. Lochman, J. E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the Coping Power Program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 77(3), 397–409.CrossRefPubMedGoogle Scholar
  28. Murphy, S., Lynch, K., Oslin, D., McKay, J., & Ten Have, T. (2007). Developing adaptive treatment strategies in substance abuse research. Drug and Alcohol Dependence, 88, S24–S30.CrossRefPubMedGoogle Scholar
  29. Murphy, S., Oslin, D., & Ten-Have, T. (2008, June). Convergence: Five committees, one dialogue. Paper presented at the NIMH conference, Crystal City, VA.Google Scholar
  30. National Research Council and Institute of Medicine (Ed.). (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: National Academy Press.Google Scholar
  31. Palinkas, L. A., Aarons, G. A., Chorpita, B. F., Hoagwood, K., Landsverk, J., & Weisz, J. R. (2009). Cultural exchange and the implementation of evidence-based practices: Two case studies. Research on Social Work Practice, 19(5), 602–612.CrossRefGoogle Scholar
  32. Palinkas, L. A., Schoenwald, S. K., Hoagwood, K., Landsverk, J., Chorpita, B. F., & Weisz, J. R. (2008). An ethnographic study of implementation of evidence-based treatments in child mental health: First steps. Psychiatric Services, 59(7), 738–746.CrossRefPubMedGoogle Scholar
  33. Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage Publications Ltd.Google Scholar
  34. Poduska, J., Kellam, S. G., Brown, C. H., Ford, C., Keegan, N., Windham, A., et al. (2009). A group randomized control trial of a classroom-based intervention aimed at preventing early risk factors for drug abuse: Integrating effectiveness and implementation research. Implementation Science, 4(56), 1–11.Google Scholar
  35. Proctor, E., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24–34.CrossRefPubMedGoogle Scholar
  36. Schwartz, D., & Lellouch, J. (2009). Explanatory and pragmatic attitudes in therapeutical trials. Journal of Clinical Epidemiology, 62(5), 499–505.CrossRefGoogle Scholar
  37. Shortell, S. M. (2004). Increasing value: A research agenda for addressing the managerial and organizational challenges facing health care delivery in the United States. Medical Care Research and Review, 61, 12S–30S.CrossRefPubMedGoogle Scholar
  38. Strecher, V., McClure, J., Alexander, G., Chakraborty, B., Nair, V., Konkel, J., et al. (2008). Web-based smoking cessation components and tailoring depth: Results of a randomized trial. American Journal of Preventive Medicine, 34(5), 373–381.CrossRefPubMedGoogle Scholar
  39. Ten Have, T., Coyne, J., Salzer, M., & Katz, I. (2003a). Research to improve the quality of care for depression: Alternatives to the simple randomized clinical trial* 1. General Hospital Psychiatry, 25(2), 115–123.CrossRefGoogle Scholar
  40. Ten Have, T., Joffe, M., & Cary, M. (2003b). Causal logistic models for non-compliance under randomized treatment with univariate binary response. Statistics in Medicine, 22(8), 1255–1283.CrossRefPubMedGoogle Scholar
  41. Tunis, S., Stryer, D., & Clancy, C. (2003). Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. JAMA, 290(12), 1624–1632.CrossRefPubMedGoogle Scholar
  42. Walshe, K. (2007). Understanding what works—and why—in quality improvement: The need for theory-driven evaluation. International Journal for Quality in Health Care 19, 57–59.CrossRefPubMedGoogle Scholar
  43. West, S., Duan, N., Pequegnat, W., Gaist, P., Des Jarlais, D., Holtgrave, D., et al. (2008). Alternatives to the randomized controlled trial. American Journal of Public Health, 98(8), 1359–1366.CrossRefPubMedGoogle Scholar
  44. Westfall, J., Mold, J., & Fagnan, L. (2007). Practice-based research—“Blue Highways” on the NIH roadmap. JAMA, 297(4), 403–406.CrossRefPubMedGoogle Scholar
  45. Zerhouni, E. (2003). The NIH roadmap. Science (Washington), 302(5642), 63–72.CrossRefGoogle Scholar
  46. Zerhouni, E. (2005). Translational and clinical science—time for a new vision. New England Journal of Medicine, 353(15), 1621–1623.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • John Landsverk
    • 1
  • C. Hendricks Brown
    • 2
  • Jennifer Rolls Reutz
    • 1
  • Lawrence Palinkas
    • 3
  • Sarah McCue Horwitz
    • 4
  1. 1.Child and Adolescent Services Research Center (CASRC)Rady Children’s Hospital San DiegoSan DiegoUSA
  2. 2.Department of Epidemiology and Public Health, School of MedicineUniversity of MiamiMiamiUSA
  3. 3.School of Social WorkUniversity of Southern California and Child and Adolescent ServicesLos AngelesUSA
  4. 4.Department of Pediatrics and the Centers for Health Policy and Primary Care and Outcomes ResearchStanford UniversityStanfordUSA

Personalised recommendations