Abstract
This article uses the Comprehensive Mixed-Methods Participatory Evaluation (CMMPE; Nastasi and Hitchcock Transforming school mental health services: Population-based approaches to promoting the competency and wellness of children, Thousand Oaks, CA: Corwin Press with National Association of School Psychologists 2008; Nastasi et al. School-based mental health services: creating comprehensive and culturally specific programs. Washington, DC: American Psychological Association 2004) model as a framework for addressing the multiplicity of evaluation decisions and complex nature of questions related to program success in multilevel interventions. CMMPE defines program success in terms of acceptability, integrity, social or cultural validity, outcomes (impact), sustainability and institutionalization, thus broadening the traditional notions of program outcomes. The authors use CMMPE and an example of a community-based multilevel sexual risk prevention program with multiple outcomes to discuss challenges of evaluating multilevel interventions. The sexual risk program exemplifies what Schensul and Tricket (this issue) characterize as multilevel intervention–multilevel evaluation (M–M), with both intervention and evaluation at community, health practitioner, and patient levels. The illustration provides the context for considering several challenges related to M–M designs: feasibility of randomized controlled trials within community-based multilevel intervention; acceptability and social or cultural validity of evaluation procedures; implementer, recipient, and contextual variations in program success; interactions among levels of the intervention; unanticipated changes or conditions; multiple indicators of program success; engaging multiple stakeholders in a participatory process; and evaluating sustainability and institutionalization. The complexity of multilevel intervention and evaluation designs challenges traditional notions of evaluation research and experimental designs. Overcoming these challenges is critical to effective translation of research to practice in psychology and related disciplines.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Note that providers were the unit of assignment for the purposes of the quasi-experimental portion of the design.
References
Bernstein, R. (2000). A demonstration of the acceptability of a mental health project through a participatory culture-specific model of consultation in the country of Sri Lanka. Unpublished dissertation. Department of Educational and Counseling Psychology, University at Albany, State University of New York.
Brantlinger, E., Jiminez, R., Klingner, J., Pugach, M., & Richardson, V. (2005). Qualitative studies in special education. Exceptional Children, 71(2), 195–207.
CPRE (1995) Policy Brief: Helping Teachers Teach Well: Transforming Professional Development. Available on-line: http://www.ed.gov/pubs/CPRE/t61/t61c.html. Retrieved September, 2007.
Fetterman, D., & Wandersman, A. (2005). Empowerment evaluation principles in practice. New York: Guilford.
Flay, R. B., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6(3), 151–175.
Nastasi, B. K., & Hitchcock, J. H. (2008). Evaluating quality and effectiveness of population-based services. In B. J. Doll & J. A. Cummings (Eds.), Transforming school mental health services: Population-based approaches to promoting the competency and wellness of children (pp. 245–276). Thousand Oaks, CA: Corwin Press with National Association of School Psychologists.
Nastasi, B. K., Hitchcock, J., Sarkar, S., Burkholder, G., Varjas, K., & Jayasena, A. (2007). Mixed methods in intervention research: Theory to adaptation. Journal of Mixed Methods Research, 1(2), 164–182.
Nastasi, B. K., Moore, R. B., & Varjas, K. M. (2004). School-based mental health services: Creating comprehensive and culturally specific programs. Washington, DC: American Psychological Association.
Nastasi, B. K., & Schensul, S. L. (2005). Contributions of qualitative research to the validity of intervention research. Special issue of Journal of School Psychology, 43(3), 177–195.
Raudenbush, S. W., & Bryk, A. S. (2003). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc.
Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. Annals of statistics, 6, 34–58.
Schensul, S. L., Nastasi, B. K., & Verma, R.K. (2006). Community-based research in India: A case example of international and interdisciplinary collaboration. American Journal of Community Psychology, pp 1-17, [online version] DOI 10.1007/s10464-006-9066-z, URL http://dx.doi.org/10.1007/s10464-006-9066-z.
Schensul, S. L., Verma, R. K., & Nastasi, B. K. (2004). Responding to men’s sexual concerns: Research and intervention in slum communities in Mumbai, India. International Journal of Men’s Health, 3, 197–220.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.
Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Erlbaum.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 3–50). Thousand Oaks, CA: Sage.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Nastasi, B.K., Hitchcock, J. Challenges of Evaluating Multilevel Interventions. Am J Community Psychol 43, 360–376 (2009). https://doi.org/10.1007/s10464-009-9239-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10464-009-9239-7