Child & Youth Care Forum

, Volume 41, Issue 2, pp 119–136 | Cite as

Translational Research in South Africa: Evaluating Implementation Quality Using a Factorial Design

  • Linda L. CaldwellEmail author
  • Edward A. Smith
  • Linda M. Collins
  • John W. Graham
  • Mary Lai
  • Lisa Wegner
  • Tania Vergnani
  • Catherine Matthews
  • Joachim Jacobs
Original Paper



HealthWise South Africa: Life Skills for Adolescents (HW) is an evidence-based substance use and sexual risk prevention program that emphasizes the positive use of leisure time. Since 2000, this program has evolved from pilot testing through an efficacy trial involving over 7,000 youth in the Cape Town area. Beginning in 2011, through 2015, we are undertaking a new study that expands HW to all schools in the Metro South Education District.


This paper describes a research study designed in partnership with our South African collaborators that examines three factors hypothesized to affect the quality and fidelity of HW implementation: enhanced teacher training; teacher support, structure and supervision; and enhanced school environment.


Teachers and students from 56 schools in the Cape Town area will participate in this study. Teacher observations are the primary means of collecting data on factors affecting implementation quality. These factors address the practical concerns of teachers and schools related to likelihood of use and cost-effectiveness, and are hypothesized to be “active ingredients” related to high-quality program implementation in real-world settings. An innovative factorial experimental design was chosen to enable estimation of the individual effect of each of the three factors.


Because this paper describes the conceptualization of our study, results are not yet available.


The results of this study may have both substantive and methodological implications for advancing Type 2 translational research.


Factorial design Implementation quality Prevention Translational research 



HealthWise was supported by NIDA grant 1R01DA029084-01A1. Preparation of this article was supported by NIDA Center Grant P50 DA100075, NIDA 1R01DA029084-01A1, and F31 DA028155.

Conflict of interest

We declare that our research team has no conflict of interest with our association with the Metro South Education District (MSED). Our relationship with the district does not involve grants (other than the one secured by the U.S. National Institutes of Health/National Institute on Drug Abuse for this study which the MSED endorsed), employment, consultancy, shared ownership, or any close relationship that would influence or alter interests, financial or otherwise, that would be affected by the publication of this paper.

Ethics statement

This study has been granted human subjects approval by the IRB boards at both Penn State University and the University of the Western Cape. We have followed and will follow all requirements for the protection of human subjects as stipulated by the IRB application and approval.


  1. Adelman, H. S., & Taylor, L. (2003). On sustainability of project innovations as systemic change. Journal of Educational and Psychological Consultation, 14, 1–25.CrossRefGoogle Scholar
  2. August, G. J., Winters, K. C., Realmuto, G. M., Tarter, R., Perry, C., & Hektner, J. M. (2004). Moving evidence-based prevention programs from basic science to practice: “Bridging the efficacy-effectiveness interface”. Substance Use and Misuse, 39, 2017–2053.PubMedCrossRefGoogle Scholar
  3. Bales, E. A., & Soren, S. A. (2000). Managing clinical knowledge for health care improvement. Yearbook of Medical Informatics, (0943-4747), 65–70.Google Scholar
  4. Bradley, S. A., Smith, E. A., Lai, M. H., Graham, J. W., & Caldwell, L. L. (2010, June). Sexual coercion-risk in South African adolescent male youth: Risk correlates and HealthWise program impacts. In E. A. Smith (Chair), Sex, drugs, and deviance: Understanding prevention in South Africa. Symposium at the Society for Prevention Research annual meeting, Denver, CO.Google Scholar
  5. Bradshaw, C. P., Koch, C. W., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115.PubMedCrossRefGoogle Scholar
  6. Bronfenbrenner, U. (1995). Developmental ecology through space and time: A future perspective. In P. Moen, G. H. Elder Jr., & K. Luscher (Eds.), Examining lives in context: Perspectives on the ecology of human development (pp. 619–647). Washington, DC: The American Psychological Association.CrossRefGoogle Scholar
  7. Bumbarger, B. K., & Perkins, D. F. (2008). After randomized trials: Issues related to dissemination of evidence-based interventions. Journal of Children’s Services, 3(2), 55–64.Google Scholar
  8. Burns, B. J., & Hoagwood, K. (Eds). (2005). Evidence-based practices Part II: Effecting change. Child and Adolescent Psychiatric Clinics of North America, 14(2), xv–xvii.Google Scholar
  9. Caldwell, L. L., Baldwin, C. K., Walls, T., & Smith, E. A. (2004). Preliminary effects of a leisure education program to promote healthy use of free time among middle school adolescents. Journal of Leisure Research, 36(3), 310–335.Google Scholar
  10. Caldwell, L. L., Patrick, M., Smith, E. A., Palen, L., & Wegner, L. (2010). Influencing adolescent leisure motivation: Intervention effects of HealthWise South Africa. Journal of Leisure Research, 42, 203–220.Google Scholar
  11. Coffman, D. L., Smith, E. A., Flisher, A. J., & Caldwell, L. L. (2011). Effects of HealthWise South Africa on condom use self-efficacy. Prevention Science, 12, 162–172.PubMedCrossRefGoogle Scholar
  12. Collins, L. M., Baker, T. B., Mermelstein, R. J., Piper, M. E., Jorenby, D. E., Smith, S. S., et al. (2011). The multiphase optimization strategy for engineering effective tobacco use interventions. Annals of Behavioral Medicine, 41, 208–226.PubMedCrossRefGoogle Scholar
  13. Collins, L. M., Chakraborty, B., Murphy, S. A., & Strecher, V. (2009a). Comparison of a phased experimental approach and a single randomized clinical trial for developing multicomponent behavioral interventions. Clinical Trials, 6, 5–15.PubMedCrossRefGoogle Scholar
  14. Collins, L. M., Dziak, J. R., & Li, R. (2009b). Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs. Psychological Methods, 14, 202–224.PubMedCrossRefGoogle Scholar
  15. Collins, L. M., Murphy, S. A., Nair, V., & Strecher, V. (2005). A strategy for optimizing and evaluating behavioral interventions. Annals of Behavioral Medicine, 30, 65–73.PubMedCrossRefGoogle Scholar
  16. Collins, L. M., Murphy, S. A., & Strecher, V. (2007). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent e-health interventions. American Journal of Preventive Medicine, 32, S112–S118.PubMedCrossRefGoogle Scholar
  17. Conduct Problems Prevention Research Group. (2007). The fast track randomized controlled trial to prevent externalizing psychiatric disorders: Findings from grades 3 to 9. Journal of the American Academy of Child and Adolescent Psychiatry, 46, 1263–1272.CrossRefGoogle Scholar
  18. Conduct Problems Prevention Research Group. (2010). The effects of a multi-year randomized clinical trial of a universal social-emotional learning program: The role of student and school characteristics. Journal of Consulting and Clinical Psychology, 78(2), 156–168.CrossRefGoogle Scholar
  19. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation efforts out of control? Clincial Pschology Review, 18(1), 23–45.CrossRefGoogle Scholar
  20. Department of Basic Education. (2011). Curriculum and Assessment Policy Statement (CAPS). Life Orientation Grades 7–9, Final Draft. Pretoria: Department of Basic Education.Google Scholar
  21. Department of Health. (2000). National guidelines for the development of health promoting schools/sites in South Africa. Pretoria, South Africa: Author.Google Scholar
  22. Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1(3), 6–28.Google Scholar
  23. Dorrington, R., Johnson, L., Bradshaw, D., & Daniel, T. J. (2006). The demographic Impact of HIV/AIDS in South Africa: National and provincial indicators for 2006. Cape Town, South Africa: Centre for Actuarial Research, South African Medical Research Council, and Actuarial Society of South Africa.Google Scholar
  24. Drummond, M., O’Brien, B., Stoddart, G., & Torrance, G. (1997). Methods for economic evaluation of health care programmes. Oxford, NY: Oxford Medical Publications.Google Scholar
  25. Durlak, J. A., & DuPre, E. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.PubMedCrossRefGoogle Scholar
  26. Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.PubMedCrossRefGoogle Scholar
  27. Dusenbury, L., Brannigan, R., Falco, M., & Lake, A. (2004). An exploration of fidelity of implementation in drug abuse prevention among five professional groups. Journal of Alcohol & Drug Education, 47, 4–19.Google Scholar
  28. Dusenbury, L., Brannigan, R., Hansen, W. B., Walsh, J., & Falco, M. (2005). Quality of implementation: Developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research, 20, 308–313.PubMedCrossRefGoogle Scholar
  29. Elliott, D. S., & Mihalik, S. (2004). Issues in disseminating and replicating effective prevention programs. Prevention Science, 5, 47–53.PubMedCrossRefGoogle Scholar
  30. Ennett, S. T., Ringwalt, C. L., Thorne, J., Rohrbach, L. A., Vincus, A., Simons-Rudolph, A., et al. (2003). A comparison of current practice in school-based substance use prevention programs with meta-analytic findings. Prevention Science, 4, 1–14.PubMedCrossRefGoogle Scholar
  31. Fagan, A. A., Hanson, K., Hawkins, J. D., & Arthur, M. W. (2008). Bridging science to practice: Achieving prevention program implementation fidelity in the community youth development study. American Journal of Community Psychology, 41, 235–249.PubMedCrossRefGoogle Scholar
  32. Fisher, R. A. (1935). The design of experiments. Oxford, England: Oliver & Boyd.Google Scholar
  33. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).Google Scholar
  34. Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon Model of Parent Management Training (PMTO). Behavior Therapy, 36, 3–14.PubMedCrossRefGoogle Scholar
  35. Furr-Holden, C. D. M., Ialongo, N., Anthony, J. C., Petras, H., & Kellam, S. (2004). Developmentally inspired drug prevention: Middle school outcomes in a school-based randomized prevention trial. Drug and Alcohol Dependence, 73, 149–158.PubMedCrossRefGoogle Scholar
  36. Gager, P. J., & Elias, M. J. (1997). Implementing prevention programs in high-risk environments: Application of the resiliency paradigm. American Journal of Orthopsychiatry, 67(3), 363–373.PubMedCrossRefGoogle Scholar
  37. Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915–945.CrossRefGoogle Scholar
  38. Giles, S., Jackson-Newsom, J., Pankratz, M. M., Hansen, W. B., Ringwalt, C. L., & Dusenbury, L. (2008). Measuring quality of delivery in a substance use prevention program. Journal of Primary Prevention, 29(6), 489–501.PubMedCrossRefGoogle Scholar
  39. Goddard, R. D., & Goddard, Y. L. (2001). A multilevel analysis of teacher and collective efficacy. Teaching and Teacher Education, 17, 807–818.CrossRefGoogle Scholar
  40. Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research on Crime and Delinquency, 39, 3–35.CrossRefGoogle Scholar
  41. Graham, J. W., Flay, B. R., Johnson, C. A., Hansen, W. B., & Collins, L. M. (1984). Group comparability: A multiattribute utility measurement approach to the use of random assignment with small numbers of aggregated units. Evaluation Review, 8, 247–260.CrossRefGoogle Scholar
  42. Greenberg, M. T., Domitrovich, C. E., Graczyk, P. A., & Zins, J. E. (2005). The study of implementation in school-based preventive interventions: Theory, research and practice. Rockville, MD: Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Prevention.Google Scholar
  43. Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679.PubMedCrossRefGoogle Scholar
  44. Harshbarger, C., Simmons, G., Coelho, H., Sloop, K., & Collins, C. (2006). An empirical assessment of implementation, adaptation, and tailoring: The evaluation of CDC’s national diffusion of VOICES/VOCES. AIDS Education and Prevention, 18(Suppl. A), 184–197.PubMedCrossRefGoogle Scholar
  45. Jenson, P. S. (2003). Commentary: The next generation is overdue. Journal of the American Academy of Adolescent Psychiatry, 42, 527–530.CrossRefGoogle Scholar
  46. Joint United Nations Programme on HIV/AIDS (UNAIDS). (2010). UNAIDS report on the global AIDS epidemic | 2010. UNAIDS/10.11E | JC195BE. UNAIDS. ISBN 978-92-9173-871-7.Google Scholar
  47. Kirk, R. E. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Belmont, CA, US: Thomson Brooks/Cole Publishing.Google Scholar
  48. Knutson, N. M., Forgatch, M. S., & Rains, L. A. (2003). Fidelity of implementation rating system (FIMP): The training manual for PMTO [revised]. Eugene, OR: Oregon Social Learning Center.Google Scholar
  49. Kulis, S., Marsiglia, F. F., Elek, E., Dustman, P., Wagstaff, D. A., & Hecht, M. L. (2005). Mexican/Mexican American adolescents and keepin’ it REAL: An evidence-based substance use prevention program. Children and Schools, 3, 133–145.CrossRefGoogle Scholar
  50. Kumpfer, K. L., Alvarado, R., & Whiteside, H. O. (2003). Family-based interventions for substance use and misuse prevention. Substance Use and Misuse, 11–13, 1759–1787.CrossRefGoogle Scholar
  51. Leach, D. J., & Conto, H. (1999). The additional effects of process and outcome feedback following brief in-service teacher training. Educational Psychology, 19, 441–462.CrossRefGoogle Scholar
  52. Meintjies, H. (2010). Orphans of the AIDS epidemic? The extent, nature and circumstances of child-lead households in South Africa. AIDS Care, 22, 40–49.CrossRefGoogle Scholar
  53. Mihalic, S., Irwin, K., Fagan, A., Ballard, D., & Elliott, D. (2004). Successful program implementation: Lessons learned from Blueprints. Washington, DC: U.S. Department of Justice, Office of Justice Programs. Retrieved from
  54. National Conference on Health Promoting Schools. (2006). Executive summary of conference proceedings. South Africa: University of the Western Cape.Google Scholar
  55. Palen, L., Smith, E. A., Caldwell, L. L., Mathews, C., & Vergnani, T. (2009). Transitions to substance use and sexual intercourse among South African high school students. Substance Use and Misuse, 44(13), 1872–1887.PubMedCrossRefGoogle Scholar
  56. Palen, L., Smith, E. A., Flisher, A. J., Caldwell, L. L., & Mpofu, E. (2006). Substance use and sexual behavior among South African eighth grade students. Journal of Adolescent Health, 39(5), 761–763.PubMedCrossRefGoogle Scholar
  57. Patrick, M. E., Collins, L. M., Smith, E., Caldwell, L., Flisher, A., & Wegner, L. (2009). A prospective longitudinal model of substance use onset among South African adolescents. Substance Use & Misuse, 44, 647–662. PMCID: PMC2796627.CrossRefGoogle Scholar
  58. Payne, A. A., Gottfredson, D. C., & Gottfredson, G. D. (2006). School predictors of the intensity of implementation of school-based prevention programs. Prevention Science, 7, 225–237.PubMedCrossRefGoogle Scholar
  59. Pettifor, A. E., Rees, H. V., Steffenson, A., Hlongwa-Madikizela, L., MacPahil, C., Vermaak, K., et al. (2004). HIV and sexual behavior among young South Africans: A national survey of 15–24 year olds. Johannesburg, South Africa: Reproductive Health Research Unit, University of the Witwatersrand.Google Scholar
  60. Reddy, S. P., Sewpaul, J. S., Koopman, F., Funani, N. I., Sifunda, S., Josie, J., et al. (2010). Umthente Uhlaba Usamila—The South African Youth Risk Behaviour Survey 2008. Cape Town: South African Medical Research Council.Google Scholar
  61. Rohrbach, L. A., Grana, R., Sussman, S., & Valente, T. W. (2006). Type II translation. Transporting prevention interventions from research to real-world settings. Evaluation and the Health Professions, 29, 302–333.CrossRefGoogle Scholar
  62. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton, Mifflin & Co.Google Scholar
  63. Shisana, O., Rehle, T., Simbayi, L. C., Parker, W., Zuma, K., Bhana, A., et al. (2005). South African national HIV prevalence, HIV incidence, behaviour and communication survey, 2005. Cape Town, South Africa: HSRC Press.Google Scholar
  64. Smith, E. A., Palen, L., Caldwell, L. L., Graham, J. W., Flisher, A. J., Wegner, L., et al. (2008). Substance use and sexual risk prevention in Cape Town, South Africa: An evaluation of the HealthWise program. Prevention Science, 9(4), 311–321.PubMedCrossRefGoogle Scholar
  65. Snyder, J., Reid, J., Stoolmiller, M., Howe, G., Brown, H., Dagne, G., et al. (2006). The role of behavior observation in measurement systems for randomized prevention trials. Prevention Science, 7, 43–56.PubMedCrossRefGoogle Scholar
  66. Storr, C., Ialongo, N., Anthony, J., & Kellam, S. (2002). A randomized prevention trial of early onset tobacco use by school and family-based interventions implemented in primary school. Drug and Alcohol Dependence, 66, 51–60.PubMedCrossRefGoogle Scholar
  67. Strecher, V. J., McClure, J. B., Alexander, G. W., Chakraborty, B., Nair, V. N., Konkel, J. M., et al. (2008). Web-based smoking cessation programs: Results of a randomized trial. American Journal of Preventive Medicine, 34, 373–381.PubMedCrossRefGoogle Scholar
  68. Tibbits, M. K., Caldwell, L. L., Smith, E. A., & Wegner, L. (2009). The relation between profiles of leisure activity participation and substance use among South African youth. World Leisure Journal, 51, 150–159. NIHMSID: NIHMS163569.PubMedGoogle Scholar
  69. Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: The Interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41, 171–181.PubMedCrossRefGoogle Scholar
  70. Wegner, L., Flisher, A. J., Caldwell, L., Vergnani, T., & Smith, E. (2008). HealthWise South Africa: Cultural adaptation of a school-based risk prevention programme. Health Education Research, 23, 1085–1096.PubMedCrossRefGoogle Scholar
  71. Woolf, S. H. (2008). The meaning of translational research and why it matters. JAMA, 299, 211–213.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Linda L. Caldwell
    • 1
    Email author
  • Edward A. Smith
    • 1
  • Linda M. Collins
    • 1
  • John W. Graham
    • 1
  • Mary Lai
    • 1
  • Lisa Wegner
    • 2
  • Tania Vergnani
    • 2
  • Catherine Matthews
    • 3
    • 4
  • Joachim Jacobs
    • 2
  1. 1.The Pennsylvania State UniversityUniversity ParkUSA
  2. 2.The University of the Western CapeCape TownSouth Africa
  3. 3.The University of Cape TownCape TownSouth Africa
  4. 4.South African Medical Research CouncilCape TownSouth Africa

Personalised recommendations