Abstract
Evaluation of primary prevention and health promotion programs contributes necessary information to the evidence base for prevention programs. There is increasing demand for high-quality evaluation of program impact and effectiveness for use in public health decision making. Despite the demand for evidence and known benefits, evaluation of prevention programs can be challenging and organizations face barriers to conducting rigorous evaluation. Evaluation capacity building efforts are gaining attention in the prevention field; however, there is limited knowledge about how components of the health promotion and primary prevention system (e.g., funding, administrative arrangements, and the policy environment) may facilitate or hinder this work. We sought to identify the important influences on evaluation practice within the Australian primary prevention and health promotion system. We conducted in-depth semi-structured interviews with experienced practitioners and managers (n = 40) from government and non-government organizations, and used thematic analysis to identify the main factors that impact on prevention program evaluation. Firstly, accountability and reporting requirements impacted on evaluation, especially if expectations were poorly aligned between the funding body and prevention organization. Secondly, the funding and political context was found to directly and indirectly affect the resources available and evaluation approach. Finally, it was found that participants made use of various strategies to modify the prevention system for more favorable conditions for evaluation. We highlight the opportunities to address barriers to evaluation in the prevention system, and argue that through targeted investment, there is potential for widespread gain through improved evaluation capacity.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baugh Littlejohns, L., Baum, F., Lawless, A., & Freeman, T. (2018). Disappearing health system building blocks in the health promotion policy context in South Australia (2003–2013). Critical Public Health, 1–13. https://doi.org/10.1080/09581596.2017.1418501.
Bourgeois, I., & Cousins, J. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34, 299–319. https://doi.org/10.1177/1098214013477235.
Bowen, S., & Zwi, A. B. (2005). Pathways to “evidence-informed” policy and practice: a framework for action. PLoS Medicine, 2, e166. https://doi.org/10.1371/journal.pmed.0020166.
Brehaut, J. C., & Eva, K. W. (2012). Building theories of knowledge translation interventions: Use the entire menu of constructs. Implementation Science, 7, 114. https://doi.org/10.1186/1748-5908-7-114.
Brug, J., Tak, N. I., & Te Velde, S. J. (2011). Evaluation of nationwide health promotion campaigns in the Netherlands: an exploration of practices, wishes and opportunities. Health Promotion International, 26, 244–254. https://doi.org/10.1093/heapro/daq058.
Carman, J. G. (2011). Understanding evaluation in nonprofit organizations. Public Performance & Management Review, 34, 350–377. https://doi.org/10.2753/PMR1530-9576340302.
Carman, J. G., & Fredericks, K. A. (2009). Evaluation capacity and nonprofit organizations: Is the glass half-empty or half-full? American Journal of Evaluation, 31, 84–104. https://doi.org/10.1177/1098214009352361.
Dahler-Larsen. (2009). Evaluation and public management. In E. Ferlie, L. E. Lynn, & C. Pollitt (Eds.), The Oxford handbook of public management. Oxford: Oxford University Press.
Datta, J., & Petticrew, M. (2013). Challenges to evaluating complex interventions: a content analysis of published papers. BMC Public Health, 13, 568. https://doi.org/10.1186/1471-2458-13-568.
Edwards, B., Stickney, B., Milat, A., Campbell, D., & Thackway, S. (2016). Building research and evaluation capacity in population health: the NSW health approach. Health Promotion Journal of Australia, 27, 264–267. https://doi.org/10.1071/HE16045.
Francis, L. J., & Smith, B. J. (2015). Toward best practice in evaluation: a study of Australian health promotion agencies. Health Promotion Practice, 16, 715–723. https://doi.org/10.1177/1524839915572574.
Huckel Schneider, C., Milat, A. J., & Moore, G. (2016). Barriers and facilitators to evaluation of health policies and programs: Policymaker and researcher perspectives. Evaluation and Program Planning, 58, 208–215. https://doi.org/10.1016/j.evalprogplan.2016.06.011.
Jackson, H., & Shiell, A. (2017). Preventive health: How much does Australia spend and is it enough? Canberra: Foundation for Alcohol Research and Education.
Jolley, G. M., Lawless, A. P., Baum, F. E., Hurley, C. J., & Fry, D. (2007). Building an evidence base for community health: a review of the quality of program evaluations. Australian Health Review, 31, 603–610.
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33, 307–338. https://doi.org/10.1177/1098214011434608.
Liket, K. C., Rey-Garcia, M., & Maas, K. E. H. (2014). Why aren’t evaluations working and what to do about it. American Journal of Evaluation, 35, 171–188. https://doi.org/10.1177/1098214013517736.
Lobo, R., McManus, A., Brown, G., Hildebrand, J., & Maycock, B. (2010). Evaluating peer-based youth programs: Barriers and enablers. Evaluation Journal of Australasia, 10, 36–43.
Lobo, R., Petrich, M., & Burns, S. K. (2014). Supporting health promotion practitioners to undertake evaluation for program development. BMC Public Health, 14. https://doi.org/10.1186/1471-2458-14-1315.
Lobo, R., Crawford, G., Hallett, J., Laing, S., Mak, D. B., Jancey, J., …, Doherty, M. (2016). A research and evaluation capacity building model in Western Australia. Health Promotion International https://doi.org/10.1093/heapro/daw088.
Milat, A., & Li, B. (2017). Narrative review of frameworks for translating research evidence into policy and practice. Public Health Research & Practice, 27, e2711704. https://doi.org/10.17061/phrp2711704.
Milat, A. J., King, L., Bauman, A. E., & Redman, S. (2013). The concept of scalability: Increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promotion International, 28, 285–298.
Napp, D., Gibbs, D., Jolly, D., Westover, B., & Uhl, G. (2002). Evaluation barriers and facilitators among community-based HIV prevention programs. AIDS Education and Prevention, 14, 38–48.
Nebot, M. (2006). Health promotion evaluation and the principle of prevention. Journal of Epidemiology and Community Health, 60, 5–6.
Nielsen, S. B., Lemire, S., & Skov, M. (2011). Measuring evaluation capacity—Results and implications of a Danish study. American Journal of Evaluation, 32, 324–344. https://doi.org/10.1177/1098214010396075.
Norton, S., Milat, A., Edwards, B., & Giffin, M. (2016). Narrative review of strategies by organizations for building evaluation capacity. Evaluation and Program Planning, 58, 1–19. https://doi.org/10.1016/j.evalprogplan.2016.04.004.
Nutbeam, D. (1998). Promoting the health of Australians-how strong is our infrastructure support? Australian and New Zealand Journal of Public Health, 22, 301–302.
Oxman, A. D., Bjørndal, A., Becerra-Posada, F., Gibson, M., Block, M. A. G., Haines, A., … Wibulpolprasert, S. (2010). A framework for mandatory impact evaluation to ensure well informed public policy decisions. The Lancet, 375, 427–431. https://doi.org/10.1016/S0140-6736(09)61251-4.
Petticrew, M. (2013). Public health evaluation: Epistemological challenges to evidence production and use. Evidence & Policy: A Journal of Research, Debate and Practice, 9, 87–95. https://doi.org/10.1332/174426413X663742.
Pettman, T. L., Armstrong, R., Doyle, J., Burford, B., Anderson, L. M., Hillgrove, T., … Waters, E. (2012). Strengthening evaluation to capture the breadth of public health practice: Ideal vs. real. Journal of Public Health (Oxford, England), 34, 151–155. https://doi.org/10.1093/pubmed/fds014.
Rychetnik, L., Bauman, A., Laws, R., King, L., Rissel, C., Nutbeam, D., … Caterson, I. (2012). Translating research for evidence-based public health: Key concepts and future directions. Journal of Epidemiology and Community Health, 66, 1187–1192. https://doi.org/10.1136/jech-2011-200038.
Schwarzman, J., Bauman, A., Gabbe, B., Rissel, C., Shilton, T., & Smith, B. J. (2018). Organizational determinants of evaluation practice in Australian prevention agencies. Health Education Research, Published online ahead of print, 1–13. https://doi.org/10.1093/her/cyy015.
Smith, B. J., Rissel, C., Shilton, T., & Bauman, A. (2016). Advancing evaluation practice in health promotion. Health Promotion Journal of Australia, 27, 184–186. https://doi.org/10.1071/HEv27n3_ED2.
South, J., & Tilford, S. (2000). Perceptions of research and evaluation in health promotion practice and influences on activity. Health Education Research, 15, 729–741. https://doi.org/10.1093/her/15.6.729.
Spoth, R., Rohrbach, L., Greenberg, M., Leaf, P., Brown, C., Fagan, A., … Hawkins, J. (2013). Addressing Core challenges for the next generation of type 2 translation research and systems: the translation science to population impact (TSci impact) framework. Prevention Science, 14, 319–351. https://doi.org/10.1007/s11121-012-0362-6.
van Koperen, T. M., Renders, C. M., Spierings, E. J. M., Hendriks, A.-M., Westerman, M. J., Seidell, J. C., & Schuit, A. J. (2016). Recommendations and improvements for the evaluation of integrated community-wide interventions approaches. Journal of Obesity, 2016, 2385698–2385698. https://doi.org/10.1155/2016/2385698.
Wutzke, S., Morrice, E., Benton, M., & Wilson, A. (2016). What will it take to improve prevention of chronic diseases in Australia? A case study of two national approaches. Australian Health Review, 41, 176–181. https://doi.org/10.1071/AH16002.
Wutzke, S., Morrice, E., Benton, M., Milat, A., Russell, L., & Wilson, A. (2017). Australia's National Partnership Agreement on preventive health: Critical reflections from states and territories. Health Promotion Journal of Australia. https://doi.org/10.1002/hpja.9.
Zardo, P., Collie, A., & Livingstone, C. (2014). External factors affecting decision-making and use of evidence in an Australian public health policy environment. Social Science and Medicine, 108, 120–127. https://doi.org/10.1016/j.socscimed.2014.02.046.
Acknowledgements
We would like to thank Emily McCluskey for her valuable contribution to participant recruitment and data analysis in this study.
Funding
This work was supported by the Australian Research Council (ARC) [DP150103575]. Prof. Gabbe was supported by ARC Future Fellowship [FT170100048] during the preparation of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic Supplementary Material
ESM 1
(DOCX 15 kb)
Rights and permissions
About this article
Cite this article
Schwarzman, J., Bauman, A., Gabbe, B.J. et al. The Funding, Administrative, and Policy Influences on the Evaluation of Primary Prevention Programs in Australia. Prev Sci 20, 959–969 (2019). https://doi.org/10.1007/s11121-019-00997-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-019-00997-4