Abstract
This chapter proposes that despite important advances in impact measurement and management practices over recent years, there remains a lack of attention to impact materiality in terms of the validity of data. It suggests that the various levels of impact materiality data risk in impact measurement are under-conceptualized. Poorly specified impact measurement methods ignore the impact materiality role of end-user voice which is both a data risk and may perpetuate structural inequalities through choices over who decides what is measured, how and for what reason. This can have serious consequences for organisational accountability [Relevant SDGs: SDG8: Decent Work and Economic Growth; SDG17: Partnerships for the Goals].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
See, for example critiques of ESG ratings systems – https://www.economist.com/finance-and-economics/2019/12/07/climate-change-has-made-esg-a-force-in-investing – as well as warnings over ‘greenwashing’ funds, https://www.ftadviser.com/investments/2020/07/16/be-critical-of-esg-credentials-to-avoid-greenwashing-funds/
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
Evidence-based policymaking first emerged with the landmark Cambridge Somerville Youth Study (Powers & Witmer, 1951). Seen as one of the first Randomised Controlled Trials (RCTs), it selected youths to partake in counselling and group recreational activities to reduce rates of juvenile delinquency.
- 15.
These studies were screened for relevance including full-text screening by two researchers. The researchers then coded these studies and synthesised their results in terms of the impact of micro-finance across financial, health, nutrition, education, child labour, women’s empowerment, housing, job creation, and social cohesion outcomes.
- 16.
- 17.
The programme recruited over a thousand participants in forty villages and towns. An RCT approach matched 20 treatment and 20 control villages with each other—with one group receiving the intervention and one group not receiving it. The RCT demonstrated positive outcomes in terms of reducing abuse, corporate punishment, and generally improving parenting behaviours. This program was so successful that it was replicated by its sponsors across multiple countries including the Philippines and Thailand.
- 18.
Two cohorts of children were observed for eight months to determine the effect of improved sewerage coverage, comparing the children across time to determine the effect of the intervention. As researchers had no control over the treatment of the sewerage intervention, an RCT was not possible, and researchers had to use an observational design. This intervention was found to be successful, reducing diarrhoea by over 20%.
- 19.
This was performed by identifying 129 young people who had made an attempt against a control of 153 control subjects chosen from the same communities. The study then attempted to identify underlying risk factors and differences between the subjects that could explain the attempt. Factors found included:
-
Low socioeconomic backgrounds and lack of formal education
-
Childhood and family adversity
-
Neuroticism and hopelessness
-
Affective and substance use disorders
-
Higher rates of reported life events
-
- 20.
This study used the Swedish government’s Living Conditions Survey as well as phone interviews. The study compared end-users’ working environment (survey) to the prevalence of the disease (interview). While the study suggested that people with more stressful working environments had a higher prevalence of disease, the method only tests for correlations not causal relationship.
- 21.
See Lean Data models such as Keystone and 60 Decibels, as well as the underpinning principles of SROI.
References
2Degrees Investing Initiative. (n.d.). Impact Washing Gets a Free Ride. Available online at https://2degrees-investing.org/resource/impact-washing-gets-a-free-ride/
Ab, H. (2005). Meta-Analysis in Medical Research. In Meta-Analysis in Medical Research (2010, 1). Wiley. https://doi.org/10.1002/9780470994894
Acumen. (2015). The Lean Data Field Guide – Tips for Collecting Customer Data to Build more Impactful Businesses. Available online at: https://acumen.org/wp-content/uploads/2015/11/Lean-Data-Field-Guide.pdf.
Arnstein, S. R. (1969). A Ladder of Citizen Participation. Journal of the American Planning Association, 35(4), 216–224. https://doi.org/10.1080/01944366908977225
Baron, J. (2018). A Brief History of Evidence-Based Policy. The Annals of the American Academy of Political and Social Science, 678(1), 40–50. https://doi.org/10.1177/0002716218763128
Barreto, M. L., Genser, B., Strina, A., Teixeira, M. G., Assis, A. M. O., Rego, R. F., Teles, C. (2018). Asian Development Bank: Strategy 2030 – Prosperous, Inclusive, Resilient, Sustainable. Asia and the Pacific.https://doi.org/10.22617/TCS189401-2
Beautrais, A. L. (1996, August). Serious Suicide Attempts in Young People: A Case Control Study. Department of Psychological Medicine.
Belfield, C., & Levin, H. M. (2010). Cost-Benefit Analysis and Cost-Effectiveness Analysis. In International Encyclopaedia of Education (pp. 199–203). Elsevier Ltd. https://doi.org/10.1016/B978-0-08-044894-7.01245-8
Better Evaluation. (2014). Choose Methods and Processes | Better Evaluation. https://www.betterevaluation.org/en/choose-methods-and-processes
Bhattacherjee, A. (2012). Social Science Research: Principles, Methods, and Practices. http://scholarcommons.usf.edu/oa_textbookshttp://scholarcommons.usf.edu/oa_textbooks/3
Bumbarger, B. K., & Owen, J. (2009). The Impact and Cost-Effectiveness of Taking Evidence-Based Prevention to Scale: Lessons Learned from One State’s Experience. https://www.purdue.edu/hhs/hdfs/fii/wp-content/uploads/2015/07/s_ncfis05c05.pdf
Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011). The Levels of Evidence and their Role in Evidence-Based Medicine. Plastic and Reconstructive Surgery, 128(1), 305–310. https://doi.org/10.1097/PRS.0b013e318219c171
Calder, B. J., Phillips, L. W., & Tybout, A. M. (1982). The Concept of External Validity. Journal of Consumer Research, https://d1wqtxts1xzle7.cloudfront.net/6629388/calderphillipstybout_jcr82_external_validity-with-cover-page-v2.pdf?Expires=1631273547&Signature=HPqL8snUHOAEOoi94eEC1RLa-DivwRc1p3eprsdkMCY7j6TL9H2AoGjFzcfhwFUPr1NJB944zor5a7xCgPlJYRUff2uh9EFIFF8tXfTf2-Uxh~nTySF8baQrPDL4i6ameWFQji48jrA97iNxRj~Drn~nZzI7VXmhaTnlmjatjxlCUymhSuE-e2GWxzWWlZGr-Ys8X0~t6dCSKBnFX6ucdyBYC98BKZQKmCeVK3j444d4tFMOZvTyic2XLXkkqDsWZBfueb8kuKwwSyjtVhjSdHuteZPbq1pwm0FLRMrJ9Amiw~6HJwf38g-XuUanddyYJX0nouQ9upzEC2aFEVDSPA__&Key-Pair-Id=APKAJLOHF5GGSLRBV4ZA
Canadian International Development Agency. (2001). How to Perform Evaluations Participatory Evaluations.
Canadian Task Force on the Periodic Health Examination. (1979). The Periodic Health Examination.
Cartwright. (2007). Are RCTs the gold standard? 11–20. https://doi.org/10.1017/S1745855207005029
Cherry, K. (2019). The Definition and Use of a Cross-Sectional Study.https://www.verywellmind.com/what-is-a-cross-sectional-study-2794978
Choueiry, G. (n.d.). Cohort vs Randomized Controlled Trials: A Simple Explanation – Quantifying Health. Retrieved April 16, 2021, from https://quantifyinghealth.com/cohort-vs-randomized-controlled-trials/
Cluver, L. D., Meinck, F., Steinert, J. I., Shenderovich, Y., Doubt, J., Romero, R. H., Lombard, C. J., Redfern, A., Ward, C. L., Tsoanyane, S., Nzima, D., Sibanda, N., Wittesaele, C., de Stone, S., Boyes, M. E., Catanho, R., Lachman, J. M. L., Salah, N., Nocuza, M., & Gardner, F. (2018). Parenting for Lifelong Health: A Pragmatic Cluster Randomised Controlled Trial of a Non-commercialised Parenting Programme for Adolescents and Their Families in South Africa. BMJ Global Health, 3(1), 539. https://doi.org/10.1136/bmjgh-2017-000539
Cohn, J. P. (2008). Citizen Science: Can Volunteers Do Real Research? Bioscience, 58(3), 192–197. https://doi.org/10.1641/B580303
Cylus, J., Smith, P. C., & Smith, P. C. (2020). The Economy of Wellbeing: What Is it and What Are the Implications for Health? The BMJ, 369. https://doi.org/10.1136/bmj.m1874
Denyer, D., & Tranfield, D. (2009). The Sage Handbook of Organisational Research Methods. Sage Publications Ltd. https://psycnet.apa.org/record/2010-00924-039
Ely, A., & Hearn, D. (2021). To Advance Equity, Impact Investors Must Examine Power, Not Just Provide Capital. Stanford Social Innovation Review. https://ssir.org/articles/entry/impact_investors_need_to_share_power_not_just_capital#
Evans, D. (2003). Hierarchy of Evidence: A Framework for Ranking Evidence Evaluating Healthcare Interventions. Journal of Clinical Nursing, 12(1), 77–84. https://doi.org/10.1046/j.1365-2702.2003.00662.x
Farnham, K. (2014). The Ladder of Citizen Participation. Retrieved April 17, 2021, from https://catcomm.org/ladder-participation/
FCA. (2018). How We Analyse the Costs and Benefits of Our Policies.
Freitas, C. (2013). Rocinha and Alemão Plan to Sue the State Over Cable Car. Retrieved April 17, 2021, from https://www.rioonwatch.org/?p=11466
Garaway, G. B. (1995). Participatory Evaluation. Studies in Educational Evaluation, 21(1), 85–102. https://doi.org/10.1016/0191-491X(95)00007-H
Genser, B., Strina, A., dos Santos, L. A., Teles, C. A., Prado, M. S., Cairncross, S., & Barreto, M. L. (2008). Impact of a City-Wide Sanitation Intervention in a Large Urban Centre on Social, Environmental and Behavioural Determinants of Childhood Diarrhoea: Analysis of Two Cohort Studies. International Journal of Epidemiology, 37(4), 831–840. https://doi.org/10.1093/ije/dyn101
Muir Gray. (1996). Evidence-based Healthcare and Public Health. Churchill Livingstone. http://sutlib2.sut.ac.th/sut_contents/H126376.pdf
Greater Manchester Combined Authority. (2019). How Can I Estimate the Programme Costs in My Charity’s Funding Proposal? https://www.inspiringimpact.org/resource-library/unit-cost-database/
Green, J., Wan, M. W., Holsgrove, S., and Mcnally, J. E. (2013). Intervention for Infants at Risk of Developing Autism: A Case Series Attention Training for Infants at Risk of ADHD (INTER-STAARS) View project PACT-Pre-school Autism Communication Trial and Follow-up Study View project. Journal of Autism and Developmental Disorders. 10.1007/s10803-013-1797-8
Guijt, I. (2014). Participatory Approaches. www.unicef-irc.org
Hariton, E., & Locascio, J. J. (2018). Randomised Controlled Trials – The Gold Standard for Effectiveness Research: Study Design: Randomised Controlled Trials. BJOG: An International Journal of Obstetrics and Gynaecology, 125(13), 1716. https://doi.org/10.1111/1471-0528.15199
Hart, R. (1992). “Children’s Participation, from Tokenism to Citizenship”. UNICEF.
Impact Management Project. (n.d.). Risk. Impact Management Project. Retrieved April 17, 2021, from https://impactmanagementproject.com/impact-management/impact-management-norms/risk/#anchor2
Involve UK. (n.d.-a). Citizens’ Assembly. Retrieved April 7, 2021, from https://www.involve.org.uk/resources/methods/citizens-assembly
Involve UK. (n.d.-b). Citizens’ Jury. Retrieved April 7, 2021, from https://www.involve.org.uk/resources/methods/citizens-jury
Alan Irwin. (1995). Citizen Science: A Study of People, Expertise and Sustainable Development – Alan Irwin, Irwin a Staff – Google Books. https://books.google.co.uk/books?hl=en&lr=&id=MFiuIsC5hAUC&oi=fnd&pg=PR9&dq=citizen+science&ots=1wtj4csea5&sig=y3Ywe68z95kASIwSwOd-2uMjF3Y&redir_esc=y#v=onepage&q=citizen%20science&f=false
Johnson, J. V., & Hall, E. M. (1988). Job Strain, Work Place Social Support, and Cardiovascular Disease: A Cross-Sectional Study of a Random Sample of the Swedish Working Population (Vol. 78, pp. 1336–1342). Journal of Public Health.
King, J. A., Cousins, J. B., & Whitmore, E. (1998). Making Sense of Participatory Evaluation: Framing Participatory Evaluation Construct Validity in Program Evaluation View Project Evaluation Capacity Building View Project. New Directions for Evaluation. https://doi.org/10.1002/ev.226
Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-Analysis. In Applied Social Research (p. 49). Sage Publications. https://psycnet.apa.org/record/2000-16602-000
MacGill, M. (2018). Cohort Study: Finding Causes, Examples, and Limitations. Medical News Today. https://www.medicalnewstoday.com/articles/281703
Masi, C. M., Chen, H.-Y., Hawkley, L. C., & Cacioppo, J. T. (2010). A Meta-Analysis of Interventions to Reduce Loneliness. Personality and Social Psychology Review. https://doi.org/10.1177/1088868310377394
McLeroy, K. R., Norton, B. L., Kegler, M. C., Burdine, J. N., & Sumaya, C. V. (2003). Community-Based Interventions. American Journal of Public Health, 93, 529–533. https://doi.org/10.2105/AJPH.93.4.529
Murad, M. H., Sultan, S., Haffar, S., & Bazerbachi, F. (2018). Methodological Quality and Synthesis of Case Series and Case Reports. Evidence-Based Medicine, 23(2), 60–63. https://doi.org/10.1136/bmjebm-2017-110853
National Community Land Trust Network. (n.d.). What Is CLT? Retrieved 17 April 2021, from http://www.communitylandtrusts.org.uk/what-is-a-clt
New Economy. (2015). Unit Cost Database. https://golab.bsg.ox.ac.uk/knowledge-bank/resources/unit-cost-database/
Nicholls, A. (2018). A General Theory of Social Impact Accounting: Materiality, Uncertainty and Empowerment. Journal of Social Entrepreneurship, 9(2), 132–153.
Nicholls, A., & Tomkinson, E. (2015a). The Peterborough Social Impact Bond. In A. Nicholls, R. Paton, & J. Emerson (Eds.), Social Finance (pp. 331–378). Oxford University Press.
Nicholls, A., & Tomkinson, E. (2015b). Risk and Return in Social Finance: ‘I Am the Market’. In A. Nicholls, R. Paton, & J. Emerson (Eds.), Social Finance (pp. 278–306). Oxford University Press.
NICHSR. (2016). Health Economics Information Resources: A Self-Study Course: Module 4. U.S. National Library of Medicine.
Parker, R., & Robinson, E. (2013). Evidence-based Practice and Service-Based Evaluation | Child Family Community Australia. https://aifs.gov.au/cfca/publications/evidence-based-practice-and-service-based-evaluation
Petticrew, M., & Roberts, H. (2003). Evidence, Hierarchies, and Typologies: Horses for Courses. Journal of Epidemiology and Community Health, 57, 527–529. http://www.evidencenetwork.org/
Pinchbeck, G. L., & Archer, D. C. (2018). How to Critically Appraise a Paper. Equine Veterinary Education, 32(2), 104–109. https://doi.org/10.1111/eve.12896
Powers, E., & Witmer, H. (1951). An Experiment in the Prevention of Delinquency; the Cambridge-Somerville Youth Study. – PsycNET. Columbia University Press. https://psycnet.apa.org/record/1952-00430-000
Sackett, D. L. (1989). Rules of Evidence and Clinical Recommendations on the Use of Antithrombotic Agents 95(2), 2S–4S. https://doi.org/10.1378/chest.95.2_supplement.2s
Sayre, J. W., Toklu, H. Z., Ye, F., Mazza, J., & Yale, S. (2017). Case Reports, Case Series – From Clinical Practice to Evidence-Based Medicine in Graduate Medical Education. Cureus, 9(8). https://doi.org/10.7759/cureus.1546
Shadish, W., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin.
Sibbald, B., & Roland, M. (1998). Understanding Controlled Trials: Why Are Randomised Controlled Trials Important? BMJ, 316(7126), 201. https://doi.org/10.1136/bmj.316.7126.201
Song, J. W., & Chung, K. C. (2010). Observational Studies: Cohort and Case-Control Studies. Plastic and Reconstructive Surgery, 126(6), 2234–2242. https://doi.org/10.1097/PRS.0b013e3181f44abc
University of Oxford. (n.d.). Parenting for Lifelong Health. Retrieved April 16, 2021, from https://www.ox.ac.uk/research/research-impact/parenting-lifelong-health
van Rooyen, C., Stewart, R., & de Wet, T. (2012). The Impact of Microfinance in Sub-Saharan Africa: A Systematic Review of the Evidence. World Development, 40(11), 2249–2262. https://doi.org/10.1016/j.worlddev.2012.03.012
VanLandingham, G. (2020). Evaluation in the Age of Evidence-Based Policymaking: Promises, Pitfalls and Paths Forward. Evaluation Journal of Australasia, 20(3), 129–139. https://doi.org/10.1177/1035719X20928164
Viswanathan, M., Ammerman, A., Eng, E., Garlehner, G., Lohr, K., Griffith, D., Rhodes, S., Samuel-Hodge, C., Maty, S., Lux, L., Webb, L., Sutton, S., Swinson, T., Jackman, A., and Whitener, L. (2004). Community-Based Participatory Research: Assessing the Evidence: Summary. AHRQ. https://www.ncbi.nlm.nih.gov/sites/books/NBK11852/.
Wagenaar, A. C., Murray, D. M., Gehan, J. P., Wolfson, M., Forster, J. L., Toomey, T. L., Perry, C. L., and Jones-Webb, R. (1998). Communities Mobilizing for Change on Alcohol: Outcomes from a Randomized Community Trial*.
Wallerstein, N., Duran, B., Oetzel, J., & Minkler, M. (2017). Community-Based Participatory Research for Health: Advancing Social and Health Equity. Google Books. John Wiley and Sons. https://books.google.co.uk/books?hl=en&lr=&id=GLg4DwAAQBAJ&oi=fnd&pg=PA31&dq=CBPR&ots=lGszxsDvzK&sig=360yFVQnIBNzX04szcVzdmOiqGc&redir_esc=y#v=onepage&q=CBPR&f=false
Wang, X., & Cheng, Z. (2020). Cross-Sectional Studies: Strengths, Weaknesses, and Recommendations. Chest, 158(1), S65–S71. https://doi.org/10.1016/j.chest.2020.03.012
WHO. (2020). Parenting for Lifelong Health (PLH) programme manuals. WHO; World Health Organization. http://www.who.int/violence_injury_prevention/violence/child/PLH-manuals/en/
WHO. (n.d.). Parenting for Lifelong Health. Retrieved April 16, 2021, from https://www.who.int/teams/social-determinants-of-health/parenting-for-lifelong-health
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Nicholls, A., Yee, E. (2022). Enhancing Impact Materiality: Lessons from Evidenced-Based Policy Making. In: Hazenberg, R., Paterson-Young, C. (eds) Social Impact Measurement for a Sustainable Future. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-83152-3_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-83152-3_13
Published:
Publisher Name: Palgrave Macmillan, Cham
Print ISBN: 978-3-030-83151-6
Online ISBN: 978-3-030-83152-3
eBook Packages: Business and ManagementBusiness and Management (R0)