Abstract
In this chapter, we examine how impact evaluations can contribute to measuring and understanding the overall effectiveness, efficiency and sustainability of forestry programmes. In most cases we find that impact evaluations have used quasi-experimental methods rather than experimental methods to identify and measure change that can be causally attributed to forestry programmes. We conclude that in measuring the change that be attributed to these programmes, impact evaluation methods help to measure the overall effect, deal with sources of potential bias and mitigate confounding factors while undertaking these measurements. Impact evaluations also hold enormous potential because they are able to leverage the potential held by big and open data. However caution must also be exercised in using these methods. Impact evaluation methodologies must also incorporate causal pathways and methods of implementation research if they are to be relevant to policy and programme managers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This is the sense in which we use the term ‘impact evaluations’. Other definitions also exist. Different definitions of impact evaluations emphasize different aspects of the causal chain. The OECD-DAC defines it as the ‘The positive and negative, primary and secondary long term effects produced by a development intervention, directly or indirectly, intended or unintended.’ The World Bank defines it as ‘assessing changes in the well-being of individuals, households, communities or firms, that can be attributed to a particular programme, policy or work’. We use the meaning of impact evaluations as the one used by 3ie: ‘Rigorous impact evaluation studies are analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific programme using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context.’
- 2.
See Cropper et al. (2001).
- 3.
Three of the four studies (one is ongoing) show that community based management systems did reduce forest clearing (see Table 15.2). The time periods over which these interventions are also examined are very short.
- 4.
The most probable mechanism for the positive income effects of national parks is increased income from tourism.
- 5.
Although this requires that several fairly stringent conditions are fulfilled – for example the intervention needs to be the same, the outcome needs to be the same and the assumption that the different datasets are coming from the same underlying statistical population which has the same underlying distribution, can be a strong one.
- 6.
Personal conversations.
- 7.
This is not new. Several other agencies have adopted this nomenclature. Innovations for Poverty Action (IPA) also uses a similar nomenclature.
References
Alix-Garcia J, Mcintosh C, Sims KRE, Welch JR (2013) The ecological footprint of poverty alleviation: evidence from Mexico’s oportunidades program. Rev Econ Stat 95:417–435
Alix-Garcia J, Aronson G, Radeloff V, Ramirez-Reyes C, Shapiro E, Sims K, Yañez-Pagans P (2014) Environmental and socioeconomic impacts of Mexico’s payments for ecosystem services program
Andam KS, Ferraro PJ, Pfaff ASP, Sanchez-Azofeifa GA (2007) Protected areas and avoided deforestation: a statistical evaluation. Final report. Global Environment Facility Evaluation Office, Washington, DC
Andersson C, Mekonnen A, Stage J (2011) Impacts of the productive safety net program in Ethiopia on livestock and tree holdings of rural households. J Dev Econ, Elsevier 94(1):119–126
Arriagada RA, Ferraro PJ, Sills EO, Pattanayak SK, Cordero-Sancho S (2012) Do payments for environmental services affect forest cover? A farm-level evaluation from Costa Rica. Land Econ 88:382–399
Bensch G, Peters J (2011) Alleviating deforestation pressures? Impacts of improved stove dissemination on Charcoal consumption in urban Senegal. Land Econ 89(4):676–698
Bowler D, Buyung-Ali L, Healey JR, Jones JPG, Knight T, Pullin AS (2010) The evidence base of community forest management as a mechanism for supplying global environmental benefits and improving local welfare. Environmental Evidence: www.environmentalevidence.org/SR48.html
Bravo-Ureta BE, Almeida AN, SolÃs D, Inestroza A (2011) The economic impact of Marena’s investments on sustainable agricultural systems in Honduras. J Agric Econ 62(2):429–448
Burwen J, Levine DI (2012) A rapid assessment randomized-controlled trial of improved cookstoves in rural Ghana. Energy Sustain Dev 16:328–338
Chibwana C, Jumbe CBL, Shively G (2013) Agricultural subsidies and forest clearing in Malawi. Environ Conserv 40(01):60–70
Cropper M, Puri J, Griffiths C (2001) How the location of roads and protected areas affects deforestation in North Thailand
Edmonds EV (2002) Government-initiated community resource management and local resource extraction from Nepal’s forests. J Dev Econ 68:89–115
Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CM (2011) Impact evaluation in practice. World Bank, Washington, DC
Hafashimana D, Jayachandran S, Stanton C, de Laat J, Kalenscher T (forthcoming) Testing the effectiveness of payments for ecosystem services to enhance conservation in productive landscapes in Uganda; a prospective randomized evaluation. The International Initiative for Impact Evaluation (3ie)
List JA, Levitt SD (2005) What do laboratory experiments tell us about the real world. NBER working paper
Nelson A, Chomitz KM (2009) Protected area effectiveness in reducing tropical deforestation: a global analysis of the impact of protection status. Eval Brief 7:31
Pfaff A, Robalino JA, Sanchez-Azofeifa GA (2008) Payments for environmental services: empirical analysis for Costa Rica. Terry Sanford Institute of Public Policy, Duke University, Durham
Puri J (2006) Factors affecting agricultural expansion in forest reserves of Thailand: the role of population and roads. Ph.D. dissertation, University of Maryland
Puri J, Khosla A, Oldenbeuving M (2015) Grants for real-world impact evaluations: What are we learning? A statistical overview and a process analysis of 3ie Open Window Grants. 3ie working paper 23. International Initiative for Impact Evaluation (3ie) (forthcoming), New Delhi
Samii C, Lisiecki M, Kulkarni P, Paler L, Chavis L (2014) Effects of Decentralized Forest Management (DFM) on deforestation and poverty in low and middle income countries: a systematic review. Campbell Syst Rev 2014:10
Scullion J, Thomas CW, Vogt KA, Perez-Maqueo O, Logsdon MG (2011) Evaluating the environmental impact of payments for ecosystem services in Coatepec (Mexico) using remote sensing and on-site interviews. Environ Conserv 38:426–434
Sims K (2008) Evaluating the local socio-economic impacts of protected areas: a system level comparison group approach. Global environment facility impact evaluation information document, Washington, DC
Somanathan E, Prabhakar R, Mehta BS (2005) Does decentralization work? Forest conservation in the Himalayas. Planning unit discussion paper (05-04), Indian Statistical Institute, Planning Unit, New Delhi Discussion Papers, Indian Statistical Institute, New Delhi
Stern E, Stame N, Mayne J, Forss K, Davies R, Befani B (2012) Broadening the range of designs and methods for impact evaluations. Report of a study commissioned by the Department for International Development, working paper 38. DFID report
Tachibana T, Adhikari S (2009) Does community-based management improve natural resource condition? Evidence from the forests in Nepal. Land Econ 85:107–131
UNEP (2011) Forests in a green economy, a synthesis. United Nations Environment Programme
Waddington H, Snilstveit B, Hombrados J, Vojtkova M, Phillips D, Davies P, White H (2014) Farmer field schools for improving farming practices and farmer outcomes: a systematic review. Campbell Syst Rev 2014:6. doi:10.4073/Csr.2014.6
Woolcock M (2013) Using case studies to explore the external validity of ‘Complex’ development interventions, Wider working paper no. 2013/096. UNU/WIDER, Helsinki
World Bank (2015) World development report 2015: mind, society, and behavior. World Bank, Washington, DC. doi:10.1596/978-1-4648-0342-0. License: Creative Commons Attribution CC BY 3.0 IGO
Wunder S, Angelsen A, Belcher B (2014) Forests, livelihoods, and conservation: broadening the empirical base. World Dev 64:S1–S11
Acknowledgements
The authors thank Raag Bhatia, research assistant, 3ie for excellent support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Annex I: Definition of Important Terms for Impact Evaluations
- Intervention:
-
Is used here interchangeably with the programme or the policy implemented or planned to increase the resilience and reduce the vulnerability of forests.
- Treatment group:
-
The stakeholders that receive or are beneficiaries of the programme or the intervention. Can be individuals, households, plots of land, communities, villages, districts etc.
- Comparison group:
-
Is the group that is typically compared with the treatment group and (at least for some time) does not get the treatment.
- Identification design:
-
Are methods that can help identify and help to attribute changes in measured effects to a programme/policy/project. Usually these require that implicit or explicit counterfactuals (also called comparisons) to understand what would not have occurred had the programme not occurred.
- Unit of assignment:
-
Is the level or unit at which a programme is implemented.
- Unit of measurement:
-
Is the unit for which measurement is undertaken and the units for which the measurement of the effects is important.
- Mixed methods:
-
Is the collection of methods that are interdisciplinary and use qualitative and quantitative methods in an integrated manner, informing each other and supporting and assisting each other to provide and supplement each other to provide a more wholistic understanding and measure of the effects of a policy, programme or project.
Annex II: List of Studies, Locations, Interventions and Identification Methods Reviewed in this Chapter
No. | Location of study (author) | The main intervention | Intended outcome | Identification method and data used |
|---|---|---|---|---|
1. | Ethiopia (Andersson et al. 2011) | A productive safety net (food for work) | Changes in livestock and tree holdings | Regression with propensity score matching. |
2. | Mexico (Alix-Garcia et al. 2013) | Conditional cash transfer programme | Deforestation | Method: Regression discontinuity along with IV discontinuity. |
3. | Ghana (Burwen and Levine 2012) | Distribution and use of improved cookstoves | Fuel use | A randomized trial. |
4. | Thailand (Cropper et al. 2001) | Road building and protected areas | Deforestation in protected areas | Instrumental variables |
5. | Senegal (Bensch and Peters 2011) | Improved cookstoves | Demand for charcoal | propensity score weighted regression approach. |
6. | India (Somanathan et al. 2005) | Managed forests | Deforestation measured by crown cover. | Difference in difference |
7. | Nepal (Tachibana and Adhikari 2009) | Community co-management of forests | Deforestation. | A switching regression model |
8. | Tanzania (Scullion et al. 2011) | Management of forests | Governance, forest conditions and local livelihoods | Quasi experimental methods |
9. | Nepal (Edmonds 2002) | Management of forests | Forest cover | Instrumental variables and regression discontinuity approach |
10. | Costa Rica (Arriagada et al. 2012) | Protected areas | Avoided deforestation | Mahalanobis weighting with propensity score matching. |
11. | Thailand (Sims 2008) | Wildlife Sanctuaries and National Parks | Forest clearing | Quasi-experimental matching techniques |
12. | Developing countries (Nelson and Chomitz 2009) | Tropical protected areas. | Deforestation fires which proxy deforestation | Differences in differences with matching |
13. | Uganda (Hafashimana et al. forthcoming) | Payment for ecosystem services | Â | The study is on-going (randomized assignment) |
14. | Mexico (Alix-Garcia et al. 2014) | Payment for ecosystem services | Forest cover and socio-economic outcomes. | Difference in difference with matching |
15. | Costa Rica (Arriagada et al. 2012) | Payment for ecosystem services | Forest cover | Difference in difference with matching |
16. | Costa Rica (Andam et al. 2007) | Payment for ecosystem services | Forest clearing | Difference in difference with matching |
Rights and permissions
Copyright information
© 2016 Springer Japan
About this chapter
Cite this chapter
Puri, J., Dhody, B. (2016). Missing the Forests for the Trees? Assessing the Use of Impact Evaluations in Forestry Programmes. In: Uitto, J., Shaw, R. (eds) Sustainable Development and Disaster Risk Reduction. Disaster Risk Reduction. Springer, Tokyo. https://doi.org/10.1007/978-4-431-55078-5_15
Download citation
DOI: https://doi.org/10.1007/978-4-431-55078-5_15
Publisher Name: Springer, Tokyo
Print ISBN: 978-4-431-55077-8
Online ISBN: 978-4-431-55078-5
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)