Modeling Mitigation and Adaptation Policies to Predict Their Effectiveness: The Limits of Randomized Controlled Trials

  • Alexandre Marcellesi
  • Nancy Cartwright


Policies to combat climate change should be supported by evidence regarding their effectiveness. But what kind of evidence is that? And what tools should one use to gather such evidence? Many argue that randomized controlled trials (RCTs) are the gold standard when it comes to evaluating the effects of policies. As a result, there has been a push for climate change policies to be evaluated using RCTs. We argue that this push is misguided. After explaining why RCTs are thought to be the gold standard, we use examples of mitigation and adaptation policies to show that RCTs provide, at best, one piece of the evidential puzzle one needs to assemble for well-supported decisions regarding climate change policies.



Both authors would like to thank the Templeton Foundation’s project ‘God’s Order, Man’s Order and the Order of Nature’, the UCSD Faculty Senate, and the AHRC project ‘Choices of evidence: tacit philosophical assumptions in debates on evidence-based practice in children’s welfare services’ for support for the research and writing of this chapter. Nancy Cartwright would in addition like to thank the Grantham Research Institute on Climate Change and the Environment at LSE.


  1. Alix-Garcia, Jennifer, Alain De Janvry, Elisabeth Sadoulet, and Juan Manuel. 2009. Lessons Learned from Mexico’s Payment for Environmental Services Program. In Payment for Environmental Services in Agricultural Landscapes, 163–188. Rome: Springer.CrossRefGoogle Scholar
  2. Balshem, Howard, Mark Helfand, Holger J. Schünemann, Andrew D. Oxman, Regina Kunz, Jan Brozek, Gunn E. Vist, et al. 2011. GRADE Guidelines: 3. Rating the Quality of Evidence. Journal of Clinical Epidemiology 64 (4): 401–406. Scholar
  3. Banerjee, Abhijit, and Esther Duflo. 2012. Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty. New York: PublicAffairs.Google Scholar
  4. Cartwright, Nancy. 2010. What Are Randomised Controlled Trials Good For? Philosophical Studies 147 (1): 59–70.CrossRefGoogle Scholar
  5. ———. Forthcoming. Will Your Policy Work? Experiments vs. Models. In The Experimental Side of Modeling, ed. I.F. Peschard and B.C. van Frassen.Google Scholar
  6. Cartwright, Nancy, and Jeremy Hardie. 2012. Evidence-Based Policy: A Practical Guide to Doing It Better. Oxford: Oxford University Press.CrossRefGoogle Scholar
  7. Coase, Ronald Harry. 1960. The Problem of Social Cost. The Journal of Law and Economics 3: 1–44.CrossRefGoogle Scholar
  8. Cole, Shawn, Xavier Giné, Jeremy Tobacman, Petia Topalova, Robert Townsend, and James Vickery. 2013. Barriers to Household Risk Management: Evidence from India. American Economic Journal: Applied Economics 5 (1): 104–135.Google Scholar
  9. de Nicola, Francesca. 2015. The Impact of Weather Insurance on Consumption, Investment, and Welfare. Quantitative Economics 6 (3): 637–661.CrossRefGoogle Scholar
  10. der Werf, Van, R. Guido, Douglas C. Morton, Ruth S. DeFries, Jos G.J. Olivier, Prasad S. Kasibhatla, Robert B. Jackson, G. James Collatz, and James T. Randerson. 2009. CO2 Emissions from Forest Loss. Nature Geoscience 2: 737–738.CrossRefGoogle Scholar
  11. Duflo, Esther, and Michael Kremer. 2005. Use of Randomization in the Evaluation of Development Effectiveness. Evaluating Development Effectiveness 7: 205–231.Google Scholar
  12. Engel, Stefanie, Stefano Pagiola, and Sven Wunder. 2008. Designing Payments for Environmental Services in Theory and Practice: An Overview of the Issues. Ecological Economics 65 (4): 663–674.CrossRefGoogle Scholar
  13. GEF. 2007. Adaptation to Climate Change through Effective Water Governance in Ecuador. Project Executive Summary.Google Scholar
  14. ———. 2010. Developing an Experimental Methodology for Testing the Effectiveness of Payments for Ecosystem Services to Enhance Conservation in Productive Landscapes in Uganda (Request for CEO Endorsement/Approval). Washington, DC: Global Environment Facility.Google Scholar
  15. ———. 2016. Developing an Experimental Methodology for Testing the Effectiveness of Payments for Ecosystem Services to Enhance Conservation in Productive Landscapes in Uganda. Washington, DC: Global Environmental Facility, June 4.
  16. Giné, Xavier, and Dean Yang. 2009. Insurance, Credit, and Technology Adoption: Field Experimental Evidence from Malawi. Journal of Development Economics 89 (1): 1–11.CrossRefGoogle Scholar
  17. Heckman, James J. 1991. Randomization and Social Policy Evaluation. National Bureau of Economic Research Cambridge, MA.
  18. Holland, Paul W. 1986. Statistics and Causal Inference. Journal of the American Statistical Association 81 (396): 945–960.CrossRefGoogle Scholar
  19. Holland, Paul W., and Donald B. Rubin. 1987. Causal Inference in Retrospective Studies. ETS Research Report Series 1987 (1): 203–231. Scholar
  20. IPCC. 2007a. Climate Change 2007: The Physical Science Basis. New York: Intergovernmental Panel on Climate Change.Google Scholar
  21. ———. 2007b. Impacts, Adaptation and Vulnerability. New York: Intergovernmental Panel on Climate Change.Google Scholar
  22. ———. 2012. Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation: Special Report of the Intergovernmental Panel on Climate Change. New York: Cambridge University Press.Google Scholar
  23. Jayachandran, S. 2013a. Evaluating a Payments for Ecosystem Services Program in Uganda, April 22.
  24. Jayachandran, Seema. 2013b. Liquidity Constraints and Deforestation: The Limitations of Payments for Ecosystem Services. The American Economic Review 103 (3): 309–313.CrossRefGoogle Scholar
  25. Jones, Harry. 2009. The ‘Gold Standard’ is Not a Silver Bullet for Evaluation. Overseas Development Institute London.
  26. Mackie, J. 1965. Causes and Conditions. American Philosophical Quarterly 2: 245–264.Google Scholar
  27. MEA. 2005. Ecosystems and Human Well-Being: Current State and Trends. Washington, DC: Millennium Ecosystem Assessment.Google Scholar
  28. Muradian, R., E. Corbera, U. Pascual, N. Kosoy, and P.H. May. 2010. Reconciling Theory and Practice: An Alternative Conceptual Framework for Understanding Payments for Environmental Services. Ecological Economics 69 (6): 1202–1208.CrossRefGoogle Scholar
  29. Parker, Susan W., and Graciela M. Teruel. 2005. Randomization and Social Program Evaluation: The Case of Progress. The Annals of the American Academy of Political and Social Science 599 (1): 199–219. Scholar
  30. Pattanayak, Subhrendu K., Sven Wunder, and Paul J. Ferraro. 2010. Show Me the Money: Do Payments Supply Environmental Services in Developing Countries? Review of Environmental Economics and Policy 4 (2): 254–274. Scholar
  31. Pirard, Romain, Raphaël Billé, and Thomas Sembrés. 2010. Questioning the Theory of Payments for Ecosystem Services (PES) in Light of Emerging Experience and Plausible Developments. Institut Pour Le Développement Durable et Les Relations Internationales (IDDRI), Analyses, 4, no. 2010/06/10, 5–22.Google Scholar
  32. Prowse, Martin, and Birte Snilstveit. 2010. Impact Evaluation and Interventions to Address Climate Change: A Scoping Study. Journal of Development Effectiveness 2 (2): 228–262.CrossRefGoogle Scholar
  33. Ravallion, Martin, et al. 2009. Should the Randomistas Rule? Economists’ Voice 6 (2): 1–5.Google Scholar
  34. Rubin, Donald B. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology 66 (5): 688–701.CrossRefGoogle Scholar
  35. ———. 2008. Comment: The Design and Analysis of Gold Standard Randomized Experiments. Journal of the American Statistical Association 103: 1350–1353.CrossRefGoogle Scholar
  36. STAP. 2010. Payments for Environmental Services and the Global Environment Facility: A STAP Advisory Document. Washington, DC: Scientific and Technical Advisory Panel. Scholar
  37. STAP. n.d. Payments for Environmental Services and the Global Environment Facility: A STAP Advisory Document. Washington, DC: Scientific and Technical Advisory Panel.Google Scholar

Copyright information

© The Author(s) 2018

Authors and Affiliations

  • Alexandre Marcellesi
    • 1
  • Nancy Cartwright
    • 2
    • 3
  1. 1.New York University School of LawNew YorkUSA
  2. 2.Department of PhilosophyDurham UniversityDurhamUK
  3. 3.University of CaliforniaSan DiegoUSA

Personalised recommendations