Skip to main content
SpringerLink
Log in

A Challenge for Evidence-Based Policy

  • Original Paper
  • Published:
Axiomathes Aims and scope Submit manuscript

Abstract

Evidence-based policy has support in many areas of government and in public affairs more generally. In this paper we outline what evidence-based policy is, then we discuss its strengths and weaknesses. In particular, we argue that it faces a serious challenge to provide a plausible, over-arching account of evidence. We contrast evidence-based policy with evidence-based medicine, especially the role of evidence in assessing the effectiveness of medicines. The evidence required for policy decisions does not easily lend itself to randomized controlled trials (the “gold standard” in evidence-based medicine), nor, for that matter, being listed in a single all-purpose hierarchy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Head (2010) provides a good overview (with an extensive reference list). Davies et al. (2000a) is an earlier, comprehensive treatment of evidence-based policy. Like this literature, this paper focuses on the role and merits of specific forms of evidence for informing policy. An example of some of the broader discussions regarding the role of science and values in policy can be found in Douglas (2009).

  2. EBM puts forward a number of hierarchies for different medical questions. We follow the literature on this topic and focus on the hierarchy provided for therapeutic decisions.

  3. As Worrall (2002) notes, randomization does not ensure all possible confounding factors are equally balanced between the experimental groups. Confounding due to an unlucky consequence of random allocation is, however, a different type of error to the selection bias that can occur in observational studies.

  4. Macintyre et al. (2001) provide examples from health policy where decisions that have ignored relevant empirical evidence have led to harm.

  5. Incremental or mixed incremental-rational models of the policy process emphasize the incremental and disjointed nature of policy progress. While evidence and arguments about evidence play a more diffuse role in these models, evidence (broadly construed) remains central (Nutley and Webb 2000, p. 28).

  6. Rawlins (2008) provides a good overview of some of the criticisms that have been raised from within medicine. For philosophical discussion, see Bluhm (2005), Cartwright (2007), Clarke et al. (2013, 2014), Grossman and Mackenzie (2005), La Caze (2008, 2009), Osimani (2014), and Worrall (2007). Cartwright (2009) and Roush (2009) discuss the use of randomized studies in policy and Montuschi (2009) discusses some of the broader problems of evidence in policy.

  7. The importance of attending to the human elements of health are emphasised by proponents of narrative medicine. See Greenhalgh (2012) for a discussion on the importance of avoiding the reduction medical decision-making to “mathematical estimates of the change of benefit and the risk of harm derived from high-quality research” (Greenhalgh 2012, p. 94).

  8. See for example, Macintyre et al. (2001) and Society for Prevention Research (2004), and for a more general discussion, see Davies et al. (2000a).

  9. Davies et al. (2000b) and Head (2008) discuss the difficulties of conducting randomized studies in some areas of public policy.

  10. See also Cartwright and Hardie (2012).

  11. See Steel (2004) for a discussion of social mechanisms.

  12. See, for example, the recent policy decision by the Australian Government’s Biosecurity Australia to allow the importation of Cavendish bananas from the Philippines. The relevant import risk analysis report and policy decisions are available from the Australian Government’s Department of Agriculture Fisheries and Forestry website devoted to the matter: http://www.daff.gov.au/ba/ira/final-plant/banana-philippines.

  13. Lindenmayer et al. (2012) provides examples of how carbon-trading policy can have unwanted downstream effects if the game-theoretic structure of the problem is not appreciated and Colyvan et al. (2011) discuss the importance of games against nature and adaptive management in conservation management decisions.

  14. This might not be so different from some areas of medicine where organisms are known to have responded to continued treatment in ways not anticipated by initial randomized controlled trials. Think, for example, of the way antibiotic-resistant Staphylococci evolved in response to clinical interventions of sequential narrow spectrum antibiotics. While game theory is usually, and most obviously, applicable to sentient and rational agents, many non-sentient systems behave as though they were agents responding in the game in question. See Skyrms (2004) for more on the evolution of cooperation in non-sentient cases via evolutionary game theory. Examples such as the antibiotic-resistant Staphylococci suggest that faith in randomized-controlled trials, even in medicine, is problematic.

  15. This advice is exemplified in Cartwright and Hardie (2012).

References

  • Angrist JD, Pischke JS (2010) The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. J Econ Perspect 24(2):3–30

    Article  Google Scholar 

  • Bluhm R (2005) From hierarchy to network: a richer view of evidence for evidence-based medicine. Perspect Biol Med 48(4):535–547

    Article  Google Scholar 

  • Bohrnstedt GW, Stecher BM (eds) (2002) What we have learned about class size reduction. California Department of Education, Sacremento

    Google Scholar 

  • Bornstein SR, Licinio J (2011) Improving the efficacy of translational medicine by optimally integrating health care, academia and industry. Nat Med 12:1567–1569

    Article  Google Scholar 

  • Cartwright N (2007) Are RCTs the gold standard? BioSocieties 2:11–20

    Article  Google Scholar 

  • Cartwright N (2009) Evidence-based policy: What’s to be done about relevance? Philos Stud 143(1):127–136

    Article  Google Scholar 

  • Cartwright N (2011) Predicting what will happen when we act. What counts for warrant? Prev Med 53(4–5):221–224

    Article  Google Scholar 

  • Cartwright N, Hardie J (2012) Evidence-based policy: a practical guide to doing it better. Oxford University Press, Oxford

    Book  Google Scholar 

  • Clarke B, Gillies D, Illari P, Russo F, Williamson J (2013) The evidence that evidence-based medicine omits. Prev Med 57:745–747

    Article  Google Scholar 

  • Clarke B, Gillies D, Illari P, Russo F, Williamson J (2014) Mechanisms and the evidence hierarchy. Topoi 33(2):339–360

    Article  Google Scholar 

  • Colyvan M, Justus J, Regan HM (2011) The conservation game. Biol Conserv 144(4):1246–1253

    Article  Google Scholar 

  • Davies HTO, Nutley SM, Smith PC (eds) (2000a) What works?: evidence based policy and practice in public services. Policy Press, Bristol

    Google Scholar 

  • Davies HTO, Nutley SM, Tilley N (2000b) Debates on the role of experimentation. In: Davies HTO, Nutley SM, Smith PC (eds) What works?: evidence based policy and practice in public services. Policy Press, Bristol, pp 251–276

    Google Scholar 

  • Douglas H (2009) Science, policy, and the value-free ideal. University of Pittsburgh Press, Pittsburgh

    Google Scholar 

  • Druker B, Talpaz M, Resta D et al (2001) Efficacy and safety of a specific inhibitor of the BCR-ABL tyrosine kinase in chronic myeloid leukemia. New Engl J Med 344(14):1031–1037

    Article  Google Scholar 

  • Evidence-Based Medicine Working Group (1992) Evidence-based medicine: a new approach to teaching the practice of medicine. J Am Med Assoc 268(17):2420–2425

    Article  Google Scholar 

  • Glass GV, Cahen LS, Smith ML, Filby NN (1982) School class size: research and policy. Sage Publications, Beverly Hills

    Google Scholar 

  • Greenhalgh T (2012) Why do we always end up here? Evidence-based medicine’s conceptual cul-de-sacs and some off-road alternative routes. J Prim Health Care 4(2):92–97

    Google Scholar 

  • Grossman J, Mackenzie FJ (2005) The randomized controlled trial: gold standard, or merely standard? Perspect Biol Med 48(4):516–534

    Article  Google Scholar 

  • Head BW (2008) Three lenses of evidence-based policy. Aust J Public Adm 67(1):1–11

    Article  Google Scholar 

  • Head BW (2010) Reconsidering evidence-based policy: key issues and challenges. Policy Soc 29(2):77–94

    Article  Google Scholar 

  • Hutton J, Smith PC (2000) Non-experimental quantitative methods. In: Davies HTO, Nutley SM, Smith PC (eds) What works?: evidence based policy and practice in public services. Policy Press, Bristol, pp 277–290

    Google Scholar 

  • La Caze A (2008) Evidence based medicine can’t be…. Soc Epistemol 22(4):353–370

    Article  Google Scholar 

  • La Caze A (2009) Evidence based medicine must be …. J Philos Med 34:509–527

    Article  Google Scholar 

  • Leamer EE (1983) Let’s take the con out of econometrics. Am Econ Rev 73(1):31–43

    Google Scholar 

  • Lindenmayer DB, Hulvey KB, Hobbs RJ, Colyvan M, Felton A, Possingham H, Steffen W, Wilson K, Youngentob K, Gibbons P (2012) Avoiding bio-perversity from carbon sequestration solutions. Conserv Lett 5(1):28–36

    Article  Google Scholar 

  • Lyseng-Williamson K, Jarvis B (2001) Imatinib. Drugs 61(12):1765–1774

    Article  Google Scholar 

  • Macintyre S, Chalmers I, Horton R, Smith R (2001) Using evidence to inform health policy: case study. Br Med J 322(7280):222–225

    Article  Google Scholar 

  • Montuschi E (2009) Questions of evidence in evidence-based policy. Axiomathes 19(4):425–439

    Article  Google Scholar 

  • Mosteller F (1995) The Tennessee study of class size in the early school grades. Future Child 5(2):113–127

    Article  Google Scholar 

  • Nutley SM, Webb J (2000) Evidence and the policy process. In: Davies HTO, Nutley SM, Smith PC (eds) What works?: evidence based policy and practice in public services. Policy Press, Bristol, pp 13–42

    Google Scholar 

  • O’Brien S, Guilhot F, Larson R, Gathmann I, Baccarani M, Cervantes F et al (2003) Imatinib compared with interferon and low-dose cytarabine for newly diagnosed chronic-phase chronic myeloid leukemia. New Engl J Med 348(11):994–1004

    Article  Google Scholar 

  • Osimani B (2014) Hunting side effects and explaining them: Should we reverse evidence hierarchies upside down? Topoi 33(2):295–312

    Article  Google Scholar 

  • Rawlins MD (2008) De testimonio: on the use of evidence for decisions about the use of therapeutic interventions. Lancet 372(9656):2152–2161

    Article  Google Scholar 

  • Roush S (2009) Randomized controlled trials and the flow of information: comment on Cartwright. Philos Stud 143(1):137–145

    Article  Google Scholar 

  • Skyrms B (2004) The stag hunt and the evolution of the social contract. Cambridge University Press, Cambridge

    Google Scholar 

  • Society for Prevention Research (2004) Standards of evidence: criteria for efficacy, effectiveness and dissemination. http://www.preventionresearch.org/advocacy/#SofE. Accessed 25 Nov 2013

  • Steel D (2004) Social mechanisms and causal inference. Philos Soc Sci 34(1):55–78

    Article  Google Scholar 

  • Walters CJ (1986) Adaptive management of renewable resources. MacMillan, New York

    Google Scholar 

  • Worrall J (2002) What evidence in evidence-based medicine? Philos Sci 69:S316–S330

    Article  Google Scholar 

  • Worrall J (2007) Evidence in medicine and evidence-based medicine. Philos Compass 2(6):981–1022

    Article  Google Scholar 

Download references

Acknowledgments

An early version of this paper was presented at the 2009 Sydney-Tilburg Philosophy of Science Conference: ‘Evidence, Science, and Public Policy’ held at The University of Sydney, 26–28 March 2009. We’d like to thank the audience at that conference for fruitful discussion and several very helpful suggestions. We are also grateful to the referees of this journal for helpful suggestions.

Author information

Authors and Affiliations

  1. School of Pharmacy, University of Queensland, Brisbane, QLD, 4072, Australia

    Adam La Caze

  2. Sydney Centre for the Foundations of Science, University of Sydney, Sydney, NSW, 2006, Australia

    Mark Colyvan

Authors
  1. Adam La Caze
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Colyvan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam La Caze.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

La Caze, A., Colyvan, M. A Challenge for Evidence-Based Policy. Axiomathes 27, 1–13 (2017). https://doi.org/10.1007/s10516-016-9291-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10516-016-9291-5

Keywords

Search

Navigation