Policy Sciences

, Volume 51, Issue 3, pp 269–290 | Cite as

Neglected challenges to evidence-based policy-making: the problem of policy accumulation

  • Christian AdamEmail author
  • Yves Steinebach
  • Christoph Knill
Research Article


Claims for evidence-based policy-making are motivated by the assumption that if practitioners and scholars want to learn about effective policy design, they also can. This paper argues that this is becoming more and more challenging with the conventional approaches due to the accumulation of national policy portfolios, characterized by (a) a growing number of different policy targets and instruments, that (b) are often interdependent and (c) reformed in an uncontrolled way. These factors undermine our ability to accurately relate outcome changes to individual components within the respective policy mix. Therefore, policy accumulation becomes an additional source of the well-known ‘attribution problem’ in evaluation research. We argue that policy accumulation poses fundamental challenges to existing approaches of evidence-based policy-making. Moreover, these challenges are very likely to create a trade-off between the need for increasing methodological sophistication on one side, and the decreasing political impact of more fine-grained and conditional findings of evaluation results on the other.


Policy design Policy mixes Policy complexity Learning Performance management Evidence-based policy Outcome-based learning Policy evaluation Policy accumulation 



This research was funded by CONSENSUS project financed under the European Commission’s 7th Framework Programm.

Supplementary material

11077_2018_9318_MOESM1_ESM.docx (115 kb)
Supplementary material 1 (DOCX 115 kb)


  1. Adam, C., Hurka, S., & Knill, C. (2017a). Four styles of regulation and their implications for comparative policy analysis. Journal of Comparative Policy Analysis: Research and Practice, 19(4), 327–344.CrossRefGoogle Scholar
  2. Adam, C., Knill, C., & Fernandez-i-Marin, X. (2017b). Rule growth and government effectiveness: Why it takes the capacity to learn and coordinate to constrain rule growth. Policy Sciences, 50(2), 241–268.CrossRefGoogle Scholar
  3. Adam, C., & Raschzok, A. (2017). Cannabis policy and the uptake of treatment for cannabis-related problems. Drug and Alcohol Review, 36(2), 171–177.CrossRefGoogle Scholar
  4. Albalate, D., Bel, G., & Fageda, X. (2009). Privatization and regulatory reform of toll motorways in Europe. Governance, 22(2), 295–318.CrossRefGoogle Scholar
  5. Alford, J., & Head, B. W. (2017). Wicked and less wicked problems: a typology and a contingency framework. Policy and Society, 36(3), 397–413.CrossRefGoogle Scholar
  6. Anderson, D. M., & Rees, D. I. (2013). The legalization of recreational marijuana: How likely is the worst-case scenario? Journal of Policy Analysis and Management, 33(1), 221–232.CrossRefGoogle Scholar
  7. Babyak, M. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66, 411–421.Google Scholar
  8. Barnow, B. S. (2000). Exploring the relationship between performance management and program impact: A case study of the job training partnership act. Journal of Policy Analysis and Management, 19(1), 118–141.CrossRefGoogle Scholar
  9. Bassanini, A., & Duval, R. (2009). Unemployment, institutions, and reform complementarities: re-assessing the aggregate evidence for OECD countries. Oxford Review of Economic Policy, 25(1), 40–59.CrossRefGoogle Scholar
  10. Bates, M. A., & Glennerster, R. (2017). The generalizability puzzle. Stanford Social Innovation Review, 2017, 50–54.Google Scholar
  11. Bauer, M. W., & Knill, C. (2012). Understanding policy dismantling: An analytical framework. In M. W. Bauer, A. Jordan, C. Green-Pedersen, & A. Héritier (Eds.), Dismantling public policies: Preferences, strategies, and effects (pp. 30–51). Oxford: Oxford University Press.CrossRefGoogle Scholar
  12. Bianchi, C. (2016). Dynamic performance management. Cham: Springer.CrossRefGoogle Scholar
  13. Blonz, J. A., Vajjhala, S. P., & Safirova, V. (2008). Growing complexities: A cross-sector review of US biofuels policies and their interactions. Washington: Resources for the Future.Google Scholar
  14. Bovaird, T. (2012). Attributing outcomes to social policy interventions-, gold standard’ or ‘fool’s gold’ in public policy and management? Social Policy and Administration, 48(1), 1–23.CrossRefGoogle Scholar
  15. Brambor, T., Clark, W. R., & Golder, M. (2006). Understanding interaction models: Improving empirical analyses. Political Analysis, 14(1), 63–82.CrossRefGoogle Scholar
  16. Capano, G., & Lippi, A. (2017). How policy instruments are chosen: Patterns of decision makers’ choices. Policy Sciences, 50(2), 269–293.CrossRefGoogle Scholar
  17. Carley, S. (2009). State renewable energy electricity policies: An empirical evaluation of effectiveness. Energy Policy, 37(8), 3071–3081.CrossRefGoogle Scholar
  18. Davies, P. (2012). The state of evidence-based policy evaluation and its role in policy formation. National Institute Economic Review, 219, 41–52.CrossRefGoogle Scholar
  19. Del Río, P. (2014). On evaluating success in complex policy mixes: The case of renewable energy support schemes. Policy Sciences, 47(3), 267–287.CrossRefGoogle Scholar
  20. Dolowitz David, P., & Marsh, D. (2002). Learning from abroad: The role of policy transfer in contemporary policy-making. Governance, 13(1), 5–23.CrossRefGoogle Scholar
  21. Eliadis, F. P., Hill, M. M., & Howlett, M. (Eds.). (2005). Designing government: From instruments to governance. Montreal, CA: McGill Queens University Press.Google Scholar
  22. Favero, N., Meier, K. J., & O’Toole, L. J. (2016). Goals, trust, participation, and feedback: Linking internal management with performance outcomes. Journal of Public Administration Research and Theory, 26, 327–343.CrossRefGoogle Scholar
  23. Glennerster, R. (2012). The power of evidence: Improving the effectiveness of government by investing in more rigorous evaluation. National Institute Economic Review, 219, 4–14.CrossRefGoogle Scholar
  24. Goggin, M. L. (1986). The “too few cases/too many variables“ problem in implementation research. Western Political Quarterly, 39(2), 328–347.CrossRefGoogle Scholar
  25. Gunningham, N., & Grabosky, P. (1998). Smart regulation: Designing environmental policy. New York: Oxford University Press.Google Scholar
  26. Gunningham, N., & Sinclair, D. (1999). Regulatory pluralism: Designing policy mixes for environmental protection. Law and Policy, 21(1), 49–76.CrossRefGoogle Scholar
  27. Head, B. W. (2016). Toward more “evidence-informed” policy making? Public Administration Review, 76(3), 472–484.CrossRefGoogle Scholar
  28. Heinrich, C. J. (2002). Outcomes-based performance management in the public sector: Implications for government accountability and effectiveness. Public Administration Review, 62, 712–725.CrossRefGoogle Scholar
  29. Horn, G.-A., & Logeay, C. (2010). Erfolg oder Misserfolg? Die Arbeitsmarktreformen im Rahmen der Agenda. In G. Bäcker, S. Lehndorff, & C. Weinkopf (Eds.), Den Arbeitsmarkt verstehen, um ihn zu gestalten. Wiesbaden: Springer VS.Google Scholar
  30. Howlett, M., & del Rio, P. (2015). The parameters of policy portfolios: Verticality and horizontality in design spaces and their consequences for policy mix formulation. Environment and Planning C: Government and Policy, 33(5), 1233–1245.CrossRefGoogle Scholar
  31. Howlett, M., & Lejano, R. P. (2013). Tales from the crypt: The rise and fall (and re-birth?) of policy design studies. Administration and Society, 45(3), 356–380.CrossRefGoogle Scholar
  32. Howlett, M., Ramesh, M., & Perl, A. (2009). Studying public policy: Policy cycles and policy subsystems. New York: Oxford University Press.Google Scholar
  33. Howlett, M., & Rayner, J. (2013). Patching vs packaging in policy formulation: Assessing policy portfolio design. Politics and Governance, 1(2), 170–182.CrossRefGoogle Scholar
  34. Kaplan, R. S., & Norton, D. P. (2004). Strategy maps: Converting intangible assets into tangible outcomes. Harvard: Harvard Business Press.Google Scholar
  35. Knill, C., Schulze, K., & Tosun, J. (2012). Regulatory policy outputs and impacts: Exploring a complex relationship. Regulation and Governance, 6(4), 427–444.CrossRefGoogle Scholar
  36. Linder, S. H., & Peters, B. G. (1989). Instruments of government: Perceptions and contexts. Journal of Public Policy, 9(1), 35–58.CrossRefGoogle Scholar
  37. Majone, G. (1989). Evidence, argument, and persuasion in the policy process. New Haven, London: Yale University Press.Google Scholar
  38. May, P. J. (1991). Reconsidering policy design: Policies and publics. Journal of Public Policy, 11(2), 187–206.CrossRefGoogle Scholar
  39. Mayne, J. (2007). Challenges and lessons in implementing results-based management. Evaluation, 13(1), 87–109.CrossRefGoogle Scholar
  40. McBeath, B., & Meezan, W. (2010). Governance in motion: Service provision and child welfare outcomes in a performance-based, managed care contracting environment. Journal of Public Administration Research and Theory, 20, 101–123.CrossRefGoogle Scholar
  41. McConnell, A. (2010). Policy success, policy failure and grey areas in-between. Journal of Public Policy, 30(3), 345–362.CrossRefGoogle Scholar
  42. Mettler, S. (2016). The policyscape and the challenges of contemporary politics to policy maintenance. Perspectives on Politics, 14(2), 369–390.CrossRefGoogle Scholar
  43. Miller, J. H., & Page, S. E. (2009). Complex adaptive systems: An introduction to computational models of social life: an introduction to computational models of social life. Princeton: Princeton University Press.CrossRefGoogle Scholar
  44. Möhring, K. (2012). The fixed effects approach as alternative to multilevel models for cross-national analyses. GK SOCLIFE Working Paper Series, pp 1–15.Google Scholar
  45. Moulton, S., & Sandfort, J. R. (2017). The strategic action field framework for policy implementation research. Policy Science Journal, 45(1), 144–169.Google Scholar
  46. Moynihan, D. P. (2005). Goal-based learning and the future of performance management. Public Administration Review, 65(2), 203–216.CrossRefGoogle Scholar
  47. O’Toole, L. J., & Meier, K. J. (2014). Public management, context, and performance: In quest of a more general theory. Journal of Public Administration Research and Theory, 25(1), 237–256.CrossRefGoogle Scholar
  48. Pacula, R. L., Powell, D., Heaton, P., & Sevigny, E. L. (2015). Assessing the effects of medical marijuana laws on marijuana use: The devil is in the details. Journal of Policy Analysis and Management (the Journal of the Association for Public Policy Analysis and Management), 34(1), 7–31.CrossRefGoogle Scholar
  49. Piotrowski, S. J., & Rosenbloom, D. H. (2002). Nonmission-based values in results-oriented public management: The case of freedom of information. Public Administration Review, 62(6), 643–657.CrossRefGoogle Scholar
  50. Pirog, M. A., & Ziol-Guest, K. M. (2006). Child support enforcement: Programs and policies, impacts and questions. Journal of Policy Analysis and Management, 25(4), 943–990.CrossRefGoogle Scholar
  51. Pollitt, C. (2011). Performance blight and the tyranny of light? Performance blight and the tyranny of light? Accountability in advanced performance measurement regimes. In M. J. Dubnick & H. G. Frederickson (Eds.), Accountable governance: Problems and promises (pp. 81–98). M.E. Sharpe: Armonk.Google Scholar
  52. Radin, B. A. (2000). The government performance and results act and the tradition of federal management reform: Square pegs in round holes? Journal of Public Administration Research and Theory, 10, 111–135.CrossRefGoogle Scholar
  53. Radin, B. A. (2006). Challenging the performance movement: Accountability, complexity, and democractic values. Washington, D.C.: Georgetown University Press.Google Scholar
  54. Radin, B. A. (2009). What can we expect from performance measurement activities? Journal of Policy Analysis and Management, 28, 505–512.CrossRefGoogle Scholar
  55. Rinne, U., & Zimmermann, K. (2012). Another economic miracle? The German labor market and the Great Recession. IZA Journal of Labor Policy, 1(1), 1–21.CrossRefGoogle Scholar
  56. Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169.CrossRefGoogle Scholar
  57. Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2003). Evaluation: A systematic approach. Thousand Oaks: SAGE Publications.Google Scholar
  58. Schickler, E. (2001). Disjointed pluralism: Institutional innovation and the development of the US congress. Princeton: Prrinceton University Press.Google Scholar
  59. Schlaufer, C., Stucki, I., & Sager, F. (2018). The political use of evidence and its contribution to democratic discourse. Public Administration Review. Scholar
  60. Schneider, A. (2012). Policy design and transfer. In E. Araral, S. Fritzen, M. Howlett, M. Ramesh, & X. Wu (Eds.), Routledge handbook of public policy (pp. 217–228). Abingdon: Routledge.Google Scholar
  61. Thelen, K. (1999). Historical institutionalism in comparative politics’. Annual Review of Political Science, 2, 369–414.CrossRefGoogle Scholar
  62. Thelen, K. (2004). How institutions evolve: The political economy of skills in Germany, Britain, the United States, and Japan. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  63. Van Dooren, W., & Van de Walle, S. (2016). Performance information in the public sector: How it is used. Berlin: Springer.Google Scholar
  64. Van Thiel, S., & Leeuw, F. L. (2002). The performance paradox in the public sector. Public Performance and Management Review, 25, 267–281.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.LMU MunichMunichGermany

Personalised recommendations