Skip to main content
SpringerLink
Log in

Neglected challenges to evidence-based policy-making: the problem of policy accumulation

  • Research Article
  • Published:
Policy Sciences Aims and scope Submit manuscript

Abstract

Claims for evidence-based policy-making are motivated by the assumption that if practitioners and scholars want to learn about effective policy design, they also can. This paper argues that this is becoming more and more challenging with the conventional approaches due to the accumulation of national policy portfolios, characterized by (a) a growing number of different policy targets and instruments, that (b) are often interdependent and (c) reformed in an uncontrolled way. These factors undermine our ability to accurately relate outcome changes to individual components within the respective policy mix. Therefore, policy accumulation becomes an additional source of the well-known ‘attribution problem’ in evaluation research. We argue that policy accumulation poses fundamental challenges to existing approaches of evidence-based policy-making. Moreover, these challenges are very likely to create a trade-off between the need for increasing methodological sophistication on one side, and the decreasing political impact of more fine-grained and conditional findings of evaluation results on the other.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  • Adam, C., Hurka, S., & Knill, C. (2017a). Four styles of regulation and their implications for comparative policy analysis. Journal of Comparative Policy Analysis: Research and Practice, 19(4), 327–344.

    Article  Google Scholar 

  • Adam, C., Knill, C., & Fernandez-i-Marin, X. (2017b). Rule growth and government effectiveness: Why it takes the capacity to learn and coordinate to constrain rule growth. Policy Sciences, 50(2), 241–268.

    Article  Google Scholar 

  • Adam, C., & Raschzok, A. (2017). Cannabis policy and the uptake of treatment for cannabis-related problems. Drug and Alcohol Review, 36(2), 171–177.

    Article  Google Scholar 

  • Albalate, D., Bel, G., & Fageda, X. (2009). Privatization and regulatory reform of toll motorways in Europe. Governance, 22(2), 295–318.

    Article  Google Scholar 

  • Alford, J., & Head, B. W. (2017). Wicked and less wicked problems: a typology and a contingency framework. Policy and Society, 36(3), 397–413.

    Article  Google Scholar 

  • Anderson, D. M., & Rees, D. I. (2013). The legalization of recreational marijuana: How likely is the worst-case scenario? Journal of Policy Analysis and Management, 33(1), 221–232.

    Article  Google Scholar 

  • Babyak, M. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66, 411–421.

    Google Scholar 

  • Barnow, B. S. (2000). Exploring the relationship between performance management and program impact: A case study of the job training partnership act. Journal of Policy Analysis and Management, 19(1), 118–141.

    Article  Google Scholar 

  • Bassanini, A., & Duval, R. (2009). Unemployment, institutions, and reform complementarities: re-assessing the aggregate evidence for OECD countries. Oxford Review of Economic Policy, 25(1), 40–59.

    Article  Google Scholar 

  • Bates, M. A., & Glennerster, R. (2017). The generalizability puzzle. Stanford Social Innovation Review, 2017, 50–54.

    Google Scholar 

  • Bauer, M. W., & Knill, C. (2012). Understanding policy dismantling: An analytical framework. In M. W. Bauer, A. Jordan, C. Green-Pedersen, & A. Héritier (Eds.), Dismantling public policies: Preferences, strategies, and effects (pp. 30–51). Oxford: Oxford University Press.

    Chapter  Google Scholar 

  • Bianchi, C. (2016). Dynamic performance management. Cham: Springer.

    Book  Google Scholar 

  • Blonz, J. A., Vajjhala, S. P., & Safirova, V. (2008). Growing complexities: A cross-sector review of US biofuels policies and their interactions. Washington: Resources for the Future.

    Google Scholar 

  • Bovaird, T. (2012). Attributing outcomes to social policy interventions-, gold standard’ or ‘fool’s gold’ in public policy and management? Social Policy and Administration, 48(1), 1–23.

    Article  Google Scholar 

  • Brambor, T., Clark, W. R., & Golder, M. (2006). Understanding interaction models: Improving empirical analyses. Political Analysis, 14(1), 63–82.

    Article  Google Scholar 

  • Capano, G., & Lippi, A. (2017). How policy instruments are chosen: Patterns of decision makers’ choices. Policy Sciences, 50(2), 269–293.

    Article  Google Scholar 

  • Carley, S. (2009). State renewable energy electricity policies: An empirical evaluation of effectiveness. Energy Policy, 37(8), 3071–3081.

    Article  Google Scholar 

  • Davies, P. (2012). The state of evidence-based policy evaluation and its role in policy formation. National Institute Economic Review, 219, 41–52.

    Article  Google Scholar 

  • Del Río, P. (2014). On evaluating success in complex policy mixes: The case of renewable energy support schemes. Policy Sciences, 47(3), 267–287.

    Article  Google Scholar 

  • Dolowitz David, P., & Marsh, D. (2002). Learning from abroad: The role of policy transfer in contemporary policy-making. Governance, 13(1), 5–23.

    Article  Google Scholar 

  • Eliadis, F. P., Hill, M. M., & Howlett, M. (Eds.). (2005). Designing government: From instruments to governance. Montreal, CA: McGill Queens University Press.

    Google Scholar 

  • Favero, N., Meier, K. J., & O’Toole, L. J. (2016). Goals, trust, participation, and feedback: Linking internal management with performance outcomes. Journal of Public Administration Research and Theory, 26, 327–343.

    Article  Google Scholar 

  • Glennerster, R. (2012). The power of evidence: Improving the effectiveness of government by investing in more rigorous evaluation. National Institute Economic Review, 219, 4–14.

    Article  Google Scholar 

  • Goggin, M. L. (1986). The “too few cases/too many variables“ problem in implementation research. Western Political Quarterly, 39(2), 328–347.

    Article  Google Scholar 

  • Gunningham, N., & Grabosky, P. (1998). Smart regulation: Designing environmental policy. New York: Oxford University Press.

    Google Scholar 

  • Gunningham, N., & Sinclair, D. (1999). Regulatory pluralism: Designing policy mixes for environmental protection. Law and Policy, 21(1), 49–76.

    Article  Google Scholar 

  • Head, B. W. (2016). Toward more “evidence-informed” policy making? Public Administration Review, 76(3), 472–484.

    Article  Google Scholar 

  • Heinrich, C. J. (2002). Outcomes-based performance management in the public sector: Implications for government accountability and effectiveness. Public Administration Review, 62, 712–725.

    Article  Google Scholar 

  • Horn, G.-A., & Logeay, C. (2010). Erfolg oder Misserfolg? Die Arbeitsmarktreformen im Rahmen der Agenda. In G. Bäcker, S. Lehndorff, & C. Weinkopf (Eds.), Den Arbeitsmarkt verstehen, um ihn zu gestalten. Wiesbaden: Springer VS.

    Google Scholar 

  • Howlett, M., & del Rio, P. (2015). The parameters of policy portfolios: Verticality and horizontality in design spaces and their consequences for policy mix formulation. Environment and Planning C: Government and Policy, 33(5), 1233–1245.

    Article  Google Scholar 

  • Howlett, M., & Lejano, R. P. (2013). Tales from the crypt: The rise and fall (and re-birth?) of policy design studies. Administration and Society, 45(3), 356–380.

    Article  Google Scholar 

  • Howlett, M., Ramesh, M., & Perl, A. (2009). Studying public policy: Policy cycles and policy subsystems. New York: Oxford University Press.

    Google Scholar 

  • Howlett, M., & Rayner, J. (2013). Patching vs packaging in policy formulation: Assessing policy portfolio design. Politics and Governance, 1(2), 170–182.

    Article  Google Scholar 

  • Kaplan, R. S., & Norton, D. P. (2004). Strategy maps: Converting intangible assets into tangible outcomes. Harvard: Harvard Business Press.

    Google Scholar 

  • Knill, C., Schulze, K., & Tosun, J. (2012). Regulatory policy outputs and impacts: Exploring a complex relationship. Regulation and Governance, 6(4), 427–444.

    Article  Google Scholar 

  • Linder, S. H., & Peters, B. G. (1989). Instruments of government: Perceptions and contexts. Journal of Public Policy, 9(1), 35–58.

    Article  Google Scholar 

  • Majone, G. (1989). Evidence, argument, and persuasion in the policy process. New Haven, London: Yale University Press.

    Google Scholar 

  • May, P. J. (1991). Reconsidering policy design: Policies and publics. Journal of Public Policy, 11(2), 187–206.

    Article  Google Scholar 

  • Mayne, J. (2007). Challenges and lessons in implementing results-based management. Evaluation, 13(1), 87–109.

    Article  Google Scholar 

  • McBeath, B., & Meezan, W. (2010). Governance in motion: Service provision and child welfare outcomes in a performance-based, managed care contracting environment. Journal of Public Administration Research and Theory, 20, 101–123.

    Article  Google Scholar 

  • McConnell, A. (2010). Policy success, policy failure and grey areas in-between. Journal of Public Policy, 30(3), 345–362.

    Article  Google Scholar 

  • Mettler, S. (2016). The policyscape and the challenges of contemporary politics to policy maintenance. Perspectives on Politics, 14(2), 369–390.

    Article  Google Scholar 

  • Miller, J. H., & Page, S. E. (2009). Complex adaptive systems: An introduction to computational models of social life: an introduction to computational models of social life. Princeton: Princeton University Press.

    Book  Google Scholar 

  • Möhring, K. (2012). The fixed effects approach as alternative to multilevel models for cross-national analyses. GK SOCLIFE Working Paper Series, pp 1–15.

  • Moulton, S., & Sandfort, J. R. (2017). The strategic action field framework for policy implementation research. Policy Science Journal, 45(1), 144–169.

    Google Scholar 

  • Moynihan, D. P. (2005). Goal-based learning and the future of performance management. Public Administration Review, 65(2), 203–216.

    Article  Google Scholar 

  • O’Toole, L. J., & Meier, K. J. (2014). Public management, context, and performance: In quest of a more general theory. Journal of Public Administration Research and Theory, 25(1), 237–256.

    Article  Google Scholar 

  • Pacula, R. L., Powell, D., Heaton, P., & Sevigny, E. L. (2015). Assessing the effects of medical marijuana laws on marijuana use: The devil is in the details. Journal of Policy Analysis and Management (the Journal of the Association for Public Policy Analysis and Management), 34(1), 7–31.

    Article  Google Scholar 

  • Piotrowski, S. J., & Rosenbloom, D. H. (2002). Nonmission-based values in results-oriented public management: The case of freedom of information. Public Administration Review, 62(6), 643–657.

    Article  Google Scholar 

  • Pirog, M. A., & Ziol-Guest, K. M. (2006). Child support enforcement: Programs and policies, impacts and questions. Journal of Policy Analysis and Management, 25(4), 943–990.

    Article  Google Scholar 

  • Pollitt, C. (2011). Performance blight and the tyranny of light? Performance blight and the tyranny of light? Accountability in advanced performance measurement regimes. In M. J. Dubnick & H. G. Frederickson (Eds.), Accountable governance: Problems and promises (pp. 81–98). M.E. Sharpe: Armonk.

    Google Scholar 

  • Radin, B. A. (2000). The government performance and results act and the tradition of federal management reform: Square pegs in round holes? Journal of Public Administration Research and Theory, 10, 111–135.

    Article  Google Scholar 

  • Radin, B. A. (2006). Challenging the performance movement: Accountability, complexity, and democractic values. Washington, D.C.: Georgetown University Press.

    Google Scholar 

  • Radin, B. A. (2009). What can we expect from performance measurement activities? Journal of Policy Analysis and Management, 28, 505–512.

    Article  Google Scholar 

  • Rinne, U., & Zimmermann, K. (2012). Another economic miracle? The German labor market and the Great Recession. IZA Journal of Labor Policy, 1(1), 1–21.

    Article  Google Scholar 

  • Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169.

    Article  Google Scholar 

  • Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2003). Evaluation: A systematic approach. Thousand Oaks: SAGE Publications.

    Google Scholar 

  • Schickler, E. (2001). Disjointed pluralism: Institutional innovation and the development of the US congress. Princeton: Prrinceton University Press.

    Google Scholar 

  • Schlaufer, C., Stucki, I., & Sager, F. (2018). The political use of evidence and its contribution to democratic discourse. Public Administration Review. https://doi.org/10.1111/puar.12923.

    Google Scholar 

  • Schneider, A. (2012). Policy design and transfer. In E. Araral, S. Fritzen, M. Howlett, M. Ramesh, & X. Wu (Eds.), Routledge handbook of public policy (pp. 217–228). Abingdon: Routledge.

    Google Scholar 

  • Thelen, K. (1999). Historical institutionalism in comparative politics’. Annual Review of Political Science, 2, 369–414.

    Article  Google Scholar 

  • Thelen, K. (2004). How institutions evolve: The political economy of skills in Germany, Britain, the United States, and Japan. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Van Dooren, W., & Van de Walle, S. (2016). Performance information in the public sector: How it is used. Berlin: Springer.

    Google Scholar 

  • Van Thiel, S., & Leeuw, F. L. (2002). The performance paradox in the public sector. Public Performance and Management Review, 25, 267–281.

    Article  Google Scholar 

Download references

Acknowledgements

This research was funded by CONSENSUS project financed under the European Commission’s 7th Framework Programm.

Author information

Authors and Affiliations

  1. LMU Munich, Munich, Germany

    Christian Adam, Yves Steinebach & Christoph Knill

Authors
  1. Christian Adam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yves Steinebach
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christoph Knill
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Adam.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 115 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Adam, C., Steinebach, Y. & Knill, C. Neglected challenges to evidence-based policy-making: the problem of policy accumulation. Policy Sci 51, 269–290 (2018). https://doi.org/10.1007/s11077-018-9318-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11077-018-9318-4

Keywords

Search

Navigation