Skip to main content

The use and non-use of policy appraisal tools in public policy making: an analysis of three European countries and the European Union

Abstract

The increasing complexity of policy problems, coupled with the political desire to base new policies on the foundation of firm evidence, has accelerated the development of policy assessment tools. These range from complex computer models and cost benefit analysis through simple checklists and decision trees. In the last decade, many governments have established formal policy assessment systems to harness these tools in order to facilitate more evidence-based policy making. These tools are potentially widely available, but to what extent are they used by policy makers and what becomes of the evidence that they generate? This paper addresses these questions by studying the empirical patterns of tool use across 37 cases in three European countries and the European Commission. It uses a simple classification of tools into advanced, formal and simple types. It finds that even when tools are embedded in policy assessment systems, their use is differentiated and on the whole very limited, in particular when it comes to more advanced tools. It then explores these patterns from contrasting theoretical perspectives to shed light on why, when and how different policy assessment tools are used in the policy process.

This is a preview of subscription content, access via your institution.

Notes

  1. As opposed to programmes/plans and project level decisions which are routinely subjected to strategic environmental assessment (SEA) or environmental impact assessment (EIA).

  2. See the exchange between Olsen (2001) and Bendor et al. (2001).

  3. Interview with German official, March 2006.

  4. Interview with UK official, June 2006.

  5. Interview with German official, March 2006.

  6. Interview with Commission official, February 2006.

  7. Interview with UK official, May 2006.

  8. Interview with UK official, June 2006.

  9. Interview with Commission official, January 2006.

  10. Interview with UK official, May 2006.

  11. Interview with UK official, May 2006.

  12. Interview with Swedish official, April 2006.

References

  • Bartlett, R., & Kurian, P. (1999). The theory of environmental impact assessment: Implicit models of policy making. Policy and Politics, 27, 415–433. doi:10.1332/030557399782218371.

    Article  Google Scholar 

  • Bendor, J., Moe, T., & Shotts, K. (2001). Recycling the garbage can: An assessment of the research program. The American Political Science Review, 95, 169–190.

    Google Scholar 

  • BMI. (2000). Moderner Staat—moderne Verwaltung : Gemeinsame Geschäftsordnung der Bundesministerien. Berlin: Bundesministerium des Innern/German Federal Ministry of the Interior.

    Google Scholar 

  • BMJ (1999). Handbuch der Rechtsförmlichkeit. Empfehlungen des Bundesministeriums der Justiz zur einheitlichen rechtsförmlichen Gestaltung von Gesetzen und Rechtsverordnungen nach § 38 Abs. 3 GGO II. Berlin: Bundesministerium der Justiz/German Federal Ministry of Justice.

  • Cabinet Office. (2000). Adding it up: Improving analysis and modelling in central government. London: HMSO.

    Google Scholar 

  • Cabinet Office. (2003). Regulatory impact assessment guidance. London: HMSO.

    Google Scholar 

  • Clarence, E. (2002). Editorial: Technocracy reinvented: The new evidence based policy movement. Public Policy and Administration, 17, 1–11. doi:10.1177/095207670201700301.

    Article  Google Scholar 

  • Clark, W. C., & Majone, G. (1985). The critical assessment of scientific enquiries with policy implications. Science, Technology & Human Values, 10, 6–19. doi:10.1177/016224398501000302.

    Article  Google Scholar 

  • Cohen, M., March, J., & Olsen, J. (1972). A garbage can model of organisational choice. Administrative Science Quarterly, 17, 1–25. doi:10.2307/2392088.

    Article  Google Scholar 

  • Cronbach, L. J. (Ed.). (1980). Toward reform of programme evaluation. San Francisco: Jossey-Bass.

    Google Scholar 

  • de Ridder, W., Turnpenny, J., Nilsson, M., & von Raggamby, A. (2007). A framework for tool selection and use in integrated assessment for sustainable development. Journal of Environmental Assessment Policy and Management, 9, 423–441. doi:10.1142/S1464333207002883.

    Article  Google Scholar 

  • European Commission. (2001). European governance: A white paper. COM (2001) 428. Brussels: European Commission.

  • European Commission. (2002). Communication: European governancebetter law-making. COM (2002) 0275. Brussels: European Commission.

  • European Commission. (2005). Impact assessment guidelines. Brussels: European Commission.

    Google Scholar 

  • European Commission. (2006). Communication: A strategic review of better regulation in the European Commission. Brussels: European Commission.

    Google Scholar 

  • Hanley, N. (2001). Cost benefit analysis and environmental policy making. Environment and Planning C, 19, 103–118.

    Google Scholar 

  • Harrington, W., & Morgenstern, R. D. (2004). Evaluating regulatory impact analyses. Washington, DC: Resources for the Future.

    Google Scholar 

  • Hertin, J., Turnpenny, J., Jordan, A., Nilsson, M., Russel, D., & Nykvist, B. (in press). Rationalising the policy mess? Ex ante assessment and the utilisation of knowledge in the policy process. Environment & Planning A.

  • Huber, G. P. (1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2, 88–115.

    Article  Google Scholar 

  • Jacobs, S. (2004). RIA and the economic transition to markets. Public Money and Management, 24, 283–290. doi:10.1111/j.1467-9302.2004.00435.x.

    Article  Google Scholar 

  • Jenkins-Smith, H. (1990). Democratic politics and policy analysis. Belmont, CA: Brooks/Cole.

    Google Scholar 

  • Jordan, A., & Schout, A. (2006). The coordination of the European Union. Oxford: Oxford University Press.

    Google Scholar 

  • Jordan, A. J., Wurzel, R. K., & Zito, A. R. (Eds.). (2003). New instruments of environmental governance?. London: Routledge.

    Google Scholar 

  • Kates, R. W., Clark, W. C., Corell, R., Hall, J., Jaeger, C., Lowe, I., et al. (2001). Sustainability science. Science, 292, 641–642. doi:10.1126/science.1059386.

    Article  Google Scholar 

  • Kingdon, J. W. (1984). Agendas, alternatives, and public policies. New York, NY: HarperCollins.

    Google Scholar 

  • Kleindorfer, P., Kunreuther, H., & Schoemaker, P. (1993). Decision sciences: An integrative perspective. Cambridge: Cambridge University Press.

    Google Scholar 

  • Lascoumes, P., & Le Gales, P. (2007). Introduction: Understanding public policy through its instruments. Governance, 20, 1–21. doi:10.1111/j.1468-0491.2007.00342.x.

    Article  Google Scholar 

  • Lee, N., & Kirkpatrick, C. (2006). Evidence-based policy making in Europe: An evaluation of European Commission integrated impact assessments. Impact Assessment and Project Assessment, 24, 23–33. doi:10.3152/147154606781765327.

    Article  Google Scholar 

  • Mitchell, R. B., Clark, W. C., Cash, D., & Dickinson, N. (2006). Global environmental assessments: Information and influence. Cambridge, MA: MIT Press.

    Google Scholar 

  • Nilsson, M. (2006). The role of assessments and institutions for policy learning: A study on Swedish climate and nuclear policy formation. Policy Sciences, 38, 225–249. doi:10.1007/s11077-006-9006-7.

    Article  Google Scholar 

  • Olsen, J. (2001). Garbage cans, the new institutionalism, and the study of politics. The American Political Science Review, 95, 191–198.

    Google Scholar 

  • Owens, S., Rayner, T., & Bina, O. (2004). New agendas for assessment: Reflections on theory, practice, and research. Environment & Planning A, 36, 1943–1959. doi:10.1068/a36281.

    Article  Google Scholar 

  • Pearce, D. (1998). Cost benefit analysis and environmental policy. Oxford Review of Economic Policy, 14, 84–100. doi:10.1093/oxrep/14.4.84.

    Article  Google Scholar 

  • Radaelli, C. (1995). The role of knowledge in the policy process. Journal of European Public Policy, 2, 159–183. doi:10.1080/13501769508406981.

    Article  Google Scholar 

  • Radaelli, C. (2004). The diffusion of regulatory impact analysis: Best practice or lesson-drawing? European Journal of Political Research, 43, 723–747. doi:10.1111/j.0304-4130.2004.00172.x.

    Article  Google Scholar 

  • Radaelli, C. (2005). Diffusion without convergence: How political context shapes the adoption of regulatory impact assessment. Journal of European Public Policy, 12, 924–943. doi:10.1080/13501760500161621.

    Article  Google Scholar 

  • Radaelli, C. (2007). Whither better regulation for the Lisbon Agenda? Journal of European Public Policy, 14, 190–207. doi:10.1080/13501760601122274.

    Article  Google Scholar 

  • Radaelli, C. M., & De Francesco, F. (2007). Regulatory quality in Europe: Concepts, measures and policy processes. Manchester: Manchester University Press.

    Google Scholar 

  • Regeringskansliet, (2000). Ds 2000:1 Kommittéhandboken. Stockholm: Regeringskansliet.

    Google Scholar 

  • Rittel, H., & Webber, M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169. doi:10.1007/BF01405730.

    Article  Google Scholar 

  • Sabatier, P. A. (Ed.). (1999). Theories of the policy process. Boulder, CO: Westview Press.

    Google Scholar 

  • Sabatier, P. A., & Jenkins-Smith, H. C. (1999). The advocacy coalition framework: An assessment. In P. A. Sabatier (Ed.), Theories of the policy process. Boulder, CO: Westview Press.

    Google Scholar 

  • Sanderson, I. (2006). Complexity, “practical rationality” and evidence-based policy making. Policy and Politics, 34, 115–132. doi:10.1332/030557306775212188.

    Article  Google Scholar 

  • Shulock, N. (1999). The paradox of policy analysis: If it is not used, why do we produce so much of it? Journal of Policy Analysis and Management, 18, 226–244. doi:10.1002/(SICI)1520-6688(199921)18:2<226::AID-PAM2>3.0.CO;2-J.

    Article  Google Scholar 

  • Simon, H. (1958). Administrative behaviour. London: Macmillan.

    Google Scholar 

  • Turnpenny, J., Nilsson, M., Russel, D., Jordan, A., Hertin, J., & Nykvist, B. (2008). Why is integrating policy assessment so hard? A comparative analysis of the institutional capacities and constraints. Journal of Environmental Planning and Management, 51 (in press).

  • Vibert, F. (2004). The EU’s new system of regulatory impact assessment: A scorecard. Brussels: The European Policy Forum.

    Google Scholar 

  • Weale, A. (2001). Science advice, democratic responsiveness and public policy. Science & Public Policy, 28, 413–421. doi:10.3152/147154301781781237.

    Article  Google Scholar 

  • Weiss, C. (1975). Evaluation research in the political context. In E. L. Struening & M. Guttentag (Eds.), Handbook of evaluation research (Vol. 1). London: Sage.

    Google Scholar 

  • Weiss, C. (1979). The many meanings of research utilization. Public Administration Review, 39, 426–431. doi:10.2307/3109916.

    Article  Google Scholar 

  • Wildavsky, A. B. (1979). Speaking truth to power: The art and craft of policy analysis. New Brunswick: Transaction.

    Google Scholar 

Download references

Acknowledgments

The research was undertaken as part of the EC’s FP6-funded project MATISSE (Methods and Tools for Integrated Sustainability Assessment). The authors thank Anneke von Raggamby and Ingmar von Homeyer whose work on five of the EU cases was funded by the FP6 project Sustainability A-Test. Duncan Russel’s participation was funded by the UK ESRC. We thank the anonymous reviewers for their comments on an earlier version of this paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Måns Nilsson.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Nilsson, M., Jordan, A., Turnpenny, J. et al. The use and non-use of policy appraisal tools in public policy making: an analysis of three European countries and the European Union. Policy Sci 41, 335–355 (2008). https://doi.org/10.1007/s11077-008-9071-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11077-008-9071-1

Keywords

  • Policy assessment
  • Impact assessment
  • Knowledge
  • Evidence
  • Tool
  • Model