Policy Sciences

, Volume 41, Issue 4, pp 335–355 | Cite as

The use and non-use of policy appraisal tools in public policy making: an analysis of three European countries and the European Union

  • Måns NilssonEmail author
  • Andrew Jordan
  • John Turnpenny
  • Julia Hertin
  • Björn Nykvist
  • Duncan Russel


The increasing complexity of policy problems, coupled with the political desire to base new policies on the foundation of firm evidence, has accelerated the development of policy assessment tools. These range from complex computer models and cost benefit analysis through simple checklists and decision trees. In the last decade, many governments have established formal policy assessment systems to harness these tools in order to facilitate more evidence-based policy making. These tools are potentially widely available, but to what extent are they used by policy makers and what becomes of the evidence that they generate? This paper addresses these questions by studying the empirical patterns of tool use across 37 cases in three European countries and the European Commission. It uses a simple classification of tools into advanced, formal and simple types. It finds that even when tools are embedded in policy assessment systems, their use is differentiated and on the whole very limited, in particular when it comes to more advanced tools. It then explores these patterns from contrasting theoretical perspectives to shed light on why, when and how different policy assessment tools are used in the policy process.


Policy assessment Impact assessment Knowledge Evidence Tool Model 



The research was undertaken as part of the EC’s FP6-funded project MATISSE (Methods and Tools for Integrated Sustainability Assessment). The authors thank Anneke von Raggamby and Ingmar von Homeyer whose work on five of the EU cases was funded by the FP6 project Sustainability A-Test. Duncan Russel’s participation was funded by the UK ESRC. We thank the anonymous reviewers for their comments on an earlier version of this paper.


  1. Bartlett, R., & Kurian, P. (1999). The theory of environmental impact assessment: Implicit models of policy making. Policy and Politics, 27, 415–433. doi: 10.1332/030557399782218371.CrossRefGoogle Scholar
  2. Bendor, J., Moe, T., & Shotts, K. (2001). Recycling the garbage can: An assessment of the research program. The American Political Science Review, 95, 169–190.Google Scholar
  3. BMI. (2000). Moderner Staat—moderne Verwaltung : Gemeinsame Geschäftsordnung der Bundesministerien. Berlin: Bundesministerium des Innern/German Federal Ministry of the Interior.Google Scholar
  4. BMJ (1999). Handbuch der Rechtsförmlichkeit. Empfehlungen des Bundesministeriums der Justiz zur einheitlichen rechtsförmlichen Gestaltung von Gesetzen und Rechtsverordnungen nach § 38 Abs. 3 GGO II. Berlin: Bundesministerium der Justiz/German Federal Ministry of Justice.Google Scholar
  5. Cabinet Office. (2000). Adding it up: Improving analysis and modelling in central government. London: HMSO.Google Scholar
  6. Cabinet Office. (2003). Regulatory impact assessment guidance. London: HMSO.Google Scholar
  7. Clarence, E. (2002). Editorial: Technocracy reinvented: The new evidence based policy movement. Public Policy and Administration, 17, 1–11. doi: 10.1177/095207670201700301.CrossRefGoogle Scholar
  8. Clark, W. C., & Majone, G. (1985). The critical assessment of scientific enquiries with policy implications. Science, Technology & Human Values, 10, 6–19. doi: 10.1177/016224398501000302.CrossRefGoogle Scholar
  9. Cohen, M., March, J., & Olsen, J. (1972). A garbage can model of organisational choice. Administrative Science Quarterly, 17, 1–25. doi: 10.2307/2392088.CrossRefGoogle Scholar
  10. Cronbach, L. J. (Ed.). (1980). Toward reform of programme evaluation. San Francisco: Jossey-Bass.Google Scholar
  11. de Ridder, W., Turnpenny, J., Nilsson, M., & von Raggamby, A. (2007). A framework for tool selection and use in integrated assessment for sustainable development. Journal of Environmental Assessment Policy and Management, 9, 423–441. doi: 10.1142/S1464333207002883.CrossRefGoogle Scholar
  12. European Commission. (2001). European governance: A white paper. COM (2001) 428. Brussels: European Commission.Google Scholar
  13. European Commission. (2002). Communication: European governancebetter law-making. COM (2002) 0275. Brussels: European Commission.Google Scholar
  14. European Commission. (2005). Impact assessment guidelines. Brussels: European Commission.Google Scholar
  15. European Commission. (2006). Communication: A strategic review of better regulation in the European Commission. Brussels: European Commission.Google Scholar
  16. Hanley, N. (2001). Cost benefit analysis and environmental policy making. Environment and Planning C, 19, 103–118.Google Scholar
  17. Harrington, W., & Morgenstern, R. D. (2004). Evaluating regulatory impact analyses. Washington, DC: Resources for the Future.Google Scholar
  18. Hertin, J., Turnpenny, J., Jordan, A., Nilsson, M., Russel, D., & Nykvist, B. (in press). Rationalising the policy mess? Ex ante assessment and the utilisation of knowledge in the policy process. Environment & Planning A.Google Scholar
  19. Huber, G. P. (1991). Organizational learning: The contributing processes and the literatures. Organization Science, 2, 88–115.CrossRefGoogle Scholar
  20. Jacobs, S. (2004). RIA and the economic transition to markets. Public Money and Management, 24, 283–290. doi: 10.1111/j.1467-9302.2004.00435.x.CrossRefGoogle Scholar
  21. Jenkins-Smith, H. (1990). Democratic politics and policy analysis. Belmont, CA: Brooks/Cole.Google Scholar
  22. Jordan, A., & Schout, A. (2006). The coordination of the European Union. Oxford: Oxford University Press.Google Scholar
  23. Jordan, A. J., Wurzel, R. K., & Zito, A. R. (Eds.). (2003). New instruments of environmental governance?. London: Routledge.Google Scholar
  24. Kates, R. W., Clark, W. C., Corell, R., Hall, J., Jaeger, C., Lowe, I., et al. (2001). Sustainability science. Science, 292, 641–642. doi: 10.1126/science.1059386.CrossRefGoogle Scholar
  25. Kingdon, J. W. (1984). Agendas, alternatives, and public policies. New York, NY: HarperCollins.Google Scholar
  26. Kleindorfer, P., Kunreuther, H., & Schoemaker, P. (1993). Decision sciences: An integrative perspective. Cambridge: Cambridge University Press.Google Scholar
  27. Lascoumes, P., & Le Gales, P. (2007). Introduction: Understanding public policy through its instruments. Governance, 20, 1–21. doi: 10.1111/j.1468-0491.2007.00342.x.CrossRefGoogle Scholar
  28. Lee, N., & Kirkpatrick, C. (2006). Evidence-based policy making in Europe: An evaluation of European Commission integrated impact assessments. Impact Assessment and Project Assessment, 24, 23–33. doi: 10.3152/147154606781765327.CrossRefGoogle Scholar
  29. Mitchell, R. B., Clark, W. C., Cash, D., & Dickinson, N. (2006). Global environmental assessments: Information and influence. Cambridge, MA: MIT Press.Google Scholar
  30. Nilsson, M. (2006). The role of assessments and institutions for policy learning: A study on Swedish climate and nuclear policy formation. Policy Sciences, 38, 225–249. doi: 10.1007/s11077-006-9006-7.CrossRefGoogle Scholar
  31. Olsen, J. (2001). Garbage cans, the new institutionalism, and the study of politics. The American Political Science Review, 95, 191–198.Google Scholar
  32. Owens, S., Rayner, T., & Bina, O. (2004). New agendas for assessment: Reflections on theory, practice, and research. Environment & Planning A, 36, 1943–1959. doi: 10.1068/a36281.CrossRefGoogle Scholar
  33. Pearce, D. (1998). Cost benefit analysis and environmental policy. Oxford Review of Economic Policy, 14, 84–100. doi: 10.1093/oxrep/14.4.84.CrossRefGoogle Scholar
  34. Radaelli, C. (1995). The role of knowledge in the policy process. Journal of European Public Policy, 2, 159–183. doi: 10.1080/13501769508406981.CrossRefGoogle Scholar
  35. Radaelli, C. (2004). The diffusion of regulatory impact analysis: Best practice or lesson-drawing? European Journal of Political Research, 43, 723–747. doi: 10.1111/j.0304-4130.2004.00172.x.CrossRefGoogle Scholar
  36. Radaelli, C. (2005). Diffusion without convergence: How political context shapes the adoption of regulatory impact assessment. Journal of European Public Policy, 12, 924–943. doi: 10.1080/13501760500161621.CrossRefGoogle Scholar
  37. Radaelli, C. (2007). Whither better regulation for the Lisbon Agenda? Journal of European Public Policy, 14, 190–207. doi: 10.1080/13501760601122274.CrossRefGoogle Scholar
  38. Radaelli, C. M., & De Francesco, F. (2007). Regulatory quality in Europe: Concepts, measures and policy processes. Manchester: Manchester University Press.Google Scholar
  39. Regeringskansliet, (2000). Ds 2000:1 Kommittéhandboken. Stockholm: Regeringskansliet.Google Scholar
  40. Rittel, H., & Webber, M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169. doi: 10.1007/BF01405730.CrossRefGoogle Scholar
  41. Sabatier, P. A. (Ed.). (1999). Theories of the policy process. Boulder, CO: Westview Press.Google Scholar
  42. Sabatier, P. A., & Jenkins-Smith, H. C. (1999). The advocacy coalition framework: An assessment. In P. A. Sabatier (Ed.), Theories of the policy process. Boulder, CO: Westview Press.Google Scholar
  43. Sanderson, I. (2006). Complexity, “practical rationality” and evidence-based policy making. Policy and Politics, 34, 115–132. doi: 10.1332/030557306775212188.CrossRefGoogle Scholar
  44. Shulock, N. (1999). The paradox of policy analysis: If it is not used, why do we produce so much of it? Journal of Policy Analysis and Management, 18, 226–244. doi:10.1002/(SICI)1520-6688(199921)18:2<226::AID-PAM2>3.0.CO;2-J.CrossRefGoogle Scholar
  45. Simon, H. (1958). Administrative behaviour. London: Macmillan.Google Scholar
  46. Turnpenny, J., Nilsson, M., Russel, D., Jordan, A., Hertin, J., & Nykvist, B. (2008). Why is integrating policy assessment so hard? A comparative analysis of the institutional capacities and constraints. Journal of Environmental Planning and Management, 51 (in press).Google Scholar
  47. Vibert, F. (2004). The EU’s new system of regulatory impact assessment: A scorecard. Brussels: The European Policy Forum.Google Scholar
  48. Weale, A. (2001). Science advice, democratic responsiveness and public policy. Science & Public Policy, 28, 413–421. doi: 10.3152/147154301781781237.CrossRefGoogle Scholar
  49. Weiss, C. (1975). Evaluation research in the political context. In E. L. Struening & M. Guttentag (Eds.), Handbook of evaluation research (Vol. 1). London: Sage.Google Scholar
  50. Weiss, C. (1979). The many meanings of research utilization. Public Administration Review, 39, 426–431. doi: 10.2307/3109916.CrossRefGoogle Scholar
  51. Wildavsky, A. B. (1979). Speaking truth to power: The art and craft of policy analysis. New Brunswick: Transaction.Google Scholar

Copyright information

© Springer Science+Business Media, LLC. 2008

Authors and Affiliations

  • Måns Nilsson
    • 1
    Email author
  • Andrew Jordan
    • 2
  • John Turnpenny
    • 2
  • Julia Hertin
    • 3
  • Björn Nykvist
    • 1
  • Duncan Russel
    • 2
  1. 1.Stockholm Environment InstituteStockholmSweden
  2. 2.School of Environmental SciencesUniversity of East AngliaNorwichUK
  3. 3.Department of Political and Social SciencesEnvironmental Policy Research CentreBerlinGermany

Personalised recommendations