Skip to main content

The Impact Evaluation of Sustainable Consumption Policy Instruments

Abstract

Putting sustainable consumption into practice is a challenge that requires the effort and coordination of numerous societal domains and actors. The paper deals with the contribution of policy making and policy evaluation. More specifically, it addresses the question of how to evaluate the effectiveness of policy instruments dedicated to rendering household consumption more sustainable. Despite the extensive literature on instrument effectiveness, sustainability assessment, and consumer behaviour, only a few accounts deal with the specific characteristics and impacts of policy instruments for sustainable consumption. Against this backdrop, a framework is suggested for the ex post analysis of effects resulting from such policy instruments. Instrument effects include changes in consumption patterns (“outcomes”), subsequent changes in the state of the environment, society and/or economy (“impacts”), and side effects. Step-by-step guidance is provided through the evaluation process. The approach helps to assess the extent to which sustainable consumption policy instruments have achieved their stated goals, but also encourages a critical reflection of these goals. In addition to evaluating instrument effects, the framework serves to explain these effects. It does so on the basis of theoretically grounded hypotheses that tackle drivers of and barriers to instrument effectiveness, thus exploring this relatively new policy field. Methodologically, a combination of qualitative methods (narrative reconstruction) and quantitative methods (e.g., material flow analysis) is recommended to causally link policies to changes in consumption patterns and impacts on sustainability.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. 1.

    Note that in some cases, an instrument combines two or more governance mechanisms (hybrid instruments).

  2. 2.

    The term was coined by EC Directive 96/61 with a view to establishing flexible environmental requirements in order to minimize emissions resulting from certain installations and to reduce the environmental impact of products and production processes as a whole.

  3. 3.

    These are negotiated interactive relationships between consumption and production in a sector.

  4. 4.

    (Only) If it can be expected that specific side effects will contribute significantly to goal attainment, too, it might be interesting to develop outcome indicators for the respective side effects.

  5. 5.

    Possible data sources include: market statistics; market research; data by producers, retailers, business associations, etc. on product specifications, market shares, dissemination or use rates, etc.; information by public administrations (including public funding agencies) on the outreach of programmes and instruments, e.g., the number of SC-relevant grants awarded; results from the formal monitoring of specific indicators, e.g., in the context of socioeconomic panels or programme evaluations; surveys on customer awareness and usage/disposal practices; other data.

  6. 6.

    Only in the case of very strong changes that trigger “system responses” would further adjustments need to be made in the material flow analysis.

  7. 7.

    For a discussion of the above presented hypotheses againgt emirical evidence from 10 European case studies-tackling SC in the fields of energy and nutrition-, see the EUPOPP website www.eupopp.net.

References

  1. Aalto, K., Heiskanen, E., Leire, C., & Thidell, Å. (2008). The Nordic Swan—From past experiences to future possibilities. TemaNord, 2008, 529.

    Google Scholar 

  2. Akerlof, G. A. (1970). The market for ‘lemons’—Quality uncertainty and the market mechanism. Quarterly Journal of Economics, 84, 488–500.

    Article  Google Scholar 

  3. ANEC. (2007). A review of the range of activity throughout Member States related to compliance with the EU energy label regulations in those countries. Brussels: European Association for the Co-ordination of Consumer Representation in Standardisation.

    Google Scholar 

  4. Bemelmans-Videc, M. L., Rist, R., & Vedung, E. (1998). Carrots, sticks and sermons—Policy instruments and their evaluation. New Brunswick: Transaction Publishers.

    Google Scholar 

  5. Bennear, L. S., & Coglianese, G. (2005). Measuring progress—Program evaluation of environmental policies. Environment, 47(2), 22–39.

    Google Scholar 

  6. Biggart, N. W., & Lutzenhiser, L. (2007). Economic sociology and the social problem of energy inefficiency. The American Behavioral Scientist, 50, 1070–1087.

    Article  Google Scholar 

  7. Bijker, W., Hughes, T., & Pinch, T. (1989). The social construction of technological systems, new directions in the sociology and history of technology. Cambridge: MIT Press.

    Google Scholar 

  8. Blamey, A., & Mackenzie, M. (2007). Theories of change and realistic evaluation—Peas in a pod or apples and oranges? Evaluation, 13, 439–455.

    Article  Google Scholar 

  9. Boonekamp, P. G. M. (2005). Matrices of interaction between policy measures on household energy efficiency in EU countries. ECEEE 2005 Summer Study Proceedings, 1, 137–148.

    Google Scholar 

  10. Bovens, M. A. P., 't Hart, P., & Kuipers, S. L. (2006). The politics of policy evaluation. In R. Goodin, M. Moran, & M. Rein (Eds.), Oxford handbook of public policy (pp. 317–333). Oxford: Oxford University Press.

    Google Scholar 

  11. Brunner, P., & Rechberger, H. (2003). Practical handbook of material flow analysis. London: CRC.

    Book  Google Scholar 

  12. Child, R., Langniss, O., Klink, J., & Gaudioso, D. (2008). Interactions of white certificates with other policy instruments in Europe. Energy Efficiency, 1, 283–295.

    Article  Google Scholar 

  13. Coenen, F. (Ed.). (2009). Public participation and better environmental decisions. The promise and limits of participatory processes for the quality of environmentally related decision-making. Berlin: Springer.

    Google Scholar 

  14. Coffman, J. (2002). Public communication campaign evaluation. An environmental scan of challenges, criticisms, practice and opportunities. Cambridge: Harvard Family Research Project.

    Google Scholar 

  15. Cohen, M., & Murphy, J. (2001). Exploring sustainable consumption—Environmental policy and the social sciences. Oxford: Pergamon.

    Google Scholar 

  16. Crabbé, A., & Leroy, P. (2008). The handbook of environmental policy evaluation. Sterling: Earthscan.

    Google Scholar 

  17. Davidson, E. J. (2005). Evaluation methodology basics. The nuts and bolts of sound evaluation. Thousand Oaks: Sage.

    Google Scholar 

  18. De Pelsmacker, P., Janssens, W., Sterckx, E., & Mielants, C. (2005). Consumer preferences for the marketing of ethically labeled coffee. International Marketing Review, 22, 512–530.

    Article  Google Scholar 

  19. De Ridder, W., Turnpenny, J., Nilsson, M., & von Raggambym, A. (2007). A framework for tool selection and use in integrated assessment for sustainable development. Journal of Environmental Assessment Policy & Management, 9, 423–441.

    Article  Google Scholar 

  20. Dolowitz, D. P., & Marsh, D. (2000). Learning from abroad. The role of policy transfer in contemporary policy making. Governance, 13, 5–24.

    Article  Google Scholar 

  21. Dunn, W. N. (2003). Public policy analysis—An introduction (3rd ed.). Upper Saddle River: Prentice Hall.

    Google Scholar 

  22. Eberle, U., Brohmann, B., & Graulich, K. (2004). Sustainable consumption needs visions. Position paper. Freiburg: Institute for Applied Ecology (Öko-Institut).

    Google Scholar 

  23. EC. (2008). Accompanying document to the proposal for a Directive of the European Parliament and of the Council on the indication by labelling and standard product information of the consumption of energy and other resources by energy-related products. Brussels: European Commission. {SEC(2008) 2863}.

  24. Edelenbos, J., & Klijn, E.-H. (2006). Managing stakeholder involvement in decision making. A comparative analysis of six interactive processes in the Netherlands. Journal of Public Administration Research and Theory, 16, 417–446.

    Article  Google Scholar 

  25. EEA. (2001). Reporting on environmental measures, are we being effective? Copenhagen: European Energy Agency. Environmental issue report No. 25.

  26. EEA. (2005). Household consumption and the environment. Copenhagen: European Energy Agency. EEA Report 11/25.

  27. Evans, M. (2004). Policy transfer in global perspective. Aldershot: Ashgate.

    Google Scholar 

  28. Fine, B., & Leopold, E. (1993). The world of consumption. London: Routledge.

    Google Scholar 

  29. Fischer, F. (1998). Beyond empiricism—Policy inquiry in postpositivist perspective. Policy Studies Journal, 26(1), 129–146.

    Article  Google Scholar 

  30. Fuchs, D., & Lorek, S. (2005). Sustainable consumption governance—A history of promises and failures. Journal of Consumer Policy, 28, 261–288.

    Article  Google Scholar 

  31. George, A. L., & Bennett, A. (2005). Case studies and theory development in the social sciences. Cambridge: MIT Press.

    Google Scholar 

  32. George, C., & Kirkpatrick, C. (2007). Impact assessment and sustainable development. European practice and experience. Cheltenham: Edward Elgar.

    Google Scholar 

  33. Giddens, A. (1984). The constitution of society. Cambridge: Polity Press.

    Google Scholar 

  34. Gronow, J., & Warde, A. (2001). Ordinary consumption. London: Routledge.

    Google Scholar 

  35. Gunningham, N., Grabowsky, P., & Sinclair, D. (2004). Smart regulation—Designing environmental policy. Oxford: Clarendon.

    Google Scholar 

  36. Guy, S., & Shove, E. (2000). The sociology of energy, buildings and the environment, constructing knowledge, designing practice. Abingdon: Taylor & Francis.

    Google Scholar 

  37. Gysen, J., Bruynickx, H., & Bachus, K. (2006). The modus narrandi—A methodology for evaluating effects of environmental policy. Evaluation, 12, 95–118.

    Article  Google Scholar 

  38. Heiskanen, E., & Schönherr, N. (2009). EUPOPP Conceptual Framework. EUPOPP Work Package 1, Deliverable 1.2. With contributions from Aalto, K., Adell, A., Alcantud, A., Barth, R., Bremere, I., Brohmann, B., Fritsche, U., Larcom, S., Schäfer, B., Schmitt, K., Wolff, F. Online at www.eupopp.net.

  39. Heiskanen, E., Halme, M., Jalas, M., Kärnä, A., & Lovio, R. (2001). Dematerialization, the potential of ICT and services. Helsinki: Ministry of Environment. Finnish Environment 438.

  40. Hertwich, E. (2005). Life cycle approaches to sustainable consumption: A critical review. Environmental Science & Technology, 39, 4673–4684.

    Article  Google Scholar 

  41. Homer-Dixon, T. (1996). Strategies for studying causation in complex ecological political systems. The Journal of Environment & Development, 5, 132–148.

    Article  Google Scholar 

  42. Howlett, M. (2005). What is a policy instrument? Policy tools, policy mixes, and policy implementation styles. In E. Pearl, M. M. Hill, & M. Howlett (Eds.), Designing government. From instruments to governance (pp. 31–50). Ithaca: McGill-Queen’s University Press.

    Google Scholar 

  43. Howlett, M., & Ramesh, M. (2003). Studying public policy, policy cycles and policy subsystems. Oxford: Oxford University Press.

    Google Scholar 

  44. Hünecke, K., & Rausch, L. (2009). Coupling consumption trends and sustainable consumption policies—In consideration of regression analysis. EUPOPP working paper. With contributions from Fritsche, U., Hook, S. Online at www.eupopp.net.

  45. IEA & OECD. (2003). Cool appliances. Policy strategies for energy-efficient homes. Vienna/Paris: International Energy Agency/Organisation for Economic Cooperation and Development.

    Google Scholar 

  46. Jackson, T. (2004). Negotiating sustainable consumption: A review of the consumption debate and its policy implications. Energy & Environment, 15, 1027–1053.

    Article  Google Scholar 

  47. Jackson, T. (2005). Motivating sustainable consumption—A review of models of consumer behaviour and behavioural change. London: Sustainable Development Research Network.

    Google Scholar 

  48. Jacobs, S. (1997). An overview of regulatory impact analysis in OECD countries. In OECD (Ed.), Regulatory impact analysis: Best practices in OECD countries. Paris: Organisation for Economic Cooperation and Development.

    Google Scholar 

  49. Jordan, A., Wurzel, R., Zito, A., & Brückner, L. (2004). Consumer responsibility-taking and national eco-labelling schemes in Europe. In M. Micheletti, A. Follesdal, & D. Stolle (Eds.), Politics, products and markets. Exploring political consumerism (pp. 161–180). Somerset: Transaction Publishers.

    Google Scholar 

  50. Kautto, P., & Similä, J. (2005). Recently introduced policy instruments and intervention theories. Evaluation, 11, 55–68.

    Article  Google Scholar 

  51. Leeuw, F. (2003). Reconstructing program theories. Methods available and problems to be solved. American Journal of Evaluation, 24(1), 5–20.

    Google Scholar 

  52. Leeuw, F., & Vaessen, J. (2009). Impact evaluations and development. Washington: NONIE Guidance on Impact Evaluation.

    Google Scholar 

  53. Leire, C., & Thidell, A. (2004). Product-related environmental information to guide consumer purchases—A review and analysis of research on perceptions, understanding and use among Nordic consumers. Journal of Cleaner Production, 13, 1061–1070.

    Article  Google Scholar 

  54. Mickwitz, P. (2003). A framework for evaluating environmental policy instruments. Evaluation, 9, 415–436.

    Article  Google Scholar 

  55. Mont, O. (2007). The role of households in SCP agenda. A one-actor show or concerted action? International Journal of Innovation and Sustainable Development, 2, 201–214.

    Article  Google Scholar 

  56. Ness, B., Urbel-Piirsalu, E., Anderber, S., & Olssan, L. (2006). Categorizing tools for sustainability assessment. Ecological Economics, 60, 498–508.

    Article  Google Scholar 

  57. Norwegian Ministry for the Environment. (1994). Symposium on Sustainable Consumption. Oslo, Norway.

  58. OECD. (2002). Towards sustainable household consumption? Trends and policies in OECD Countries. Paris: Organisation for Economic Cooperation and Development.

    Google Scholar 

  59. OECD. (2008). Household behaviour and the environment. Paris: Organisation for Economic Cooperation and Development.

    Google Scholar 

  60. Papaconstantinou, G., & Polt, W. (1998). Policy evaluation in innovation and technology, an overview. In OECD (Ed.), Policy evaluation in innovation and technology. Towards best practices. Paris: Organisation for Economic Cooperation and Development.

    Google Scholar 

  61. Persson, Å., & Nilsson, M. (2007). Towards a framework for SEA follow-up—Theoretical issues and lessons learned from policy evaluation. Journal of Environmental Assessment Policy & Management, 9, 473–496.

    Article  Google Scholar 

  62. Prittwitz, V. (2001). Politikanalyse. Stuttgart: Lucius & Lucius Verlagsgesellschaft.

    Google Scholar 

  63. Rohracher, H. (2001). Managing the technological transition to sustainable construction of buildings—A socio-technical perspective. Technology Analysis & Strategic Mangement, 13, 137–150.

    Article  Google Scholar 

  64. Rohracher, H. (2003). The role of users in the social shaping of environmental technologies. Innovation, 16, 177–192.

    Google Scholar 

  65. Rose, R. (1993). Lesson-drawing in public policy. Chatham: Chatham House.

    Google Scholar 

  66. Rossi, P., Freeman, H., & Lipsey, M. (1999). Evaluation. A systematic approach (6th ed.). Thousand Oaks: Sage.

    Google Scholar 

  67. Rubik, F., Scholl, G., Biedenkopf, K., Kalimo, H., Mohaupt, F., Söebech, Ó. et al. (2009). Innovative approaches in European sustainable consumption policies. Berlin: Schriftenreihe des IÖW 192/09.

  68. Russell, S., & Williams, R. (2002). Social shaping of technology: Frameworks, findings and implications for policy with glossary of social shaping concepts. In K. H. Sørensen & R. Williams (Eds.), Shaping technology, guiding policy: Concepts, spaces and tools. Cheltenham: Edward Elgar.

    Google Scholar 

  69. Sanne, C. (2002). Willing consumers—Or locked-in? Policies for a sustainable consumption. Ecological Economics, 42, 273–287.

    Article  Google Scholar 

  70. Schettkat, R. (2009). Analyzing rebound effects. Wuppertal: Wuppertal Paper No. 177.

  71. Schipper, L. (2000). On the rebound—The interaction of energy efficiency, energy use and economic activity. Energy Policy, 28, 351–501.

    Article  Google Scholar 

  72. Schönherr, N., Brohmann, B., Fritsche, U., Heiskanen, E., & Wolff, F. (2010). Policies to promote sustainable consumption. First results from an EU-wide impact evaluation project. Paper presented at the Knowledge Collaboration & Learning for Sustainable Innovation ERSCP-EMSU Conference, Delft, The Netherlands, October 25-29, 2010.

  73. Schubert, U., & Störmer, E. (Eds.). (2007). Sustainable development in Europe—Concepts, evaluation and application. Cheltenham: Edward Elgar.

    Google Scholar 

  74. Schultz, I., & Stieß, I. (2008). Linking sustainable consumption to everyday life: A social–ecological approach to consumption research. In A. Tukker, M. Charter, & C. Vezzoli (Eds.), Perspectives on radical changes to sustainable consumption and production 1. System innovation for sustainability (pp. 288–300). Sheffield: Greenleaf Publishing Ltd.

    Google Scholar 

  75. Scriven, M. (1991). Evaluation thesaurus. London: Sage.

    Google Scholar 

  76. Shove, E. (2003a). Comfort, cleanliness and convenience—The social organisation of normality. Oxford: Berg Publishers.

    Google Scholar 

  77. Shove, E. (2003b). Converging conventions of comfort, cleanliness and convenience. Journal of Consumer Policy, 26, 395–418.

    Article  Google Scholar 

  78. Simoes, S., Huppes, G., & Seixas, J. (2005). A tangled web—Assessing overlaps between energy and environmental policy instruments in place along electricity systems. eceee 2005 Summer Study Proceedings, 1, 125–126.

    Google Scholar 

  79. Sorrell, S. (Ed.). (2003). Interaction in EU climate policy. Final report. Project No. EVK2- CT-2000-0067. Brighton: SPRU.

    Google Scholar 

  80. Southerton, D., Chappells, H., & Van Vliet, B. (2004). Sustainable consumption—The implications of changing infrastructures of provision. Cheltenham: Edward Elgar.

    Google Scholar 

  81. Spaargaren, G. (2003). Sustainable consumption—A theoretical and environmental policy perspective. Society and Natural Resources, 16, 687–701.

    Article  Google Scholar 

  82. Spaargaren, G., & Mol, A. P. J. (2008). Greening global consumption—Redefining politics and authority. Global Environmental Change, 18, 350–359.

    Article  Google Scholar 

  83. Stame, N. (2004). Theory-based evaluation and types of complexity. Evaluation, 10(1), 58–70.

    Article  Google Scholar 

  84. Stiglitz, J. E. (2000). The contributions of the economics of information to twentieth century economics. The Quarterly Journal of Economics, 115, 1441–1478.

    Article  Google Scholar 

  85. Stone, D. (1999). Learning lessons and transferring policy across time, space and disciplines. Politics, 19, 51–59.

    Article  Google Scholar 

  86. Stufflebeam, D. L. (2001). Evaluation models. New directions for evaluation, 89. San Francisco: Jossey-Bass.

    Google Scholar 

  87. Thøgersen, J. (2005). How may consumer policy empower consumers for sustainable lifestyles? Journal of Consumer Policy, 28, 143–178.

    Article  Google Scholar 

  88. Tukker, A., Charter, M., Vezzoli, C., Sto, E., & Munch Andersen, M. (Eds.). (2008). System innovation for sustainability 1. Perspectives on radical changes to sustainable consumption and production. Sheffield: Greenleaf Publishing.

    Google Scholar 

  89. UNDESA & UNEP. (2005). Second International Expert Meeting on a 10-Year Framework of Programmes for Sustainable Consumption and Production. San José, Costa Rica, 5–8 September 2005. Summary by the Co-Chairs.

  90. van der Doelen, F. C. J. (1998). The “give and take” packaging of policy instruments: Optimizing legitimacy and effectiveness. In M.-L. Bemelmans-Videc, R. Rist, & E. Vedung (Eds.), Carrots, sticks and sermons—Policy instruments and their evaluation (pp. 129–146). New Brunswick: Transaction Publishers.

    Google Scholar 

  91. van der Meer, F.-B. (1999). Evaluation and the social construction of impacts. Evaluation, 5, 387–406.

    Article  Google Scholar 

  92. Vedung, E. (2004). Public policy and program evaluation. New Brunswick: Transaction Publishers.

    Google Scholar 

  93. Vreuls, H. (2004). Evaluating energy efficiency policy measures & DSM programmes, vol. 1. Evaluation guidebook. IEA-DSM Programme. INDEEP Analysis Report 2004. Paris: International Energy Agency Demand-Side Management Programme.

  94. Weiss, C. (1998). Evaluation. Upper Saddle River: Prentice Hall.

    Google Scholar 

  95. Wiegmann, K., Eberle, U., Fritsche, U., & Hünecke, K. (2005). Umweltauswirkungen von Ernährung—Stoffstromanalysen und Szenarien. Darmstadt/Hamburg: Öko-Institut e.V.. Diskussionspapier No. 7.

  96. Wilhite, H., Shove, E., Lutzenhiser, L., & Kempton, W. (2000). The legacy of twenty years of energy demand management—We know more about individual behaviour but next to nothing about demand. In E. Jochem et al. (Eds.), Society, behaviour, and climate change mitigation (pp. 109–126). Dordrecht: Kluwer.

    Google Scholar 

  97. Wolff, F. (2004). Staatlichkeit im Wandel—Aspekte kooperativer Umweltpolitik. München: Oekom Verlag.

    Google Scholar 

  98. Zauberman, G. (2003). The intertemporal dynamics of consumer lock-in. Journal of Consumer Research, 30, 405–419.

    Article  Google Scholar 

Download references

Acknowledgements

The paper is based on research within the project “Policies to Promote Sustainable Consumption Patterns” (EUPOPP, www.eupopp.net), funded within the EU’s 7th Framework Programme, Grant Agreement No. 212236. We would like to thank Eva Heiskanen for her contributions to the paper, Uwe Fritsche for critical reading, and all other colleagues from the EUPOPP project for their helpful comments on earlier versions of the text. Our special thanks also go to Ulf Schrader, John Thøgersen, and two anonymous reviewers for their highly valuable suggestions. All remaining errors and inconsistencies are the sole responsibility of the authors.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Franziska Wolff.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wolff, F., Schönherr, N. The Impact Evaluation of Sustainable Consumption Policy Instruments. J Consum Policy 34, 43–66 (2011). https://doi.org/10.1007/s10603-010-9152-3

Download citation

Keywords

  • Sustainable consumption
  • Policy instruments
  • Policy evaluation
  • Impact assessment
  • Effectiveness