Abstract
Realizing a 20% energy efficiency improvement in Europe by 2020 requires the introduction of good new energy efficiency policies as well as strengthening and enforcing the existing policies. This raises the question: what characterizes good and effective energy efficiency policies and their implementation? Systematic ex post evaluation of energy efficiency policies can reveal factors determining not only what works and what does not but also explain why. Ex post evaluation of 20 energy efficiency policy instruments applied across different sectors and countries in Europe among others showed that ex post evaluation does not yet have a high priority among policy makers: Often, quantitative targets and clear timeframes are lacking, and monitoring information is not collected on a regular basis. Our analysis, however, did reveal some general factors in the process of design and implementation of policy instruments that appear as important including (1) existence of clear goals and a mandate for the implementing organization, (2) the ability to balance and combine flexibility and continuity, (3) the involvement of stakeholders, and (4) the ability to adapt to and integrate adjacent policies or develop consistent policy packages. The analysis was performed using a uniform methodology called “theory-based policy evaluation”. The general principle behind this approach is that a likely theory is drawn up on the program’s various steps of logic of intervention to achieve its targeted impact in terms of energy efficiency improvement. The approach has several benefits over other ex post evaluation methods because (1) the whole policy implementation process is evaluated and the focus is not just on the final impacts, (2) through the development of indicators for each step in the implementation process, the “successes and failures” can be determined to the greatest extent possible, and (3) by applying this approach, we not only learn whether policies are successful or not but also why they succeeded or failed and how they can be improved.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Active Implementation of the Directive on Energy Efficiency (AID-EE)
References
Blumstein, C., Goldstone, S., & Lutzenhiser, L. (2000). A theory-based approach to market transformation. Energy Policy, 28(2000), 137–144.
Bongardt, D., & Kebeck, K. (2007). New governance or symbolic policy?—Evaluation and recommendations for the agreement between the European Commission and the Automobile Industry. Paper presented at the ECEEE Summer Study 2007, France, June.
Boonekamp, P. (2005). Improved methods to evaluate realised energy efficiency. PhD thesis, The Netherlands, University of Utrecht.
Dunn, W. (2003). Public policy analysis an introductory (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
EC (2003). European energy and transport: Trends to 2030. Brussels: European Commission.
EC (2006a). Action plan for energy efficiency: Realising the potential. Brussels: European Comission 19.10.2006 COM(2006)545 final.
EC (2006b) Directive 2006/32/EC of the European Parliament and the Council of 5 April 2006 on Energy End-use Efficiency and Energy Services and Repealing Council Directive 93/76/EEC.
EC (2007). An energy policy for Europe. Brussels: European Commision 10.1.2007 COM(2007)1.
Ecofys, Lund University, Politecnico di Milano, Wuppertal Institute (2007). Success and failure in energy efficiency policies. Ex-post evaluation of 20 instruments to improve energy efficiency across Europe. Reports published within the framework of the AID-EE Project by Ecofys, Lund University, Politecnico di Milano, Wuppertal Institute (http://www.aid-ee.org).
Goldstone, S., Rufo, M., & Wilson, J. (2000). Applying a theory-based approach to California’s non-residential standard performance contract program. Lessons learned by Goldstone and John Wilson, California Energy Commission; and Michael Rufo, Xenergy Inc. from the 2000 ACEEE Summer Study on Energy Efficiency in Buildings.
Hall, N., Roth, J., & Best, C. (2006). California energy efficiency evaluation protocols: Technical, methodological, and reporting requirements for evaluation professionals. Prepared for the California Public Utilities Commission by The TecMarket Works Team, Oregon, USA, April 2006.
Harmelink, M., Joosen, S., & Blok, K. (2005). The theory-based policy evaluation method applied to the ex-post evaluation of climate change policies in the built environment in The Netherlands. Paper presented at the ECEEE Summer Study 2005, May.
Harmelink, M., Joosen, S., Eising, K., & de Visser, E. (2006). Mid-term evaluation of the programme on non-CO 2 greenhouse gases. The Netherlands: Ecofys.
Harmsen, R., van den Hoed, R., & Harmelink, M. (2007) Improving the energy efficiency of private road transport: the case of Ecodriving in the Netherlands. Paper presented at the ECEEE Summer Study 2007, France, June.
IEA (2005). Evaluating energy efficiency policy measures & DSM Programmes. Volume I: Evaluation guidebook. International Energy Agency Implementing Agreement on Demand-Side Management Technologies and Programmes.
Joosen, S., & Harmelink, M. (2006). Guidelines for the ex-post evaluation of 20 energy efficiency instruments applied across Europe. Report published within the framework of the AID-EE Project by Ecofys, Lund, Politecnico, Wuppertal. Retrieved from http://www.aid-ee.org.
Joosen, S., Harmelink, M., & Blok, K. (2004). Evaluation of climate change policies in the built environment in The Netherlands for the period 1995–2002. The Netherlands: Ecofys.
Khan, J., Nordqvist, J. (2007). Success and failure in the promotion of an increased energy efficiency in industry. Paper presented at the ECEEE Summer Study 2007, France.
Megdal, L. (2005). Using program logic model analysis to evaluate & better deliver what works. Paper presented at the ECEEE Summer Study 2005, May.
Michelsen, C. (2005). Exploring policy space: Interactions between policy instruments on household energy efficiency. The case of domestic cold appliances in Germany. Sweden: Lund, IIIEE (September 2005, Reports 2005:04).
Nordqvist J. (2007). The top-runner policy concept: pass it down. Paper presented at the ECEEE Summer Study 2007, France.
NYSERDA (2008). Retrieved February 13, 2008 from http://www.nyserda.org/default.asp.
Reed, J. H., Jordan, G., & Vine, E. (2007). Impact evaluation framework for technology deployment programs. Prepared for the US DOE by Innovol Sandia National Laboratories ogie LLC. Lawrence Berkeley National Laboratory: Washington, USA.
Rossi, P., Lipsey, M., & Freeman, H. (2004). Evaluation; A systematic approach (7th ed.). Thousand Oaks, CA, USA: Sage.
Rufo, M., Prahl, R., & Landry, P. (1999) Evaluation of the 1998 California non-residential standard performance contracting program: A theory-driven approach, evaluation in transition: Working in a competitive energy industry environment. Proceedings of the 1999 Energy Evaluation Conference, Denver.
SRC (2001). A European ex-post evaluation guidebook for DSM and EE Service Programmes. Denmark: SRC International.
Acknowledgment
The authors would like to thank the other project partners of the AID-EE Project, the Wuppertal Institute for Climate, Environment and Energy, and the Politecnico de Milano for their participation and contributions to the results of the AID-EE Project. Without their efforts, we would not have been able to compile this paper. We also thank two anonymous reviewers for valuable comments.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Harmelink, M., Nilsson, L. & Harmsen, R. Theory-based policy evaluation of 20 energy efficiency instruments. Energy Efficiency 1, 131–148 (2008). https://doi.org/10.1007/s12053-008-9007-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12053-008-9007-9