Skip to main content

Advertisement

SpringerLink
Log in

Using Evaluation Research as a Means for Policy Analysis in a ‘New’ Mission-Oriented Policy Context

  • Published:
Minerva Aims and scope Submit manuscript

Abstract

Grand challenges stress the importance of multi-disciplinary research, a multi-actor approach in examining the current state of affairs and exploring possible solutions, multi-level governance and policy coordination across geographical boundaries and policy areas, and a policy environment for enabling change both in science and technology and in society. The special nature of grand challenges poses certain needs in evaluation practice: (a) the need for learning at the operational, policy and, especially, system level; and (b) the importance of a wider set of impacts and behavioural change. The examination of the usefulness of evaluations as learning tools thus becomes relevant as does the way current evaluation practices address broader impacts and issues such as behavioural additionality. The suitability of existing evaluation contexts in meeting the specific issues posed by the ‘grand challenges’ orientation is also worth examining. The paper argues that learning at the policy and system levels is largely unaddressed while concepts such as behavioural additionality are still underexploited.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. The Europe 2020 Strategy is explicitly oriented towards tackling grand societal challenges and has formulated ambitious policy objectives for climate change, energy security, demographic ageing and resource efficiency.

  2. The conference "New Worlds—New Solutions" in Lund saw 350 researchers, research funding agencies, industrialists and politicians from all over Europe gathered together to discuss how European research should be developed. The participants agreed on a document, “The Lund Declaration”, which was submitted to Sweden’s Higher Education and Research Minister. http://www.vr.se/inenglish/aboutus/policies/lunddeclaration.4.44482f6612355bb5ee780003083.html. Accessed 2 November 2012.

  3. The main features of grand challenges are summarised in the left column of Fig. 2.

  4. This database has been constructed with the support of European Commission and the large project team included the authors of this paper as well as others. More information on this database can be found at http://www.proinno-europe.eu/page/inno-appraisal and other studies utilising this database including Edler et al. (2012); Gök and Edler (2012).

References

  • Abma, T.A., and R.E. Stake. 2001. Stake’s responsive evaluation: Core ideas and evolution. New Directions for Evaluation, Special Issue: Responsive Evaluation 92: 7–22.

    Article  Google Scholar 

  • Alkin, Marvin C. (ed.). 2012. Evaluation Roots: Tracing Theorists’ Views and Influences. Thousand Oaks: Sage.

    Google Scholar 

  • Boden, M., C. Cagnin, V. Carabias, K. Haegeman, and T. Konnola. 2010. Facing the future: time for the EU to meet global challenges. Luxembourg: Publications Office of the European Union. EUR 24364 EN, ISSN 1018-5593, ISBN 978-92-79-15786-8. doi:10.2791/4223. http://ftp.jrc.es/EURdoc/JRC55981.pdf. Accessed October 2011.

  • Buisseret, T.J., H.M. Cameron, and L. Georghiou. 1995. What Difference Does It Make—Additionality in The Public Support Of R&D In Large Firms. International Journal of Technology Management 10(4–6): 587–600.

    Google Scholar 

  • Cagnin, C., D. Loveridge, and O. Saritas. 2011. FTA and equity: New approaches to governance. Futures 43: 279–291.

  • Cagnin, C., Effie Amanatidou, and M. Keenan. 2012. Orienting European Innovation Systems towards Grand Challenges and the Roles that FTA Can Play. Science and Public Policy 39: 140–152.

    Article  Google Scholar 

  • Cronbach, L.J., and K. Shapiro. 1982. Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.

  • Depledge, M., A. Bartonova, and A. Cherp. 2010. Responsible and transformative innovation for sustainable societies. Fundamental and applied research. Report of the Environment Advisory Group. Brussels: December 2010.

  • Edler, Jakob, Martin Berger, Michael Dinges, and Abdullah Gök. 2012. The practice of evaluation in innovation policy in Europe. Research Evaluation 21(3): 167–182. doi:10.1093/reseval/rvs014.

    Article  Google Scholar 

  • Foray, D., David C. Mowery, and Richard R. Nelson. 2012. Public R&D and social challenges: What lessons from mission R&D programs? Research Policy 41: 1697–1702.

    Article  Google Scholar 

  • Forss, K., C.C. Rebien, and J. Carlsson. 2002. Process Use of Evaluations. Evaluation 8: 29–45.

    Article  Google Scholar 

  • Gassler, H., W. Polt, and C. Rammer. 2008. Priority setting in technology policy – historical developments and recent trends. In Innovation Policy in Europe, eds. C. Nauwelaers, and R. Wintjes. Cheltenham: Edward Elgar.

    Google Scholar 

  • Georghiou, Luke, and Philippe Laredo. 2006. Evaluation of Publicly Funded Research: Recent Trends and Perspectives. In OECD Science, Technology and Industry Outlook, ed. OECD. Paris: OECD Publishing.

  • Georghiou, Luke, and David Roessner. 2000. Evaluating technology programs: tools and methods. Research Policy 29: 657–678.

    Article  Google Scholar 

  • Georghiou, Luke, John Rigby, and Hugh Cameron (eds.). 2002. Assessing the Socio-economic Impacts of the Framework Programme. Manchester: University of Manchester.

    Google Scholar 

  • Gök, Abdullah, and Jakob Edler. 2010. Exploring the Use of Behavioural Additionality. In INNO-Appraisal Final Report, ed. J. Edler. Manchester: Report to EC DG Enterprise and Industry.

    Google Scholar 

  • Gök, Abdullah. 2010. An Evolutionary Approach to Innovation Policy Evaluation: Behavioural Additionality and Organisational Routines. Manchester: University of Manchester.

    Google Scholar 

  • Gök, Abdullah, and Jakob Edler. 2012. The use of behavioural additionality evaluation in innovation policy making. Research Evaluation 21(4): 306–318. doi:10.1093/reseval/rvs015.

    Article  Google Scholar 

  • Guba, E.G., and Y.S. Lincoln. 1989. Fourth Generation Evaluation. U.S.A: Sage Publications.

    Google Scholar 

  • Edelenbos, J., and A. van Buuren. 2005. The Learning Evaluation: A Theoretical and Empirical Exploration. Evaluation Review 29: 591–611.

    Article  Google Scholar 

  • European Commission. 2010. EUROPE 2020 A strategy for smart, sustainable and inclusive growth. COM(2010) 2020, 3.3.2010. Brussels: European Commission.

  • House, E.R., and K.R. Howe. 1998. The Deliberative Democratic View. Chicago: Presentation at the annual Meeting of the American Evaluation Association.

    Google Scholar 

  • KED/VINNOVA. 2004. Public Research and Innovation Policy for the Good of Society: How To Assess The Way Forward? Stockholm.

  • Lähteenmäki-Smith, K. (ed.). 2007. Learning through Evaluation: the Nordic Experience. Nordregio Report 2007:3.

  • McDavid, James C., and Laura R.L. Hawthorn. 2006. Program Evaluation & Performance Measurement. Thousand Oaks: Sage.

    Google Scholar 

  • Miles, I., and P. Cunningham. 2006. A Practical Guide to Evaluating Innovation Programmes. Brussels: ECSC-EC-EAEC.

    Google Scholar 

  • Millennium Project. 2007. Global Challenges Facing Humanity. http://www.millennium-project.org/millennium/Global_Challenges/chall-08.html. Accessed 29th march 2013. Accesses October 2011.

  • Patton, M.Q. 2007. Utilization-focused evaluation: the new century text. Thousand Oaks: Sage Publ.

    Google Scholar 

  • Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. 2004. Evaluation: A Systemic Approach, 7th ed. Thousand Oaks: Sage.

    Google Scholar 

  • Scriven, Michael. 1991. Evaluation Thesaurus. Newbury Park: Sage.

  • Shadish, William R., Thomas D. Cook, and Laura C. Leviton. Foundations of program evaluation: Theories of practice. Thousand Oaks, CA: Sage, 1991.

  • Shadish, W.R., T.D. Cook Jr, and L.C. Leviton. 2001. Foundations of Program Evaluation: Theories of Practice. Newbury Park. CA: Sage.

    Google Scholar 

  • Shapira, P., and S. Kuhlmann. (eds). 2001. Learning from Science and Technology Policy Evaluation. In Proceedings from the 2000 US-EU Workshop on Learning from Science and Technology Policy Evaluation. Bad Herrenalb, Germany.

  • Stufflebeam, Daniel L., and Anthony J. Shinkfield. 2007. Evaluation Theory, Models & Applications. San Francisco: Jossey-Bass.

    Google Scholar 

  • Weber, Matthias, and Luke Georghiou. 2010. Dynamising innovation policy: Giving innovation a central role in European policy. The results of a Foresight Workshop organised as part of the FP7 Blue Skies Project FarHorizon 27–28 May 2010. Brussels: European Commission.

  • Weber, Matthias K., and Harald Rohracher. 2012. Legitimizing research, technology and innovation policies for transformative change. Combining insights from innovation systems and multi-level perspective in a comprehensive “failures” framework. Research Policy 41: 1037–1047.

Download references

Author information

Authors and Affiliations

  1. Manchester Institute of Innovation Research, Manchester Business School, University of Manchester, Oxford Road, Manchester, M13 9PL, UK

    Effie Amanatidou, Paul Cunningham & Abdullah Gök

  2. Systasi Consulting, 9th Klm, 57001, Thessaloniki-Thermi, Greece

    Ioanna Garefi

Authors
  1. Effie Amanatidou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Cunningham
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdullah Gök
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ioanna Garefi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Effie Amanatidou.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Amanatidou, E., Cunningham, P., Gök, A. et al. Using Evaluation Research as a Means for Policy Analysis in a ‘New’ Mission-Oriented Policy Context. Minerva 52, 419–438 (2014). https://doi.org/10.1007/s11024-014-9258-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11024-014-9258-x

Keywords

Search

Navigation