, Volume 52, Issue 4, pp 419–438 | Cite as

Using Evaluation Research as a Means for Policy Analysis in a ‘New’ Mission-Oriented Policy Context

  • Effie AmanatidouEmail author
  • Paul Cunningham
  • Abdullah Gök
  • Ioanna Garefi


Grand challenges stress the importance of multi-disciplinary research, a multi-actor approach in examining the current state of affairs and exploring possible solutions, multi-level governance and policy coordination across geographical boundaries and policy areas, and a policy environment for enabling change both in science and technology and in society. The special nature of grand challenges poses certain needs in evaluation practice: (a) the need for learning at the operational, policy and, especially, system level; and (b) the importance of a wider set of impacts and behavioural change. The examination of the usefulness of evaluations as learning tools thus becomes relevant as does the way current evaluation practices address broader impacts and issues such as behavioural additionality. The suitability of existing evaluation contexts in meeting the specific issues posed by the ‘grand challenges’ orientation is also worth examining. The paper argues that learning at the policy and system levels is largely unaddressed while concepts such as behavioural additionality are still underexploited.


Mission-oriented research and innovation policy Usefulness Utility Evaluation Behavioural additionality 


  1. Abma, T.A., and R.E. Stake. 2001. Stake’s responsive evaluation: Core ideas and evolution. New Directions for Evaluation, Special Issue: Responsive Evaluation 92: 7–22.CrossRefGoogle Scholar
  2. Alkin, Marvin C. (ed.). 2012. Evaluation Roots: Tracing Theorists’ Views and Influences. Thousand Oaks: Sage.Google Scholar
  3. Boden, M., C. Cagnin, V. Carabias, K. Haegeman, and T. Konnola. 2010. Facing the future: time for the EU to meet global challenges. Luxembourg: Publications Office of the European Union. EUR 24364 EN, ISSN 1018-5593, ISBN 978-92-79-15786-8. doi: 10.2791/4223. Accessed October 2011.
  4. Buisseret, T.J., H.M. Cameron, and L. Georghiou. 1995. What Difference Does It Make—Additionality in The Public Support Of R&D In Large Firms. International Journal of Technology Management 10(4–6): 587–600.Google Scholar
  5. Cagnin, C., D. Loveridge, and O. Saritas. 2011. FTA and equity: New approaches to governance. Futures 43: 279–291.Google Scholar
  6. Cagnin, C., Effie Amanatidou, and M. Keenan. 2012. Orienting European Innovation Systems towards Grand Challenges and the Roles that FTA Can Play. Science and Public Policy 39: 140–152.CrossRefGoogle Scholar
  7. Cronbach, L.J., and K. Shapiro. 1982. Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.Google Scholar
  8. Depledge, M., A. Bartonova, and A. Cherp. 2010. Responsible and transformative innovation for sustainable societies. Fundamental and applied research. Report of the Environment Advisory Group. Brussels: December 2010.Google Scholar
  9. Edler, Jakob, Martin Berger, Michael Dinges, and Abdullah Gök. 2012. The practice of evaluation in innovation policy in Europe. Research Evaluation 21(3): 167–182. doi: 10.1093/reseval/rvs014.CrossRefGoogle Scholar
  10. Foray, D., David C. Mowery, and Richard R. Nelson. 2012. Public R&D and social challenges: What lessons from mission R&D programs? Research Policy 41: 1697–1702.CrossRefGoogle Scholar
  11. Forss, K., C.C. Rebien, and J. Carlsson. 2002. Process Use of Evaluations. Evaluation 8: 29–45.CrossRefGoogle Scholar
  12. Gassler, H., W. Polt, and C. Rammer. 2008. Priority setting in technology policy – historical developments and recent trends. In Innovation Policy in Europe, eds. C. Nauwelaers, and R. Wintjes. Cheltenham: Edward Elgar.Google Scholar
  13. Georghiou, Luke, and Philippe Laredo. 2006. Evaluation of Publicly Funded Research: Recent Trends and Perspectives. In OECD Science, Technology and Industry Outlook, ed. OECD. Paris: OECD Publishing.Google Scholar
  14. Georghiou, Luke, and David Roessner. 2000. Evaluating technology programs: tools and methods. Research Policy 29: 657–678.CrossRefGoogle Scholar
  15. Georghiou, Luke, John Rigby, and Hugh Cameron (eds.). 2002. Assessing the Socio-economic Impacts of the Framework Programme. Manchester: University of Manchester.Google Scholar
  16. Gök, Abdullah, and Jakob Edler. 2010. Exploring the Use of Behavioural Additionality. In INNO-Appraisal Final Report, ed. J. Edler. Manchester: Report to EC DG Enterprise and Industry.Google Scholar
  17. Gök, Abdullah. 2010. An Evolutionary Approach to Innovation Policy Evaluation: Behavioural Additionality and Organisational Routines. Manchester: University of Manchester.Google Scholar
  18. Gök, Abdullah, and Jakob Edler. 2012. The use of behavioural additionality evaluation in innovation policy making. Research Evaluation 21(4): 306–318. doi: 10.1093/reseval/rvs015.CrossRefGoogle Scholar
  19. Guba, E.G., and Y.S. Lincoln. 1989. Fourth Generation Evaluation. U.S.A: Sage Publications.Google Scholar
  20. Edelenbos, J., and A. van Buuren. 2005. The Learning Evaluation: A Theoretical and Empirical Exploration. Evaluation Review 29: 591–611.CrossRefGoogle Scholar
  21. European Commission. 2010. EUROPE 2020 A strategy for smart, sustainable and inclusive growth. COM(2010) 2020, 3.3.2010. Brussels: European Commission.Google Scholar
  22. House, E.R., and K.R. Howe. 1998. The Deliberative Democratic View. Chicago: Presentation at the annual Meeting of the American Evaluation Association.Google Scholar
  23. KED/VINNOVA. 2004. Public Research and Innovation Policy for the Good of Society: How To Assess The Way Forward? Stockholm.Google Scholar
  24. Lähteenmäki-Smith, K. (ed.). 2007. Learning through Evaluation: the Nordic Experience. Nordregio Report 2007:3.Google Scholar
  25. McDavid, James C., and Laura R.L. Hawthorn. 2006. Program Evaluation & Performance Measurement. Thousand Oaks: Sage.Google Scholar
  26. Miles, I., and P. Cunningham. 2006. A Practical Guide to Evaluating Innovation Programmes. Brussels: ECSC-EC-EAEC.Google Scholar
  27. Millennium Project. 2007. Global Challenges Facing Humanity. Accessed 29th march 2013. Accesses October 2011.
  28. Patton, M.Q. 2007. Utilization-focused evaluation: the new century text. Thousand Oaks: Sage Publ.Google Scholar
  29. Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. 2004. Evaluation: A Systemic Approach, 7th ed. Thousand Oaks: Sage.Google Scholar
  30. Scriven, Michael. 1991. Evaluation Thesaurus. Newbury Park: Sage.Google Scholar
  31. Shadish, William R., Thomas D. Cook, and Laura C. Leviton. Foundations of program evaluation: Theories of practice. Thousand Oaks, CA: Sage, 1991. Google Scholar
  32. Shadish, W.R., T.D. Cook Jr, and L.C. Leviton. 2001. Foundations of Program Evaluation: Theories of Practice. Newbury Park. CA: Sage.Google Scholar
  33. Shapira, P., and S. Kuhlmann. (eds). 2001. Learning from Science and Technology Policy Evaluation. In Proceedings from the 2000 US-EU Workshop on Learning from Science and Technology Policy Evaluation. Bad Herrenalb, Germany.Google Scholar
  34. Stufflebeam, Daniel L., and Anthony J. Shinkfield. 2007. Evaluation Theory, Models & Applications. San Francisco: Jossey-Bass.Google Scholar
  35. Weber, Matthias, and Luke Georghiou. 2010. Dynamising innovation policy: Giving innovation a central role in European policy. The results of a Foresight Workshop organised as part of the FP7 Blue Skies Project FarHorizon 27–28 May 2010. Brussels: European Commission.Google Scholar
  36. Weber, Matthias K., and Harald Rohracher. 2012. Legitimizing research, technology and innovation policies for transformative change. Combining insights from innovation systems and multi-level perspective in a comprehensive “failures” framework. Research Policy 41: 1037–1047.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Effie Amanatidou
    • 1
    Email author
  • Paul Cunningham
    • 1
  • Abdullah Gök
    • 1
  • Ioanna Garefi
    • 2
  1. 1.Manchester Institute of Innovation Research, Manchester Business SchoolUniversity of ManchesterManchesterUK
  2. 2.Systasi ConsultingThessaloniki-ThermiGreece

Personalised recommendations