Responsible Research and Innovation (RRI): A Critical Reflection Toward Evaluation Standards

  • Ilse Marschalek
  • Katharina Handler
  • Margit HoferEmail author
  • Maria Schrammel
  • Elisabeth Unterfrauner
Living reference work entry



In order to address todays’ grand challenges and to fulfil ambitious goals, such as excellent science, competitive industries, and a better society, a number of initiatives have been undertaken within the European Union. New frames are required to reach these ambitious goals and to achieve a better alignment of research and innovation with societal and environmental needs. The concept of Responsible Research and Innovation (RRI) is such a new framework. The importance of this comprehensive approach is also indicated by the incorporation of the concept into the EU research funding program Horizon 2020. Future research and innovation activities should become aware of this evolving concept and imply its core aspects and processes. However, there is not yet a commonly agreed definition of RRI or a clear understanding of its implementation in practice (Wickson and Carew 2014; Eden et al. 2013). To enable a common understanding of RRI and to provide a concept which can be implemented by a variety of stakeholders in a range of projects by organizations of different sizes, structures, and goals, there is the need for flexibility of the concept. At the same time, quality criteria and indicators are required to enable a proper and comparable evaluation of RRI. The following chapters give a brief history of the RRI notion and exemplify state-of-the-art discussions on assessing RRI which reflect a tension between benchmarking and “one-fits-all” approaches and flexible and adaptable evaluation models.

Towards an RRI Definition

Brief History

Although RRI gained prominence only recently, it has been already discussed in various forms for more than a decade. A first conceptualization of responsible innovation had appeared within the discourse of nanotechnologies, first in the United States but immediately followed by the European Commission (EC) in 2004 with a communication “Towards a European strategy for nanotechnology” and later with a code of conduct for responsible nanosciences and nanotechnologies research (COM 2004; Commission recommendation 2009). As nanotechnologies and other emerging technologies advanced, a broader discourse on possible consequences of research and more accountable science and innovation had started. RRI thus builds on a longer history of reflection on research and innovation, such as technology assessment (TA) and science and technology studies (STS), but extends its scope of consideration even beyond (Grunwald 2011). Formally, the starting ground for RRI was laid in 2001 with the formulation of the “Science and Society Action Plan” to foster communication between science and society which later, in 2007, was further shaped into the “Science in Society” program in the FP 7 (Framework Programme, funded by the EC). RRI as concept was firstly mentioned in 2010 and became an overarching strategic guiding principle in the EC latest research framework programme Horizon 2020. Finally it was further confirmed in the recent Rome Declaration on Responsible Research and Innovation in Europe which says that “Responsible Research and Innovation (RRI) is the on-going process of aligning research and innovation to the values, needs and expectations of society” (SiS-RRI Conference 2014).

However, it is still unclear what exactly should be understood by the term and what the concept should entail. Different definitions and interpretations have been offered already, and some current EC-funded projects are working toward a common understanding and suggestions for implementation.

The first and most widely used definition of RRI is the one by Von Schomberg (2011) in which RRI is defined as “… transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view on the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society)” (p. 48). Mainly, in this definition, RRI is characterized as a process. According to Von Schomberg’s understanding, not only the products (i.e., desirable outcomes) are important but also the way how they are produced. Both societal actors and innovators should be involved in this process.

Partly building on Von Schomberg’s definition of RRI, another definition is proposed by Owen, Macnaghten, and Stilgoe. It states that “Responsible [Research and] innovation is a collective commitment of care for the future through responsive stewardship of science and innovation in the present” (Owen et al. 2013, p. 36). Although rather vaguely formulated as intended by the authors, it offers some suggestions to the evaluation questions: Being responsive should answer how these processes should be carried out, and “the care for the future” and “a collective commitment” would be the goals to achieve. As they hint to the present, one could also understand the current generations as those who should be engaged in RRI. Like Von Schomberg’s definition, this also implies an active role of both those involved and those affected by research and innovation now and in the future.

Stahl is putting responsibility at the core of their account. They consider RRI not as an additional field of responsibility but as a higher-level responsibility or meta-responsibility “that aims to shape, maintain, develop, coordinate and align existing and novel research and innovation-related processes, actors and responsibilities with a view to ensuring desirable and acceptable research outcomes” (Stahl 2013, p. 708). According to Stahl, fundamental normative principles are needed in order to evaluate whether research or innovation activities are desirable or acceptable.

Other definitions of RRI might slightly differ and offer other interpretations. However, as described by Wickson and Carew (2014), a common accordance of these varying definitions in four main characteristics is that RRI should (Wickson and Carew 2014) focus on significant socio-ecological needs and challenges; (Eden et al. 2013) actively engage a range of stakeholders; (COM 2004) anticipate potential problems, assess available alternatives, and reflect on underlying values, assumptions, and beliefs; and (Commission recommendation 2009) act and adapt accordingly to these ideas. Wickson and Carew are considering RRI mainly as a process, although it also contains elements that relate to preconditions, products, and people.

A Holistic Approach

The RRI Tools project (, an EU-funded project, which aims at further elaborating the notion of RRI and promoting it throughout Europe and beyond, undertook the attempt to address all different dimensions of RRI and to incorporate them into a holistic perception of RRI.

The RRI Tools project adopted Von Schomberg’s definition into: “Responsible Research and Innovation is a dynamic, iterative process by which all stakeholders involved in the R&I practice become mutually responsive and share responsibility regarding both the outcomes and process requirements” (RRI Tools) (Klaassen et al. 2014, p. 4).

In other words, research and innovation can only be considered as being “responsible” if they cover the formulated key dimension as outlined by the EC and are encompassing both particular outcomes and certain process requirements as described as follows.

Policy Agendas

The EC has identified six key dimensions or “keys” for the RRI framework. According to the RRI Tools project, each of them should be seen as powerful policy agenda with its own potential to realize the defined process requirements and outcomes (European Union 2012) as shown in Fig. 1 and described below.
Fig. 1

Six RRI key dimensions: policy agendas. (Source: P Klassen, F Kupper, M Rijen, S Vermeulen, J Broerse: 2014. ‘Policy Brief on the State of the Art on RRI and a Working Definition of RRI’. RRI Tools report.

  • Public Engagement: Engagement of all societal actors (researchers, industry, policymakers, civil society) in a reflective research process

  • Gender: Gender equality

  • Open Access: Open access to research results and publications to boost innovation and increase the use of scientific results

  • Ethics: Research must respect ethical standards and fundamental rights to respond to societal challenges

  • Science Education: Enhancement of current education processes to better equip future researchers and society as a whole with the necessary competences to participate in research processes

  • Governance: Governance of policymakers to prevent harmful or unethical developments in research and innovation


As the keys could not cover all RRI aspects, as projected within RRI definitions, additional thematic categorization of RRI outcomes is necessary to describe the complex issues.

Addressing these policy agendas might lead to the following outcomes divided in three main categories (Fig. 2):
Fig. 2

Desired outcomes. (Source: P Klassen, F Kupper, M Rijen, SVermeulen, J Broerse: 2014. ‘Policy Brief on the State of the Art on RRI and a Working Definition of RRI’. RRI Tools report.

Learning outcomes should lead to empowered and responsible actors across the whole range of stakeholders including civil society and the general public. Structures and organizations should create opportunities and create support for actors. The outcomes should be ethically acceptable, desirable, and sustainable. Solutions should be found in opening up science through continuous and meaningful deliberation with societal actors. The topics should by all means address the seven grand societal challenges as formulated by the European Commission as one of the pillars of the Horizon 2020 programme. (The seven priority challenges are the following fields where targeted investment in research and innovation can have a real impact benefitting the citizen: health, demographic change, and well-being; food security, sustainable agriculture and forestry, marine and maritime and inland water research, and the bioeconomy; secure, clean, and efficient energy; smart, green, and integrated transport; climate action, environment, resource efficiency, and raw materials; Europe in a changing world, inclusive, innovative, and reflective societies; and secure societies – protecting freedom and security of Europe and its citizens.

Process Requirements

In order to achieve these outcomes, Responsible Research and Innovation processes have to consider certain process requirements. The RRI Tools project identified eight such requirements and divided them into four clusters (Klaassen et al. 2014):
  1. 1.

    Diversity and inclusion

  2. 2.

    Anticipation and reflection

  3. 3.

    Openness and transparency

  4. 4.

    Responsiveness and adaptive change

All three axes, including policy agendas, outcomes, and process requirements, built the basis for the formulation of the first set of quality criteria of RRI (Fig. 3).
Fig. 3

Quality criteria (Hofer and Schrammel 2015). (Source: Hofer and Schrammel 2015)

These quality criteria are then further broken down into the different manifestations, e.g., “sufficient amount” of engaged stakeholders. Rather than asking for simple yes/no answers, questions should stimulate critical reflection on the specific criteria. The approach of raising critical questions was considered as being most appropriate for (self-)evaluation purposes.

Indicators for Responsible Research and Innovation and the Difficulty of Evaluating RRI

So far, evaluation designs did not include RRI aspect as such; therefore evaluations are not targeted toward RRI either. Consequently, in order to guarantee an evaluation of RRI, the implementation of RRI in the original project or research plan has to be made sure. In other words, “if the concept of RRI and the evaluation of it shall be implemented on a sustainable basis, it needs to be explicitly requested by the funding bodies” (Marschalek et al. 2014, p. 24). Accordingly it needs adapted evaluation concepts that include RRI aspects.

There is an ever-pressing need for evaluation criteria for RRI by national and international funding bodies to guide them in the selection of research proposals, as well as by all different kinds of stakeholder groups who aim at implementing RRI and are eager to monitor their compliance with the RRI notion. However, there are a few challenges to face in the development of evaluation criteria and indicators for RRI.

Fundamental questions in this respect are as follows: who and how can someone judge whether RRI is complete, that all principles of RRI have been followed? How can benchmarks be established? Are there minimum criteria for RRI?

While there seems to be an agreement that benchmarking criteria for RRI is not a viable approach since it is difficult to tackle RRI this way and also this would rather contradict the spirit of RRI (Owen et al. 2013; Kupper et al. 2015; Stilgoe 2012), the question of how to evaluate RRI in practice after all is still unanswered. Currently the RRI discourse of the experts and the respective projects has launched a discussion around this question in different settings and environments. In addition, evaluation concepts have to rely on the interpretations of the RRI concepts different experts consider (Marschalek et al. 2014, p. 23).

Wickson and Carew (2014) have proposed a rubric for assessing RRI which differentiates between different compliance levels from routine to exemplary. Although a promising approach, the scheme bears some blind spots. It is not congruent with the key dimensions as defined by the European Commission (Jacob et al. 2013). Two of the six key dimensions are not reflected in the RRI scheme, namely, gender equality and science education. The best proof of concept, after all, is testing it in practice with all relevant stakeholders, but according to our knowledge, so far, no scientific results or reports were published that would emphasize the usability of the concept. While some refer to it as being a promising example to RRI assessment (e.g., Sparks project (, others define it only as a good starting point that lacks the direct link with innovation projects (Flipse et al. 2015).

The European Commission has set up an expert group that worked on policy indicators for RRI (European Commission 2015). In addition to the six already defined key dimensions, they added further two, namely, sustainability and inclusion, while they found the key dimension of governance on a different level compared to the other dimensions. The expert group came to the conclusion that it cannot offer a criteria list for all but that a smaller set of indicators shall be chosen which is then tailored to the different needs.

Similarly, one of the core aims of the EU-funded MoRRI project (Ravn et al. 2015) (Monitoring the Evolution and Benefits of Responsible Research and Innovation) was to measure RRI and to come to clear indicators. The initial 98 tentative RRI indicators were assessed systematically according to relevance, robustness, and data richness in covering RRI, which resulted in 36 indicators for RRI. These indicators distinguish between several levels such as global, national, program/project, and individual level. However, aggregating all the data of the 36 indicators seems quite challenging as some indicators rely on secondary while others rely on primary data with each owns’ complexities. Its usefulness lies more in the fact that it reveals valuable indicators for specific questions that relate to RRI, for instance, which questions could be used to analyze research ethics at university level as it was also the intention of the authors to provide “individual measures of RRI at specified dimensions and levels of analysis” (p. 9).

From Assessment of RRI to Reflection on RRI

So far the various attempts to operationalize RRI and to provide indicators for RRI rarely found its way into practice because they were either too complex or vaguely defined or simply did not tackle the complexity of the notion.

However, there seems to be an agreement that any evaluation scheme of RRI has to be flexible and adaptive. Thus, if indicators are applied, not all indicators will necessarily be relevant. An evaluation concept would have to be broken down into an operationalization that offers a toolbox of mixed methods.

In this respect, the role of evaluation would not only be the justification of resources spent but to provide a framework for identifying the top RRI performances. The discourse on RRI should not be limited to certain topics but allow for an actual discussion on change. The evaluation of RRI could influence this possible danger by fostering methods that enable critical discourse and questions.

Self-reflection could play an important role in evaluating RRI. If sustainable implementation shall be reached, researchers and institutions need to be supported in understanding and adapting the fundamental concept of RRI and its evaluation criteria.

Anticipation and reflection as core process requirements are considered as crucial in RRI. The RRI Tools project developed the so-called Self-Reflection Tool that respects these core principles and that constitutes one attempt toward moving from assessing RRI to reflect upon RRI. Based on the discourse on indicators for RRI and the quality criteria catalogue developed within the RRI Tools project, the Self-Reflection Tool has been developed with the aim to stimulate reflection on different aspects of RRI.

Instead of providing a set of indicators to measure or assess RRI within teams, organizations or projects independently from their size, stakeholder group addressed, or research field, the development process of the RRI Self-Reflection Tool rather showed the need to stimulate reflection and provide room for individual adaptions and individually defined indicators. As outlined above the concept of RRI leaves this space for flexible adjustment. Different stakeholder groups have different needs and approach RRI in different ways. Tools such as the Self-Reflection Tool can support them to adapt the concept of RRI to their personal needs and to finally start or improve RRI implementation.

Conclusion and Future Directions

In summary, RRI leaves room for further development and improvement when it comes to the application of a sound evaluation concept. The development of the evaluation framework on RRI goes hand in hand with the development of the RRI concept itself: it requires a commonly agreed understanding of RRI, a comprehensive and consequent but still flexible concept, on which a general commitment of all stakeholders have to build on. Recommendations for actions could be guiding principles, illustrated by examples of promising practices. A set of RRI tools for implementation has to be established, offering different means of examples, supporting documentations of practices and methods applicable for various target groups, topics and contexts.

Elaborated lists of quality criteria, as presented, for instance, by the RRI tools project should not be considered as a fixed set of evaluation principles which are carved out in stone, but flexible and adaptable. Secondly “such lists cannot be sufficient for either evaluation or designing research and innovation with an eye to responsibility. Taking such lists to be satisfying increases the risk that the bigger picture is lost out of sight – in that case the project would suffer, in other words, from a lack of vision.

And indeed, what distinguishes thoroughly responsible research and innovation practices from other such practices is the way in which the integration of process requirements and outcomes transpires in them” (Kupper et al. 2015, p. 41).

What still lacks is a RRI narrative on European level for a better understanding of the approach, the concept, and its implementation. There is a need for better describing the idea and also containing some recommendations for actions. This is vital to foster the urgently needed cultural change toward a holistic understanding of RRI and its connected evaluation framework. A paradigm shift toward a code of conduct to which all stakeholders are self-responsibly committed could be envisaged.

Rather, a common understanding of basic values should inform Responsible Research and Innovation practices. The question is how these values could be translated into concrete norms that can be used for the assessment of research and innovation practices (Kupper et al. 2015, p. 42).

The integration of the different interests and needs of all stakeholders might contribute well to define the different normative values and possibly also the connected evaluation. Especially the integration of the general public might enrich these efforts for a sound evaluation framework. They might bring in different views, needs, and requirements that would also influence the set of quality criteria of research and innovation. However, this fact raises the fear of many researchers that research and innovation might suffer from unjustified limitations. To open up for relevant topics and the definition of demands for action needs to be carried out in bottom-up approaches. The future will show how to deal with different stakeholders and conflicting interest in this framework and if their active involvement can also contribute to settle a sound evaluation framework for research and innovation.



  1. COM. Towards a European strategy for nanotechnology. Brussels: EC; 2004. (Communication from the Commission).Google Scholar
  2. Commission recommendation on a code of conduct for responsible nanosciences and nanotechnologies research & Council conclusions on Responsible nanosciences and nanotechnologies research [Internet]. Luxembourg: Publications Office of the European Union; 2009. Report No.: EU§ 23906.
  3. Eden G, Jirotka M, Stahl B. Responsible research and innovation: critical reflection into the potential social consequences of ICT. In: IEEE; 2013 [cited 2015 Jul 6]. p. 1–12.
  4. European Commission. Indicators for promoting and monitoring responsible research and innovation. Report from the Expert Group on Policy Indicators for Responsible Research and Innovation [Internet]. Brussels; 2015.
  5. European Union, editor. Responsible research and innovation – Europe’s ability to respond to societal challenges [Internet]. 2012.
  6. Flipse S, Vrielink JO, Sanden van der M. Building interactive communication tools to support interdisciplinary responsible innovation. J Innov Manag. 2015;3: 119–33.CrossRefGoogle Scholar
  7. Grunwald A. Responsible innovation: bringing together technology assessment, applied ethics, and STS research. Enterp Work Innov Stud. 2011;7:9–31.Google Scholar
  8. Hofer M, Schrammel M. Developing tools for RRI. In: Proceedings STS conference Graz 2015 [Internet]. Graz; 2015. (IAS-STS).
  9. Jacob K, Van Den Hoven J, Nielsen L, Roure F, Rudze L, Stilgoe J, et al. Options for strengthening responsible research and innovation: report of the expert group on the state of the art in Europe on responsible research and innovation. Brussels: European Commission; 2013.Google Scholar
  10. Klaassen P, Kupper F, Broerse J. Polcy brief on the state of the art on RRI and a working definition of RRI [Internet]. Amsterdam: Athena Institute, VU University Amsterdam; 2014 Oct. (RRI tools). Report No.: D 1.1.
  11. Kupper F, Klaassen P, Rijnen M, Vermeulen S, Broerse J. D1.3 Report on the quality criteria of Good Practice Standards in RRI. 2015.Google Scholar
  12. Marschalek I, Handler K, Hofer M, Voigt C. Report on ‘Coverage of RRI aspects in STI (Science, technology, innovation) evaluations’. Vienna: Centre for Social Innovation; 2014. (RRI tools). Report No.: D 5.1.Google Scholar
  13. Owen R, Stilgoe J, Macnaghten P, Gorman M, Fisher E, Guston D. A framework for responsible innovation. In: Responsible innovation. Managing the responsible emergence of science and innovation in society. John Wiley & Sons, Ltd; 2013. p. 27–50.Google Scholar
  14. Ravn T, Nielsen WM, Mejlgaard N. Metrics and indicators of responsible research and innovation. MoRRI Report [Internet]. 2015.
  15. SiS-RRI Conference. Rome Declaration on Responsible Research and Innovation in Europe [Internet]. 2014.
  16. Stahl BC. Responsible research and innovation: The role of privacy in an emerging framework. Sci Public Policy. 2013;40(6):708–16.CrossRefGoogle Scholar
  17. Stilgoe J. Responsible research and innovation: from science in society to science for society, with society. Sci Public Policy. 2012;39(6):751–60.CrossRefGoogle Scholar
  18. Von Schomberg R. Prospects for technology assessment in a framework of responsible research and innovation. In: Technikfolgen abschätzen lehren: Bildungspotenziale transdisziplinärer Methode. Wiesbaden: Springer VS; 2011. p. 39–61.Google Scholar
  19. Wickson F, Carew AL. Quality criteria and indicators for responsible research and innovation: learning from transdisciplinarity. J Responsib Innov. 2014;1(3): 254–73.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Ilse Marschalek
    • 1
  • Katharina Handler
    • 1
  • Margit Hofer
    • 1
    Email author
  • Maria Schrammel
    • 1
  • Elisabeth Unterfrauner
    • 1
  1. 1.Centre for Social InnovationViennaAustria

Section editors and affiliations

  • David F. J. Campbell
    • 1
    • 2
    • 3
    • 4
  1. 1.Unit for Quality Enhancement (UQE)University of Applied Arts ViennaViennaAustria
  2. 2.Faculty for Interdisciplinary Studies (iff), Institute of Science Communication and Higher Education Research (WIHO)Alpen-Adria-University KlagenfurtViennaAustria
  3. 3.Department of Political ScienceUniversity of ViennaViennaAustria
  4. 4.Department for Continuing Education Research and Educational Management, Centre for Educational Management and Higher Education DevelopmentDanube University KremsKremsAustria