Keywords

1 Introduction

Although all Swiss universities share a strong focus on research, the effective monitoring of quality academic research has yet to be satisfactorily developed. The ‘Mesurer les performances de la recherche’ project was an attempt of the Rectors’ Conference of the Swiss Universities to identify the best ways for Swiss universities to implement a system of research evaluation according to their specific needs and institutional strategy. The project was funded over the period 2008–2011 through project-related subsidies of the Swiss Confederation allocated by the Swiss University Conference. The project was finalized in 2012 and has since been followed by the ‘Performances de la recherche en sciences humaines et sociales’ programme, which will be funded from 2013–2016 through project-related subsidies as well. The main focus of this programme is the visibility of research performance and impact in terms of highlighting the quality and strengths of research in different fields and disciplines. In what follows, we will delimit the scope and intended purposes of the project and the programme while addressing the following five questions:

  • What should be evaluated in research?

  • For what purpose should we evaluate research?

  • How should we evaluate research?

  • What are the ties between evaluation and quality?

  • How can the quality and impact of research be made visible to different stakeholders both within and outside the universities?

We will briefly describe the main features of the project and its results, detail current developments in the on-going programme and then present certain perspectives of swissuniversities on the remaining period of the programme.

2 Making a Variety of Research Visible

2.1 What Should We Evaluate in Research?

Academic research includes a wide array of aspects, from the discovery of new knowledge and promoting young researchers to potential impacts on the scientific community and society. However, the relevance of these aspects to different stakeholders (universities, faculties, researchers, authorities and the public) varies according to disciplinary and institutional differences. Thus, the ‘Mesurer les performances de la recherche’ project paid particular attention to these differences, not only considering the impact of research evaluation on the scientific community, but also disciplinary diversity, the significance of interdisciplinary research, the interaction between research and teaching, technological innovation, and linguistic and cultural specificities, such as language and the form of publication. Many of these differences—like language and the form of publication—are particularly important in the social sciences and humanities (Huang and Chang 2008; Czellar and Lanares 2013).

Therefore, the understanding that all these aspects should be taken into account in research evaluation is one of the main reasons why the ‘Performances de la recherche en sciences humaines et sociales’ programme focuses on specific research circumstances in the humanities and social sciences.

2.2 For What Purpose Should We Evaluate Research?

The evaluation of research requires different levels of focus depending on whether a given body of research addresses authorities, peers, or the public at large. One important purpose of evaluating research is to make research accountable both to political authorities and the public. In this sense, research evaluation plays a major role in developing and adapting the institutional strategies of Swiss universities. At both the individual and institutional levels, attaining knowledge of research strengths and weaknesses is another crucial purpose of research evaluation. Lastly, research evaluation also serves to make quality and, consequently, the importance of research visible for external stakeholders. While the ‘Mesurer les performances de la recherche’ project explored various possibilities for measuring research performance and compared their scope, the ‘Performances de la recherche en sciences humaines et sociales’ programme fosters the development of instruments to increase the visibility of research performance and impact for the benefit of universities and their faculties.

2.3 How Should We Evaluate Research?

Conventional methods of research evaluation, particularly advanced bibliometric analyses based on the Web of Science or Scopus, both of which are online scientific citation indexing services, are quite useful for describing the impact of research in natural sciences, such as chemistry or medicine, within the scientific community (van Leeuwen 2013; Engels et al. 2012).

However, these methods are less useful for describing the social impact of research in the humanities. The ‘Mesurer les performances de la recherche’ project encouraged the exploration and the development of broader approaches that may better suit the needs of different disciplines and reflect the impact of other aspects of research, such as its social relevance or its applied uses, including teaching. The ‘Performance de la recherche en sciences humaines et sociales’ programme builds on the resulting activities of the previous ‘Mesurer les performances de la recherche’ project in order to develop further methods of evaluating research that will pay greater attention to specific circumstances in the humanities and social sciences, such as linguistic characteristics, informal researcher networks and different forms of publication in the respective disciplines.

2.4 Evaluation, Quality and Mission

As the CRUS points out in ‘The Swiss Way to Quality in the Swiss universities’ (CRUS 2008), the quality of research is not an end in itself, but rather is at the service of further aims that are derived from each university’s self-determined strategy regarding its role in Switzerland and the international community. The CRUS underlines this principle while stressing the following aspects:

  1. 1.

    The CRUS recognizes that member universities are bound by different missions as established by their respective responsible bodies. The CRUS is therefore convinced that each university is responsible for setting its own strategy according to its mission, thereby autonomously determining its role in the Swiss and international university landscape.

  2. 2.

    The CRUS is further convinced that it is best that its member universities themselves determine the body of objective quality criteria that most appropriately fit the deliverables emanating from these strategies. However, no university shall abstain from committing itself to a body of objective quality criteria for its self-chosen deliverables or from communicating them broadly.

As a consequence of these statements, the ‘Mesurer les performances de la recherche’ project and the ‘Performance de la recherche en sciences humaines et sociales’ programme have supported accounting for research evaluation rather than controlling the researchers involved. Both the project and the programme have aimed to develop useful tools for internal quality assessment procedures, stakeholder communications and different approaches to deal with rankings and to achieve greater visibility of research performances. For these purposes, a dedicated decentralized network of specialists has been assembled.

3 The ‘Mesurer les Performances de la Recherche’ Project

Given the considerations mentioned above, the Swiss University Conference decided to finance the ‘Mesurer les performances de la recherche’ project to achieve three purposes:

  • To establish university-based specialists that possess the necessary knowledge in the field of research evaluation.

  • To generalize the use of bibliometry in Swiss universities in order to better judge its potential and its limits.

  • To develop initiatives and actions for those aspects of research quality and performance that are not covered by conventional bibliometry.

The specialists in research evaluation established at every Swiss university represented a central pillar of the prior project and will remain as actors in the current programme. These specialists are organized within a network that guarantees the exchange of experiences and the diffusion of acquired competences by meeting several times a year.

For a better understanding and a more general use of bibliometry, Swiss universities conducted bibliometric analyses in collaboration with the Centre for Science and Technology Studies (CWTS) of Leiden. The main results of this bibliometry project can be summarized as follows: publications of Swiss universities recorded by the Web of Science are far more frequently cited than the world average. In contrast, research that is not published in the Web of Science, especially in the humanities and (to a lesser extent) in the social sciences, is not yet on the radar and remains largely invisible to the conventional bibliometry (CRUS 2009).

In addition to this bibliometric approach mentioned above, the ‘Mesurer les performances de la recherche’ project supported the following three peer-reviewed initiatives:

  • ‘Entwicklung und Erprobung von Qualitätskriterien in den Geisteswissenschaften am Beispiel der Literaturwissenschaften und der Kunstgeschichte [Developing and testing quality criteria for research in the humanities]’, Universities of Zurich and Basel.

  • ‘Measuring Research Output in Communication Sciences and Educational Sciences between international benchmarks, cultural differences and social relevance’, Universities of Lugano, Fribourg, Bern and Zurich.

  • ‘Décrire et mesurer la fécondité de la recherche en sciences humaines et sociales à partir d’études de cas [Describe and measure the fecundity of research in the humanities and social sciences from case studies]’, Universities of Neuchatel, Lausanne and Lugano.

These three projects focused on different issues. ‘Developing and testing quality criteria for research in the humanities’ focused on quality criteria and indicators that researchers in the humanities and social sciences consider important (Hug et al. 2013, 2014; Ochsner et al. 2013, 2014). ‘Measuring Research Output in Communication Sciences and Educational Sciences between international benchmarks, cultural differences and social relevance’ studies the different profiles within and between different research institutions in communication sciences (Probst et al. 2011). The project ‘Describe and measure the fecundity of research in the humanities and social sciences from case studies’ concentrates on making visible the manifold relationships between researchers, institutions and other stakeholders.

Additionally, the project supported four actions to achieve the following:

  • Integrate another language into the initiative ‘Measuring Research Output in Communication Sciences and Educational Sciences between international benchmarks, cultural differences and social relevance’.

  • Organize workshops in an effort to transfer knowledge and experiences developed within the initiatives between the representatives of the involved universities.

  • Organize a workshop to measure research performance in the field of law.

  • Organize workshops and establish an experimental module on the added value of research assessments.

As the final report of the project (CRUS 2013) points out, the participation of all Swiss universities in the project as well as the development of different and complementary contributions represent the main achievements of the project. Both the participation and contributions of the Swiss universities—as leaders of the initiatives and actions or through participating in the experts network—built the foundation for frequent and constructive exchanges, especially within the specialists network. On the other hand, a number of goals were not fully achieved by the time the project was finalized. The CRUS decided to pursue these remaining goals during the period spanning 2013–2016.

4 The ‘Performances de la Recherche en Sciences Humaines et Sociales’ Programme

The financial efforts and implemented measures during the financing period 2008–2012 to support the project were not sufficient. The CRUS thus suggested to continue pursuing the goals of the project from 2013 to 2016 in the ‘Performances de la recherche en sciences humaines et sociales’ programme. This will allow for the sustainable development of competences in research evaluation in universities by allocating project-related subsidies to specialist posts. The launch of the programme also allows for calls for further initiatives with institutional partners that can cover domains and aspects of research not yet covered by the three initiatives of the previous project. The measures of the programme should further promote the development of competences at the national level and enhance international collaboration in the field of research evaluation.

The programme supports seven initiatives that were submitted either by a single university or as the result of collaboration among several universities:

  • ‘Developing indicators for the usage of research in Communication Sciences. Testing the productive interactions approach’, Universities of Fribourg and Lugano

  • ‘Der Wertbeitrag betriebswirtschaftlicher Forschung in Praxis und Gesellschaft [The impact of economics research]’, University of St. Gallen

  • ‘Scientometrics 2.0: Wissenschaftliche Reputation und Vernetzung [Scientometrics 2.0: academic reputation and networks]’, University of St. Gallen

  • ‘Forschungsevaluation in der Rechtswissenschaft [Research evaluation in law]’, Universities of Geneva and Bern

  • ‘Ressourcen-basiertes Instrument zur Abbildung geisteswissenschaftlicher Forschung am Beispiel der Theologie [Resource-based instrument for documenting and assessing research in the humanities and the social sciences as exemplified by theology]’, Universities of Fribourg and Lucerne

  • ‘Cartographier les réseaux de recherche. Interactions et partenariats en sciences humaines et sociales [Mapping research networks. Interactions and partnerships in social sciences and humanities]’, University of Neuchatel

  • ‘National vergleichbare Daten für die Darstellung und Beurteilung von Forschungsleistungen [Comparable data on national level for the presentation and evaluation of research performance]’, University of Basel

As with the previous project, this programme has a special focus on the question of how the diversity concerning the approaches to research and its outcomes can be presented and evaluated appropriately in the context of research evaluation. This includes making visible the manifold interactions and co-operations between researchers and research institutions and the interactions of research institutions in social sciences and humanities with different external stakeholders. The project also investigates how research cultures and the specificities of different disciplines can be taken into account in order to find better ways of evaluating research. Additionally, two projects in law and theology are dedicated to making notions of quality in their disciplines more visible. It will thus also be possible to develop procedures for finding a consensus concerning quality criteria in a particular discipline.

Both programmes together include a total of ten projects. An additional eight so-called ‘Implementation Projects’ are being funded for the years 2015–2016. The aim of these smaller projects is to transfer the results of the initiatives into different institutional and thematic contexts and to test the applicability of the instruments and sets of indicators, examples of which include the following: Based on the results of the project ‘Developing and testing quality criteria for research in the humanities’, a rating form is going to be developed at the University of Zurich that serves to assess the research proposals of junior researchers in the humanities. In addition to ensuring a more appropriate evaluation of emerging researchers proposals, this will also demonstrate the potential of broader sets of qualitative indicators for research evaluation. The University of Lausanne is going to use the mapping tool developed in the project ‘Describe and measure the fecundity of research in the humanities and social sciences from case studies’ for a detailed analysis of this institutions collaborations and partnerships. Based on its own project, ‘Scientometrics 2.0’ (Hoffmann et al. 2015; Jobmann et al. 2014), the University of St. Gallen is incorporating alternative metrics of research impacts into its own repository.

In addition to the 18 total projects, a network consisting of specialists in bibliometrics and research evaluation from all Swiss universities and the individuals in charge of the different initiatives accompanies the programme. This network will allow for an important transfer of knowledge in a decentralized and university-based landscape. The network meets regularly and also invites national and international experts and representatives of the different stakeholders.

The programme has also received a further boost by hiring a full-time scientific coordinator. Besides coordinating the diverse components of the programme, he is also assigned a variety of additional tasks. He is responsible for the internal and external communication on a national and international level as well as the networking with the different stakeholders. He also elaborates on the synthesis of the results. Part of this synthesis is going to be a manual, which introduces the ‘Swiss Way to Quality’ and will enable practitioners to profit from the outcomes of the different projects.

Since the project is still ongoing, most of the results have not been published. However, a website (http://www.performances-recherche.ch) provides information about the current state of the project and the contact information of those responsible for the projects. Overall, both the Swiss universities unique approaches to the challenges in the field of research evaluation and the transfer of knowledge through the ‘Mesurer les performances de la recherche’ project and the ‘Performances de la recherche en sciences humaines et sociales’ programme represent crucial contributions toward an adequate system of research evaluation in the Swiss landscape of higher education, which is currently going through major changes due to the implementation of the new Federal Act on Funding and Coordination of the Swiss Higher Education Sector planned for 2015. At the same time, the programme is a Swiss contribution to the current research debate about how quality in research can best be evaluated.