Publication-Based Funding: The Norwegian Model

Abstract

The ‘Norwegian Model’ attempts to comprehensively cover all the peer-reviewed scholarly literatures in all areas of research—including the preferred formats and languages of scholarly publishing in the humanities—in one single weighted indicator which makes the research efforts comparable across departments and faculties within and between research institutions. This article describes the main components of the model and how it has been implemented, as well as the effects and experiences in three of the countries that are making use of the model, and where it has been evaluated: Belgium (Flanders), Denmark and Norway. The article concludes with a discussion of the model from the perspective of the humanities.

1 Introduction

The so-called ‘Norwegian Model’ (Ahlgren et al. 2012; Schneider 2009), which so far has been adopted at the national level by Belgium (Flanders), Denmark, Finland, Norway and Portugal, as well as at the local level by several Swedish universities, has three components:

• (A) A complete representation in a national database of structured, verifiable and validated bibliographical records of the peer-reviewed scholarly literature in all areas of research;

• (B) A publication indicator with a system of weights that makes field-specific publishing traditions comparable across fields in the measurement of ‘Publication points’ at the level of institutions;

• (C) A performance-based funding model which reallocates a small proportion of the annual direct institutional funding according the institutions’ shares in the total of Publication points.

In principle, component C is not necessary to establish components A and B. The experience is, however, that the funding models in C support the need for completeness and validation of the bibliographic data in component A. Since the largest commercial data sources, such as Scopus or Web of Science, so far lack the completeness needed for the model to function properly, the bibliographic data are delivered by the institutions themselves through Current Research Information Systems (CRIS).

The Norwegian model is designed to represent all areas of research equally and properly. The typical mode of implementation in each country has been for the governments to involve prominent researchers in each major area of research, e. g. deans appointed by the rector’s conference to represent the respective faculties at all universities, or experts appointed by the learned societies on the national level. The representative researchers have then been involved directly in the national adaptation and design of the publication indicator (component B). The result of these design processes has been one single and simple pragmatic compromise—the first bibliometric indicator to cover all areas of research comprehensively and comparably—rather than several separate and ideal representations of scholarly publishing standards in each individual field.

The Norwegian model usually attracts more attention in the social sciences and humanities than in the other areas. Initially, the reaction is negative or sceptical because the model turns scholarly values into measurable points. There are also concerns about the fact that, although it covers book publishing and the national level of publishing better than other indicators, it still disregards other valuable publication practices by concentrating on the peer-reviewed literature and giving extra incentives to publishing on the international level.

The model has been evaluated three times. I will refer results from the evaluation in Belgium (Flanders) here in the introduction and return to the evaluations in Denmark and Norway later on.

Flanders introduced a performance-based funding model called the BOF-key for the five Flemish universities in 2003. The bibliometric part of the funding formula was initially based on data from the Web of Science only. As a response to criticisms from the social sciences and the humanities, the Government decided in 2008 to supplement the commercial data source by introducing modifications of component A and B in the Norwegian model. Since 2009, the Flemish Academic Bibliographic Database for the Social Sciences and the Humanities (Vlaams Academisch Bibliografisch Bestand voor de Sociale en Humane Wetenschappen, VABB-SHW) has collected supplementing bibliographic data from the five universities (Engels et al. 2012). An evaluation of the VABB-SHW was performed in 2012 by the Technopolis Group for the Flemish Government. They found these effects of the initiative (Technopolis Group 2013, pp. 9–10):

• ‘The VABB-SHW protects certain types of publications in the SSH from becoming marginal.

• The VABB-SHW boosts publications in peer-reviewed journals and those with publishers who are using peer review procedures. It thus provides some guidance to publication behaviour of researchers in the SSH domain.

• More generally, the VABB-SHW has led to a greater emphasis on using peer review procedures in journals and by publishers.

• The VABB-SHW has contributed to an increased visibility of both the SSH and the recognition of SSH publications within the academic community.

• The VABB-SHW has also contributed to an increased quality of the bibliographic databases in the SSH domain of the university associations. This provides, in turn, new opportunities for strategic intelligence’.

In the following, I will shortly present the three components of the Norwegian model in more detail. I will then present more results from evaluations of the model. I will conclude by discussing the model from the perspective of the humanities.

My contribution here is not a neutral and objective study of the Norwegian model as seen from the outside. I designed the model in 2003–2004 in collaboration with academic representatives from Norwegian universities and as a consultant to the Norwegian Association of Higher Education Institutions and the Norwegian Ministry of Education and Research (Sivertsen 2010). I still have a role in the further development of the model, both in Norway and in Denmark.

2 Component A: Delimitation and Collection of Data

The Norwegian model is designed to serve a partly indicator-based funding system for research institutions. Since institutions have different research profiles (e. g. a general university versus a technical university), the model needs to represent all research areas in a comprehensive and comparable way.

There is no single comprehensive international data source for all scholarly publications in all research areas. Figure 1 exhibits the patterns and degrees of coverage in the two largest commercial data sources, Scopus and Web of Science. We know from the complete data set that we use here for comparison, which is based on data from the Norwegian model in Norway since 2005, that the deficiencies in coverage of the social sciences and humanities are mainly due to incomplete coverage of the international journals, limited or no coverage of national scholarly journals and very limited coverage of peer-reviewed scholarly books (Sivertsen 2014).

Fig. 1

Coverage in Scopus and Web of Science of 70,500 peer-reviewed scholarly publications in journals, series and books from the higher education sector in Norway 2005–2012

The data for the Norwegian model are delimited by a definition which all areas of research contributed to develop and agree on before it was published in 2004 (Sivertsen and Larsen 2012, p. 569). According to this definition, a scholarly publication must:

• 1. present new insight

• 2. in a scholarly format that allows the research findings to be verified and/or used in new research activity

• 3. in a language and with a distribution that makes the publication accessible for a relevant audience of researchers

• 4. in a publication channel (journal, series, book publisher) which represents authors from several institutions and organizes independent peer review of manuscripts before publication.

While the first two requirements of the definition demand originality and scholarly format in the publication itself, the third and fourth requirement are supported by a dynamic register of approved scholarly publication channels at http://dbh.nsd.uib.no/kanaler/. Suggestions for additions can be made at any time through the same web page.¹ Publications in local channels (serving only one institution’s authors) are not included in the definition, partly because independent peer-review cannot be expected in local channels, and partly because the indicator connected to institutional funding of research is not meant to subsidize in-house publishing.

The definition is not meant to cover the researchers’ publishing activities in general. It is meant to represent research, not publications. Accordingly, it is limited to original research publications.

In addition to a definition, there is need for a comprehensive data source with bibliographic data that can be connected to persons and their institutional affiliations. These data need to be well-structured (thereby comparable and measurable), verifiable (in external data sources, e. g. in the library) and validated (inter-subjective agreement on what is included according to the definition). These needs are now possible to serve due to the development during the last two decades of Current Research Information Systems (CRIS). They can be designed to produce quality assured metadata at the level of institutions or countries.

CRIS systems on the institutional level have become widespread recently, both in locally and commercially developed solutions. Norway is one of a few countries that has a fully integrated non-commercial CRIS system at the national level. Cristin (The Current Research Information System in Norway; cristin.no) is a shared system for all research organizations in the public sector: universities, university colleges, university hospitals and independent research institutes. The Norwegian model, which is now used for institutional funding in all sectors, was a driver in the development of a shared system. One reason is that many publications are affiliated with more than one institution and need to be treated as such in the validation process and in the indicator. Another reason is that transparency across institutions stimulates data quality. Every institution can see and check all other institutions’ data. The publication database in the CRIS system is also online and open to society at large.

The costs of running Cristin would not be legitimate without multiple use of the same data. References to publications are registered only once, after which they can be used in CV’s, applications to research councils, evaluations, annual reports, internal administration, bibliographies for Open Archives, links to full text, etc.

3 Component B: Comparable Measurement

In the measurement for the funding formula by the end of each year, the publications are weighted as they are counted. The intention is to balance between field specific publishing patterns, thereby making the publication output comparable across research areas and institutions that may have different research profiles. In one dimension, three main publication types are given different weights: articles in journals and series (ISSN), articles in books (ISBN) and books (ISBN). In another dimension, publication channels are divided into two levels in order to stimulate publishing in the most prestigious and demanding publication channels within each field of research. The highest level is named ‘Level 2’. It includes only the leading and most selective international journals, series and book publishers. There is also a quantitative restriction, since the publication channels selected for Level 2 can only in total represent up to 20 % of the world’s publications in each field. The weighting of publications by type and channel is shown in Table 1.

Table 1 Publication points in Norway

Publication points are measured at the level of institutions, not at the level of individual researchers. The points for publications with multiple authors representing several institutions are fractionalized among the participating institutions according to their number of participating authors.

The list of journals, series and book publishers on ‘Level 2’ is revised annually in collaboration with national councils in each discipline or field of research (Sivertsen 2010). These councils propose changes to an interdisciplinary National Publishing Board, which governs the process on behalf of all institutions and has the final decision. Bibliometric statistics (world production versus national production in channels on both levels, and citation statistics for publication channels) are used as an aid in this process, but not as criteria by themselves.

4 Component C: Incentives and Funding

There are two main variants of performance-based funding of research institutions in Europe: the evaluation-based variants (United Kingdom and Italy, also being developed in the Czech Republic and in Sweden), and the indicator-based variants (many smaller European countries). The Norwegian model was developed for indicator-based funding. It is, however, not an alternative to research evaluation. In all of the countries using the Norwegian model presently, research evaluations with expert panels are also practiced, but not with direct consequences for institutional funding.

Countries with indicator-based funding of research institutions do not rely solely on bibliometric indicators. Other indicators may be for example be external funding or the number of doctoral degrees. In addition, the indicators usually reallocate only a minor part of the total funding. Consequently, the economic consequences of an institution’s score on the publication indicator in the Norwegian model are therefore relatively small in all countries. In Norway, the publication indicator reallocates less than 2 % of the total expenses in the Higher Education Sector. One publication point represents less than 5,000 Euro.

Still, the publication indicator receives a lot of attention from the researchers, much more attention than is given other and more consequential parts of the funding system. A reason might be that this indicator can be influenced directly by the researchers themselves. Consequently, the Norwegian model seems to be able to change the behaviour of researchers—and that might be a problem.

Fig. 2

Publication points in the Norwegian Higher Education Sector 2004–2013. Level 2 represents internationally leading publication channels expected to publish around 20 % of the total. The red line and the axis on the right side represent the observed percentages on Level 2

5 Evaluations of Effects and Experiences

There have been several studies already of the effects of the Norwegian model in different contexts in Denmark, Flanders, Norway and Sweden (Ahlgren et al. 2012; Hammarfelt and de Rijcke 2014; Ossenblok et al. 2012). In addition, there have been three evaluations commissioned by the Governments in Denmark, Flanders and Norway. Above, we referred to the Flemish evaluation in 2012.

The evaluation of the model in Denmark (Sivertsen and Schneider 2012) covered all of the universities and their research areas. As it was performed only three years after the implementation, not much could be said about the effects and possible unintended consequences. Instead, based on a dialogue with each university, the evaluation identified a number of ideas for improvement of the model which have been taken forward into development work.

The Norwegian model, introduced in 2004, has influenced the funding of Norwegian research institutions since 2005. An evaluation of the effects and experiences was undertaken in 2013. The evaluation was commissioned by the Norwegian Association of Higher Education Institutions and performed by the Danish Centre for Studies in Research and Research Policy at Aarhus University. The report from the evaluation (Dansk Center for Forskningsanalyse 2014), which is in Danish with a ten page summary in English, is being supplemented by a journal article that discusses the results (Aaagaard et al. 2015).

Interviews with researchers and surveys to a large number of them was part of the evaluation in Norway. Since no broad general discontent with the model was found except for the identified problems (see below), and since unintended changes in the researchers’ behaviour could not be detected, at least at the macro level, the Ministry of Education and Research has decided to continue using the model as part of the performance-based funding.

The evaluation identified one major effect of the indicator, increased productivity, along with three major problems, all of which I will discuss shortly here.

A main finding was an increased publication rate above what could be expected from the increase of funding. Figure 2 below shows the increase in publication points in the higher education sector since 2004. Figure 3 below has a more independent measurement based on Web of Science. It shows the development in world shares of articles for four Scandinavian countries. Note that the incentive to publish was introduced in Norway in 2004 and in Denmark and Sweden in 2009. It will be introduced in Finland in 2015.

The evaluation in Norway found no other changes in the publication patterns than the increase. The balances between publication types (books, articles in books, articles in journals and series) and publication languages (the native language versus international languages) remain the same. Collaboration in authorship is increasing at the same rate as in other countries of the same size. The length of publications remains the same. The citation impact on country level is also stable. And, as seen in Fig. 2, the percentage publications in the most internationally influential publication channels has been stable around 20 %, while the absolute number of those publications has almost doubled.

The evaluation in Norway identified three major problems with the model; one problem in the design of the indicator, and two problems with how the model is practiced.

As mentioned above, the publication points for publications with multiple authors representing several institutions are fractionalized among the participating institutions according to their number of participating authors. The evaluation found that this method of fractionalization favours the social sciences and humanities. The average annual publication points per researcher are higher in these areas. Without fractionalization, however, it would be the other way round. Researchers in science, technology and medicine on average contribute to a significantly higher number of publications per year—with the help of their co-authors. The intermediate solution seems to be to use the square root of the institution’s fraction of the publication.

Fig. 3

Shares in the world’s scientific output in Web of Science 2000–2013. Source National Science Indicators (NSI), Thomson Reuters

The transparency and thereby the legitimacy of the annual nomination process for Level 2 (described above in component B) is the second problem identified in the evaluation. Here, the Norwegian Association of Higher Education Institutions has started a project to make the whole process of decisions (and their explicit grounds) available in an internet portal open to all researchers, both for influence and for information.

The third problem is the local use of the indicator. Although the Norwegian model was developed for institutional funding on the national level, the indicator has become widely used also for internal purposes at the level of institutions, faculties, departments, etc. Some of these practices may be reasonable; other practices can be highly problematic, especially if the indicator replaces responsible leadership and human judgment. Norwegian research institutions are relatively autonomous and cannot be instructed from the outside with regard to leadership practices. However, a large national conference was arranged early in 2015 where leaders of research organizations at all levels shared their views and experiences related to the use of the publication indicator at the local level.

6 Discussion: The Norwegian Model from the Perspective of the Humanities

The humanities are known to have more heterogeneous publication patterns than other areas of research. On the one hand, original peer-reviewed research is published in a wider range of formats. Book publishing (monographs or articles in edited volumes) may even be more important than journal publishing in some of the disciplines (Sivertsen and Larsen 2012). On the other hand, scholars in the humanities, more often than their colleagues in the sciences, publish directly for a wider audience in the societies and cultures that they relate to in their research (Bentley and Kyvik 2011). Even the peer-reviewed scholarly publications may appear in the national language if this is more relevant with regard to contents and outreach (Hicks 2004). In addition, nationally adapted textbooks for students are often preferred over international standard editions. Consequently, scholars in the humanities more often appear as authors of textbooks and other educational material.

Publications for wider audiences and for students can be regarded as the most important expression of societal relevance for the humanities. Furthermore, it can often be difficult to draw a line between publications resulting from new research and publications for students and wider audiences. From this perspective, the Norwegian model seems to be restrictive and disincentivising. However, publishing for wider audiences has in fact increased in Norway after the implementation of the model (Kyvik and Sivertsen 2013). From another perspective, the limitation of the indicator to peer-reviewed publications representing original research can be questioned in relation to its purpose: Does it give a balanced representation of the humanities compared to other research areas? The experience is that it does; the research efforts in the humanities can in fact be matched to the efforts in other areas.

The disciplines within the humanities are heterogeneous in their publication patterns. As an example, the degree of international publishing differs a lot across disciplines, and even within them (e. g. in classical versus local archaeology). However, generally, one will find that humanistic scholars will be publishing in a minimum of two languages, one of which is the native language and the other the dominant international language of the field (which in certain humanistic disciplines needs not be English). This is not a new phenomenon; it has been a humanistic practice for two thousand years. Certainly, in our time, we see a gradual and stable increase in English language publishing in the humanities, but there are also large differences between the disciplines (van Leeuwen 2006; Ossenblok et al. 2012), indicating that the bilingual situation will prevail in the humanities due to the societal obligations and wider audiences, as explained above. Furthermore, there is no evidence that book publishing is being replaced by journal publishing in the humanities. The monograph, the edited volume and the journal article, all exist in the humanities because they represent supplementing methodologies in the research itself. Accordingly, all publication types and all languages need to be represented comprehensively in a publication indicator from the perspective of the humanities. From this point of view, the Norwegian model represents a defence of the humanities in a situation where other bibliometric indicators are misrepresenting the disciplines or even creating tensions between them (because there are large variations within the humanities in the representation of the disciplines in commercial data sources).

Access to other publications is perhaps the most important research infrastructure in the humanities. It is a paradox, therefore, that this infrastructure is not in place in the humanities as comprehensively as in other research areas. Web of Science, Scopus, PubMed, Chemical Abstracts, etc., were not created for the purpose of research evaluation, but for bibliographic information retrieval. Figure 1 above is, from this perspective, a demonstration of the deficiency of the library system in serving the humanities with an international infrastructure. Figure 1 also illustrates how the Norwegian model can detect this deficiency. A move forward in the direction of making the scholarly output of the humanities searchable and accessible across countries and languages is more needed now, but also more feasible, with the internationalization of research communication. Visibility and availability can be gained for the humanities by the same move forward. However, this goal is less attainable if we regard the humanistic literatures as endless and want everything that we write to be included. As a first step, the Norwegian model provides definitions, thresholds and empirical statistics that can help delimit the scholarly literatures from other literatures and thereby make them internationally searchable and available.