Introduction

The use of research performance indicators is often discussed in the literature on science and technology policy. Such discussions focus on, amongst others, the optimal measurement of research performance, research impact and research quality and on the possibly detrimental consequences of improper use of research indicators (e.g. Hicks et al. 2015). Many researchers seem to take for granted that there is an ever increasing level of use of research performance indicators, but actually there is no systematic analysis of the level of use. A first step towards such a systematic study would be a qualitative classification of the types of use. The literature provides a few studies on some types of use, but usually these merely aim to propose an alternative indicator or concentrate on a particular indicator or evaluation system which includes research performance indicators.

The purpose of the present paper is to develop a comprehensive classification of the types of use of research indicators which can be used in quantitative studies. The classification is derived from the information on research indicator use given in the journal literature in the field of scientometrics/bibliometrics. We illustrate the different categories in the classification by means of examples of the mentions of use from which the classification has been derived. In this way, we will also give an overview of recent studies which mention the use of research performance indicators.

In this paper, we use the term indicator in the sense of a quantitative characterization of research performance. This does not necessarily mean that an indicator should be based on measurement, it can also result from qualitative assessments. Moreover, we focus on research performance indicators, or research indicators for briefness. Therefore, we do not include the use of input indicators such as the number of staff nor the number of PhD students. In contrast, indicators such as third-party funding are included if they are used as a proxy for quality.

We collected relevant studies containing references to the use of research indicators primarily from Scientometrics. This journal has an established position in the field and pays much attention to research indicators. In order to create a classification which is based on the most up-to-date information, we focussed on the most recent period (2003–2013). Recent volumes of other journals have been used to verify whether the information on indicator use they provide also fits the initial classification. Some of these journals were in adjacent fields such as research and science policy, and research evaluation.

Though the present paper is the first to focus on indicator use, there have been comparative studies of other subjects that provided information on indicator use as a collateral: studies of funding systems, organization and management and, in one case, of rankings. These studies with collateral comparative information on indicator use are reviewed in the Online Resource section 2.

This paper is structured as follows. In the “Methods and data collection” section we set out our methodology. The “Classification” section is devoted to the classification itself. The “Concluding remarks” section draws some conclusions about the available literature on the use of research performance indicators and provides some reflections on the usefulness of the classification and future work.

Methods and data collection

Naturally, the development of a classification of the use of research indicators requires as a ‘data source’ some kind of inventory of the various types of use to which indicators have been put. The literature contains no recent comprehensive overview of indicator use. Thus far, the recent literature only provides overviews of some areas of use (see Only Resource section 1). Therefore, we had to develop our own data source. In principle, this could be done in various ways. Since use can take many different forms, one approach would be to consider papers of organizations that use the indicators: universities and other research organizations, funding organizations, governments, international organizations, and so on. In fact, to obtain a quantitative picture of indicator use, this might be the only valid approach. However, for the purpose of the design of a classification, this approach has three problems. Firstly, the selection of the ‘users’ would be rather arbitrary. Secondly, the selection of the ‘papers’ of the use would be subjective and might easily cause biases. And finally, the approach would probably be much more laborious than the development of a classification requires, since the question guiding this research is not how often a type of use occurs but which types of use can be distinguished, which precedes the other question. Therefore, another approach has been chosen, namely to use a well-defined scientific body of literature as source. The topic and size of this body of literature should be such that it covers the most important topics regarding the use of research indicators. We are only interested in the types of research indicator use and not in how often those types of indicator use are mentioned in the literature or applied in practice. Therefore it does not matter if some papers referring to research indicator use are missed by the searching methods we apply as long as all types of indicator use are mentioned at least once.

Scientometrics

Given the focus on (bibliometric and scientometric) research indicators, the journal Scientometrics was selected as a primary source.Footnote 1 Given Scientometrics’ focus on “quantitative features and characteristics of science and scientific research” and its role as “indispensable [journal] to research workers and research administrators” and “provide[r of] valuable assistance to librarians and documentalists in central scientific agencies, ministries, research institutes and laboratories” (Scientometrics 2014). Including descriptions of practices which are clearly obsolete nowadays would not make sense. Therefore, only papers published in the volumes of Scientometrics during the period January 2003 till March 2013 have been analyzed (vol. 56(1)–94(3)).

From this body of papers, the contributions containing information on indicator use were selected as follows. Identification of relevant papers started by browsing 75 Scientometrics issues published from 2007 until 2013 (vol. 70 until vol. 94) manually. Every article of which the title or the abstract provided an indication that it could contain a description of the use of research indicators was scrutinized in order to ascertain if it actually contained such a description. Once information about the application of research indicators was found, the paper was used as a source of information for the classification. To be included in our collection, descriptions should be clear and specific enough to know where and in which context research indicators are used. The context in which such a description is given did not matter. Therefore, our selection includes papers by authors whose work does not focus on research indicator use and who might themselves use other sources to describe the use of indicators. In this way, the 26 articles selected for further analysis form a subset of all papers in Scientometrics in vol. 70–94.

After this first group of articles had been identified, we enlarged the set of papers used as source for the classification by means of a keyword search. This approach is less time consuming than a ‘manual’ search while it would not necessarily yield less relevant results, provided the keywords have sufficient specificity. Since a combination of the keywords ‘research’ and ‘evaluation’ gives results about research evaluation, but any kind of research on evaluation, such as the results of research on the evaluation of certain types of medication more specific results are needed. Therefore, the selection of keywords that are both sufficiently inclusive and sufficiently discriminatory is of great importance if we want to expand this searching method to a broader set of journals. Eventually, from the initial set of 177 papers resulting from out searching method, we obtained twelve papers by reading them as in the first set of papers. This is less papers per volume than the result of the manual search. More details on our keywords based search method can be found in Online Resource section 2.

Contributions in other journals

Scientometrics in itself might have certain limitations as a source for descriptions of the use of research indicators.Footnote 2 This medium contains mostly papers on methods of bibliometric or scientometric measurement without reference to the actual use of research indicators. Since the use of research indicators may be important for any researcher or scientist, this topic might be covered by any journal. Still, we only had a limited amount of resources available. Therefore, the preliminary results were verified with contributions to eight other journals with a similar or a slightly different focus compared with Scientometrics: Higher Education, JASIST, Journal of Informetrics, Minerva, Research Evaluation, Research Policy, Science and Public Policy, and Social Studies of Science. Moreover, we analyzed the PLOS ONE sections ‘Science policy’, ‘Research assessment’ and ‘Library science’. Papers were selected in the same way as the Scientometrics papers. This process resulted in 59 papers with relevant information for the classification. These contributions were used to verify the preliminary classification and to enrich it with additional categories and examples. Online resource section 2 contains an overview of the results per journal; it shows there are considerable differences per journal.

Classification

This section presents the result of our study: a classification of all types of research indicator use. The classification (Table 1) has two levels. Five main categories of use are identified: general science policy, funding allocation, management and organization, content decisions, and consumer information. Within these categories 21 different types of activities are identified in which research indicators are often used. Table 1 contains both the classification and the description of the different types of use. In the next section we give examples from the literature with more specific information and references to current practices.

Table 1 Classification
Full size table

Examples

Table 2 contains a number of important examples to illustrate the various categories. Sources of these examples are both the dataset collected for this analysis and the studies mentioned in the first section of the Online Resource. Forms of use can be assigned to more than one category. A complete overview of all examples which led to the classification can be found in the Online Resource section 3.

Table 2 Examples
Full size table

Concluding remarks

We analysed the mentioning of use of research indicators in recent issues of a number of international journals on bibliometrics, scientometrics, science policy, research management, and research evaluation. The use of research indicators is not a prominent topic in this scientific literature and often appears as a ‘collateral’ when the theoretical characteristics of research indicators are discussed. Much research focuses on the development and properties of the indicators while disregarding their actual use. By and large, this is true for all journals considered.

Research indicators are used for many different purposes, ranging from high level nationwide policies to micro considerations regarding the choice of a publication outlet. In our classification we have identified five main types of use: General science policy, Funding allocation, Organization and management, Content management and decisions, and Consumer information. Within these main categories, a diversity of specific types of use has been classified and illustrated with recent examples.

There are still a number of questions which deserve further attention. To what extent is the use of research indicators widespread? And if research indicators are used, what is their influence? The former question can be answered by means of quantitative methods while the latter question can be addressed by means of both quantitative as well as qualitative methods. In both cases, our classification is a convenient point of departure. In a way, Hazelkorn (2007) has addressed part of these questions in her survey of the effects of league tables. As indicators are often used for funding in ways that differ quite substantially between countries, further research could prioritize the effects of this type of use.

What are the implications of our findings on indicator use for the grand challenges in data integration? The overview of the current uses of research indicators presented in this paper clearly reveals the relevance of data integration and interoperability: there is such a diversity of uses that no single indicator can serve all of them, nor is it likely that any indicator is useful for just one of these purposes. This means that most purposes can only be served by a combination of indicators and that, at the same time, many indicators will serve several purposes but together with different combinations of other indicators. This, in turn, means that it is highly desirable that data from different sources and different indicators can be related to each other, and share common definitions and classifications so that they can be used in combination. To give one example: anyone who has ever worked with data on universities from different sources knows that it is highly desirable to develop a standard delineation of universities that is used in all sources, so that bibliometric data from various sources as well as altmetric data and financial data can be more easily related to each other.

An obvious way to move forward is to analyze which data sources and indicators are used for which purposes. This means that the data sources and indicators are classified according to our classification of types of use. This will show where comparability or ‘relatability’ is most needed and what type of action has uttermost importance. Thus, if it turns out that a certain set of indicators is often used for the purpose of the allocation of funding to universities, then it makes sense to prioritize the work on a common delineation of universities in this set of indicators and the underlying data sources. Consequently, the grand challenges posed to the current infrastructure can be addressed more easily and even be guided by using our classification of types of use.