A classification of the use of research indicators

Research performance indicators are broadly used, for a range of purposes. The scientific literature on research indicators has a strong methodological focus. There is no comprehensive overview or classification of the use of such indicators. In this paper we give such a classification of research indicator use. Using the journal Scientometrics as a starting point we scrutinized recent journal literature on scientometrics, bibliometrics, research policy, research evaluation, and higher education in order to spot paragraphs or sections that mention indicator use. This led to a classification of research indicator use with 21 categories which can be grouped into five main categories. Electronic supplementary material The online version of this article (doi:10.1007/s11192-016-1904-7) contains supplementary material, which is available to authorized users.


Introduction
The use of research performance indicators is often discussed in the literature on science and technology policy. Such discussions focus on, amongst others, the optimal measurement of research performance, research impact and research quality and on the possibly detrimental consequences of improper use of research indicators (e.g. Hicks et al. 2015). Many researchers seem to take for granted that there is an ever increasing level of use of research performance indicators, but actually there is no systematic analysis of the level of use. A first step towards such a systematic study would be a qualitative classification of the types of use. The literature provides a few studies on some types of use, but usually these merely aim to propose an alternative indicator or concentrate on a particular indicator or evaluation system which includes research performance indicators.
The purpose of the present paper is to develop a comprehensive classification of the types of use of research indicators which can be used in quantitative studies. The classification is derived from the information on research indicator use given in the journal literature in the field of scientometrics/bibliometrics. We illustrate the different categories in the classification by means of examples of the mentions of use from which the classification has been derived. In this way, we will also give an overview of recent studies which mention the use of research performance indicators.
In this paper, we use the term indicator in the sense of a quantitative characterization of research performance. This does not necessarily mean that an indicator should be based on measurement, it can also result from qualitative assessments. Moreover, we focus on research performance indicators, or research indicators for briefness. Therefore, we do not include the use of input indicators such as the number of staff nor the number of PhD students. In contrast, indicators such as third-party funding are included if they are used as a proxy for quality.
We collected relevant studies containing references to the use of research indicators primarily from Scientometrics. This journal has an established position in the field and pays much attention to research indicators. In order to create a classification which is based on the most up-to-date information, we focussed on the most recent period (2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013). Recent volumes of other journals have been used to verify whether the information on indicator use they provide also fits the initial classification. Some of these journals were in adjacent fields such as research and science policy, and research evaluation.
Though the present paper is the first to focus on indicator use, there have been comparative studies of other subjects that provided information on indicator use as a collateral: studies of funding systems, organization and management and, in one case, of rankings. These studies with collateral comparative information on indicator use are reviewed in the Online Resource section 2. This paper is structured as follows. In the ''Methods and data collection'' section we set out our methodology. The ''Classification'' section is devoted to the classification itself. The ''Concluding remarks'' section draws some conclusions about the available literature on the use of research performance indicators and provides some reflections on the usefulness of the classification and future work.

Methods and data collection
Naturally, the development of a classification of the use of research indicators requires as a 'data source' some kind of inventory of the various types of use to which indicators have been put. The literature contains no recent comprehensive overview of indicator use. Thus far, the recent literature only provides overviews of some areas of use (see Only Resource section 1). Therefore, we had to develop our own data source. In principle, this could be done in various ways. Since use can take many different forms, one approach would be to consider papers of organizations that use the indicators: universities and other research organizations, funding organizations, governments, international organizations, and so on. In fact, to obtain a quantitative picture of indicator use, this might be the only valid approach. However, for the purpose of the design of a classification, this approach has three problems. Firstly, the selection of the 'users' would be rather arbitrary. Secondly, the selection of the 'papers' of the use would be subjective and might easily cause biases. And finally, the approach would probably be much more laborious than the development of a classification requires, since the question guiding this research is not how often a type of use occurs but which types of use can be distinguished, which precedes the other question. Therefore, another approach has been chosen, namely to use a well-defined scientific body of literature as source. The topic and size of this body of literature should be such that it covers the most important topics regarding the use of research indicators. We are only interested in the types of research indicator use and not in how often those types of indicator use are mentioned in the literature or applied in practice. Therefore it does not matter if some papers referring to research indicator use are missed by the searching methods we apply as long as all types of indicator use are mentioned at least once.

Scientometrics
Given the focus on (bibliometric and scientometric) research indicators, the journal Scientometrics was selected as a primary source. 1 Given Scientometrics' focus on ''quantitative features and characteristics of science and scientific research'' and its role as ''indispensable [journal] to research workers and research administrators'' and ''provide[r of] valuable assistance to librarians and documentalists in central scientific agencies, ministries, research institutes and laboratories'' (Scientometrics 2014). Including descriptions of practices which are clearly obsolete nowadays would not make sense. Therefore, only papers published in the volumes of Scientometrics during the period January 2003 till March 2013 have been analyzed (vol. 56(1)-94 (3)).
From this body of papers, the contributions containing information on indicator use were selected as follows. Identification of relevant papers started by browsing 75 Scientometrics issues published from 2007 until 2013 (vol. 70 until vol. 94) manually. Every article of which the title or the abstract provided an indication that it could contain a description of the use of research indicators was scrutinized in order to ascertain if it actually contained such a description. Once information about the application of research indicators was found, the paper was used as a source of information for the classification. To be included in our collection, descriptions should be clear and specific enough to know where and in which context research indicators are used. The context in which such a description is given did not matter. Therefore, our selection includes papers by authors whose work does not focus on research indicator use and who might themselves use other sources to describe the use of indicators. In this way, the 26 articles selected for further analysis form a subset of all papers in Scientometrics in vol. 70-94.
After this first group of articles had been identified, we enlarged the set of papers used as source for the classification by means of a keyword search. This approach is less time consuming than a 'manual' search while it would not necessarily yield less relevant results, provided the keywords have sufficient specificity. Since a combination of the keywords 'research' and 'evaluation' gives results about research evaluation, but any kind of research on evaluation, such as the results of research on the evaluation of certain types of 1 The relevance of Scientometrics as selected source of articles can be deduced from the results if one looks up a combination of ''research evaluation'' or ''research assessment'' and ''indicator'' (with a wildcard): a total of 226 results are found, with the largest number of papers in Scientometrics (52)  medication more specific results are needed. Therefore, the selection of keywords that are both sufficiently inclusive and sufficiently discriminatory is of great importance if we want to expand this searching method to a broader set of journals. Eventually, from the initial set of 177 papers resulting from out searching method, we obtained twelve papers by reading them as in the first set of papers. This is less papers per volume than the result of the manual search. More details on our keywords based search method can be found in Online Resource section 2.

Concluding remarks
We analysed the mentioning of use of research indicators in recent issues of a number of international journals on bibliometrics, scientometrics, science policy, research management, and research evaluation. The use of research indicators is not a prominent topic in this scientific literature and often appears as a 'collateral' when the theoretical characteristics of research indicators are discussed. Much research focuses on the development and properties of the indicators while disregarding their actual use. By and large, this is true for all journals considered.
Research indicators are used for many different purposes, ranging from high level nationwide policies to micro considerations regarding the choice of a publication outlet. In our classification we have identified five main types of use: General science policy, 14 Quality management and quality assessment Quality assurance or quality improvement cannot only be achieved by funding or inducements, but also by means of quality assessment. Committees assessing quality might use research indicators as information about the research performance.
15 Reputation management Use of research indicators to advertise strengths of a research institution or individual researchers.
16 Selection of partners and members Use of research indicators to inform institutions about the research performance of possible partners or candidates which apply for membership of professional associations.

D Content management and decisions
Research indicators can play a role in issues which are immediately linked to the contents of research.

Publication channel selection
Use of indicators such as the journal impact factor by authors to decide in which medium they will try to publish their work.
18 Research profile management Use of research indicators by research institutions to manage their research profile and evaluate their strengths and weaknesses.
19 Journal and database management Use of research indicators to manage or support journals and bibliographic databases.
20 Library collection management Use of research indicators by librarians to inform themselves about publications which should be adopted by their institutions.

E Consumer information
Research indicators such as rankings are proxies for quality of institutes and serve different groups of consumers as important sources of information.
21 Consumer information (not elsewhere specified) Use of research indicators as source of information by different types of consumers, e.g., students and their parents about the performance of higher education institutions, or researchers looking for another employer.
Funding allocation, Organization and management, Content management and decisions, and Consumer information. Within these main categories, a diversity of specific types of use has been classified and illustrated with recent examples. There are still a number of questions which deserve further attention. To what extent is the use of research indicators widespread? And if research indicators are used, what is their influence? The former question can be answered by means of quantitative methods while the latter question can be addressed by means of both quantitative as well as qualitative methods. In both cases, our classification is a convenient point of departure. In a way, Hazelkorn (2007) has addressed part of these questions in her survey of the effects of league tables. As indicators are often used for funding in ways that differ quite substantially between countries, further research could prioritize the effects of this type of use.
What are the implications of our findings on indicator use for the grand challenges in data integration? The overview of the current uses of research indicators presented in this paper clearly reveals the relevance of data integration and interoperability: there is such a diversity of uses that no single indicator can serve all of them, nor is it likely that any indicator is useful for just one of these purposes. This means that most purposes can only be served by a combination of indicators and that, at the same time, many indicators will serve several purposes but together with different combinations of other indicators. This, in

D Content management and decisions
Research indicators can serve researchers to help them make decisions on the journal in which they should publish a contribution. The UK-based Association of Business Schools developed a journal ranking with such a purpose in mind.

E Consumer information
Research evaluation results from the UK RAE and the Turkish ranking of universities are publicly made available to help prospective students and their parents choose an institution for further education.
Scientometrics (2016) 108:457-464 463 turn, means that it is highly desirable that data from different sources and different indicators can be related to each other, and share common definitions and classifications so that they can be used in combination. To give one example: anyone who has ever worked with data on universities from different sources knows that it is highly desirable to develop a standard delineation of universities that is used in all sources, so that bibliometric data from various sources as well as altmetric data and financial data can be more easily related to each other. An obvious way to move forward is to analyze which data sources and indicators are used for which purposes. This means that the data sources and indicators are classified according to our classification of types of use. This will show where comparability or 'relatability' is most needed and what type of action has uttermost importance. Thus, if it turns out that a certain set of indicators is often used for the purpose of the allocation of funding to universities, then it makes sense to prioritize the work on a common delineation of universities in this set of indicators and the underlying data sources. Consequently, the grand challenges posed to the current infrastructure can be addressed more easily and even be guided by using our classification of types of use.