Introduction

In bibliometric and scientometric research, there is a trend towards developing more and more sophisticated citation-based research performance indicators. In this paper, we are concerned with two streams of research. One stream of research focuses on the development of indicators that aim to correct for the fact that the density of citations (i.e., the average number of citations per publication) differs among fields. Two basic approaches can be distinguished. One approach is to normalize citation counts for field differences based on a classification scheme that assigns publications to fields (e.g., Braun and Glänzel 1990; Moed et al. 1995; Waltman et al. 2011). The other approach is to normalize citation counts based on the number of references in citing publications or citing journals (e.g., Moed 2010; Zitt and Small 2008). The latter approach, which is sometimes referred to as source normalization (Moed 2010), does not need a field classification scheme.

A second stream of research focuses on the development of recursive indicators, typically inspired by the well-known PageRank algorithm (Brin and Page 1998). In the case of recursive indicators, citations are weighed differently depending on the status of the citing publication (e.g., Chen et al. 2007; Ma et al. 2008; Walker et al. 2007), the citing journal (e.g., Bollen et al. 2006; Pinski and Narin 1976), or the citing author (e.g., Radicchi et al. 2009; Życzkowski 2010). The underlying idea is that a citation from an influential publication, a prestigious journal, or a renowned author should be regarded as more valuable than a citation from an insignificant publication, an obscure journal, or an unknown author. It is sometimes argued that non-recursive indicators measure popularity while recursive indicators measure prestige (e.g., Bollen et al. 2006; Yan and Ding 2010).

Based on the above discussion, we have two binary dimensions along which we can distinguish citation-based research performance indicators, namely the dimension of normalization based on a field classification scheme versus source normalization and the dimension of non-recursive mechanisms versus recursive mechanisms. These two dimensions yield four types of indicators. This is shown in Table 1, in which we list some examples of the different types of indicators. It is important to note that all currently existing indicators that use a classification scheme for normalizing citation counts are of a non-recursive nature. Hence, there currently are no recursive indicators that make use of a classification scheme.Footnote 1 Instead, the currently existing recursive indicators can best be regarded as belonging to the family of source-normalized indicators. This is because these indicators, like non-recursive source-normalized indicators, are based in one way or another on the idea that each unit (i.e., each publication, journal, or author) has a certain weight which it distributes over the units it cites. We refer to Waltman and Van Eck (2010a) for a detailed analysis of the close relationship between a source-normalized indicator (i.e., the audience factor) and two recursive indicators (i.e., the Eigenfactor indicator and the influence weight indicator).

Table 1 A classification of some citation-based research performance indicators based on their normalization approach and the presence or absence of a recursive mechanism
Full size table

In this paper, we focus on the empty cell in the lower left of Table 1. Hence, we focus on recursive indicators that use a field classification scheme for normalizing citation counts. We first propose a recursive variant of the mean normalized citation score (MNCS) indicator (Waltman et al. 2011). We then present an empirical analysis of this recursive MNCS indicator. In the analysis, the recursive MNCS indicator is used to study the citation impact of journals and research institutes in the field of library and information science. Our aim is to get insight into the validity of recursive indicators that use a classification scheme for normalizing citation counts. We pay special attention to the sensitivity of such indicators to the classification scheme that is used.

Recursive mean normalized citation score

The ordinary non-recursive MNCS indicator for a set of publications equals the average number of citations per publication, where for each publication the number of citations is normalized for differences among fields (Waltman et al. 2011). The normalization is performed by dividing the number of citations of a publication by the publication’s expected number of citations. The expected number of citations of a publication is defined as the average number of citations per publication in the field in which the publication was published. An example of the calculation of the non-recursive MNCS indicator is provided in Table 2.

Table 2 Example of the calculation of the ordinary non-recursive MNCS indicator
Full size table

The non-recursive MNCS indicator can also be referred to as the first-order MNCS indicator. We define the second-order MNCS indicator in the same way as the first-order MNCS indicator except that citations are weighed differently. In the first-order MNCS indicator, all citations have the same weight. In the second-order MNCS indicator, on the other hand, the weight of a citation is given by the value of the first-order MNCS indicator for the citing journal. Hence, citations from journals with a high value for the first-order MNCS indicator are regarded as more valuable than citations from journals with a low value for the first-order MNCS indicator. We have now defined the second-order MNCS indicator in terms of the first-order MNCS indicator. In the same way, we define the third-order MNCS indicator in terms of the second-order MNCS indicator, the fourth-order MNCS indicator in terms of the third-order MNCS indicator, and so on. This yields the recursive MNCS indicator that we study in this paper.

To make the above definition of the recursive MNCS indicator more precise, we formalize it mathematically. We use i, j, k, and l to denote, respectively, a publication, a journal, a field, and an institute. We define

$$ c_{ii'} = \left\{ {\begin{array}{*{20}c} 1 \hfill & {{\text{if\;publication}}\;i\;{\text{cites\;publication}}\;i'} \hfill \\ 0 \hfill & {\text{otherwise,}} \hfill \\ \end{array} } \right. $$
(1)
$$ p_{ij} = \left\{ {\begin{array}{*{20}c} 1 \hfill & {{\text{if\;publication }}\;i\;{\text{ is\;published\;in\;journal }}j} \hfill \\ 0 \hfill & {\text{otherwise,}} \hfill \\ \end{array} } \right. $$
(2)
$$ b_{ik} = \left\{ {\begin{array}{*{20}c} 1 \hfill & {{\text{if\;publication}}\;i\;{\text{ belongs\;to\;field}}\;k} \hfill \\ 0 \hfill & {\text{otherwise}}, \hfill \\ \end{array} } \right. $$
(3)
$$ a_{il} = \left\{ {\begin{array}{*{20}c} 1 \hfill & {{\text{if\;publication}}\;i\;{\text{ is\;authored\;by\;institute }}l} \hfill \\ 0 \hfill & {{\text{otherwise}}.} \hfill \\ \end{array} } \right. $$
(4)

For α = 1, 2,…, the αth-order citation score of publication i is defined as

$$ {\text{CS}}_{i}^{(\alpha )} = \sum\limits_{i'} {w_{i'}^{(\alpha )} c_{i'i} } , $$
(5)

where w (α) i′ denotes the weight of a citation from publication i′. For α = 1, w (α) i′  = 1 for all i′. For α = 2, 3,…,w (α) i′ is given by

$$ w_{i'}^{(\alpha )} = \sum\limits_{j} {p_{i'j} {\text{MNCS}}_{j}^{(\alpha - 1)} } . $$
(6)

It follows from (5) and (6) that the αth-order citation score of a publication equals a weighted sum of the citations received by the publication. For α = 1, all citations have the same weight. For α = 2, 3,…, the weight of a citation is given by the (α − 1)th-order MNCS of the citing journal.

We define the αth-order mean citation score of a field as the average αth-order citation score of all publications belonging to the field, that is,

$$ {\text{MCS}}_{k}^{(\alpha )} = \frac{{\sum\nolimits_{i} {b_{ik} {\text{CS}}_{i}^{(\alpha )} } }}{{\sum\nolimits_{i} {b_{ik} } }}. $$
(7)

The αth-order expected citation score of a publication is defined as the αth-order mean citation score of the field to which the publication belongs,Footnote 2 and the αth-order normalized citation score of a publication is defined as the ratio of the publication’s αth-order citation score and its αth-order expected citation score. This yields

$$ {\text{ECS}}_{i}^{(\alpha )} = \sum\limits_{k} {b_{ik} {\text{MCS}}_{k}^{(\alpha )} } , $$
(8)
$$ {\text{NCS}}_{i}^{(\alpha )} = \frac{{{\text{CS}}_{i}^{(\alpha )} }}{{{\text{ECS}}_{i}^{(\alpha )} }}. $$
(9)

If the αth-order normalized citation score of a publication is greater (less) than one, this indicates that the αth-order citation score of the publication is greater (less) than the average αth-order citation score of all publications in the field.

We define the αth-order MNCS of a set of publications as the average αth-order normalized citation score of the publications in the set. In the case of journals and institutes, we obtain, respectively

$$ {\text{MNCS}}_{j}^{(\alpha )} = \frac{{\sum\nolimits_{i} {p_{ij} {\text{NCS}}_{i}^{(\alpha )} } }}{{\sum\nolimits_{i} {p_{ij} } }}, $$
(10)
$$ {\text{MNCS}}_{l}^{(\alpha )} = \frac{{\sum\nolimits_{i} {\bar{a}_{il} {\text{NCS}}_{i}^{(\alpha )} } }}{{\sum\nolimits_{i} {\bar{a}_{il} } }}, $$
(11)

where

$$ \bar{a}_{il} = \frac{{a_{il} }}{{\sum\nolimits_{l'} {a_{il'} } }}. $$
(12)

It follows from (11) and (12) that in the case of institutes we take a fractional counting approach. That is, a publication resulting from a collaboration of, say, three institutes is counted for each institute as 1/3 of a full publication. Alternatively, a full counting approach could have been taken. A collaborative publication would then be counted as a full publication for each of the institutes involved. A full counting approach is obtained by replacing \( \bar{a}_{il} \) by a il in (11).

Until now, we have only discussed the issue of normalization for the field in which a publication was published. We have not discussed the issue of normalization for the age of a publication. The latter type of normalization can be used to correct for the fact that older publications have had more time to earn citations than younger publications. Normalization for the age of a publication can easily be incorporated into indicators that make use of a field classification scheme, such as the MNCS indicator. It is more difficult to incorporate into (recursive or non-recursive) source-normalized indicators (see however Waltman and Van Eck 2010b). In the empirical analysis presented later on in this paper, the recursive MNCS indicator performs a normalization both for the field in which a publication was published and for the age of a publication. This means that in the above mathematical description of the recursive MNCS indicator k in fact represents not just a field but a combination of a field and a publication year. As a consequence, b ik indicates whether a publication was published in a certain field and year, and \( {\text{MCS}}_{k}^{(\alpha )} \) indicates the average αth-order citation score of all publications published in a certain field and year.

Data

To test our recursive MNCS indicator, we use the indicator to study the citation impact of journals and research institutes in the field of library and information science (LIS). We focus on the period from 2000 to 2009. Our analysis is based on data from the Web of Science database.

We first needed to delineate the LIS field. We used the Journal of the American Society for Information Science and Technology (JASIST) as the ‘seed’ journal for our delineation. We decided to select the 47 journals that, based on co-citation data, are most strongly related with JASIST. Only journals in the Web of Science subject category Information Science & Library Science were considered. JASIST together with the 47 selected journals constituted our delineation of the LIS field. From the journals within our delineation, we selected all 12,202 publications in the period 2000–2009 that are of the document type ‘article’ or ‘review’. It is important to emphasize that in our analysis we only take into account citations within the set of 12,202 publications. Citations given by publications outside this set are not considered.Footnote 3 Self citations are also excluded.

In our analysis, we also study the effect of splitting up the LIS field in a number of subfields. The following procedure was used to split up the LIS field. We first collected bibliographic coupling data for the 48 journals in our analysis.Footnote 4 Based on the bibliographic coupling data, we created a clustering of the journals. The VOS clustering technique (Waltman et al. 2010), available in the VOSviewer software (Van Eck and Waltman 2010), was used for this purpose. We tried out different numbers of clusters. We found that a solution with three clusters yielded the most satisfactory interpretation in terms of well-known subfields of the LIS field. We therefore decided to use this solution. The three clusters can roughly be interpreted as follows. The largest cluster (27 journals) deals with library science, the smallest cluster (7 journals) deals with scientometrics, and the third cluster (14 journals) deals with general information science topics. The assignment of the 48 journals to the three clusters is shown in Table 3. The clustering of the journals is also shown in the journal map in Fig. 1. This map, produced using the VOSviewer software, is based on bibliographic coupling relations between the journals.

Table 3 48 LIS journals and their assignment to three clusters
Full size table
Fig. 1
figure 1

Journal map of 48 LIS journals based on bibliographic coupling data. The color of a journal indicates the cluster to which it belongs. The map was produced using the VOSviewer software

Full size image

Results

As discussed in the previous section, we can treat LIS either as a single integrated field or as a field consisting of three separate subfields (i.e., library science, information science, and scientometrics). In the latter case, the recursive MNCS indicator normalizes for differences among the three subfields in the average number of citations per publication. Below, we first present the results obtained when LIS is treated as a single integrated field. We then present the results obtained when LIS is treated as a field consisting of three separate subfields. We also present a comparison of the results obtained using the two approaches. We note that all correlations that we report are Spearman rank correlations. Furthermore, we emphasize once more that in our analysis citations given by publications outside our set of 12,202 publications are not taken into account.

Single integrated LIS field

We first consider the case of a single integrated LIS field. The recursive MNCS indicator is said to have converged for a certain α if there is virtually no difference between values of the αth-order MNCS indicator and values of the (α + 1)th-order MNCS indicator. For our data, convergence of the recursive MNCS indicator can be observed for α = 20. In our analysis, our main focus therefore is on comparing the first-order MNCS indicator (i.e., the ordinary non-recursive MNCS indicator) with the 20th-order MNCS indicator.

In Table 4, we list the top 10 journals according to both the first-order MNCS indicator and the 20th-order MNCS indicator. In the case of the first-order MNCS indicator, the top 10 consists of journals from all three subfields. However, journals from the information science and scientometrics subfields seem to slightly dominate journals from the library science subfield. There are three library science journals in the top 10, at ranks 4, 8, and 10. Given that more than half of the journals in our analysis belong to the library science subfield (27 of the 48 journals), the library science journals seem to be underrepresented in the top 10. Also, within the top 10, the three library science journals have relatively low ranks.

Table 4 Top 10 journals according to both the first-order MNCS indicator and the 20th-order MNCS indicator
Full size table

Let’s now turn to the top 10 journals according to the 20th-order MNCS indicator. This top 10 provides a much more extreme picture. The top 10 is now almost completely dominated by information science and scientometrics journals. There is only one library science journal left, at rank 9. Moreover, when looking at the values of the MNCS indicator, large differences can be observed within the top 10. Especially the extremely high value of the MNCS indicator for Journal of Informetrics, the highest ranked journal, is striking. The value of the MNCS indicator for this journal is more than three times as high as the value of the MNCS indicator for Annual Review of Information Science and Technology, which is the second-highest ranked journal.

In Fig. 2, the first-order MNCS indicator is compared with the second-order MNCS indicator (left panel) and the 20th-order MNCS indicator (right panel) for the 48 LIS journals in our data set. As can be seen in the figure, the differences between the first- and the second-order MNCS indicator are relatively small, although there is one journal (Journal of Informetrics) that benefits quite a lot from going from the first-order MNCS indicator to the second-order MNCS indicator. The differences between the first- and the 20th-order MNCS indicator are much larger. As was also seen in Table 4, there are a number of journals for which the value of the 20th-order MNCS indicator is higher than the value of the first-order MNCS indicator. These are all information science and scientometrics journals. The library science journals all turn out to have a lower value for the 20th-order MNCS indicator than for the first-order MNCS indicator.

Fig. 2
figure 2

Comparison of the first-order MNCS indicator with the second-order MNCS indicator (left panel; ρ = 0.90) and the 20th-order MNCS indicator (right panel; ρ = 0.75) for 48 LIS journals. LIS is treated as a single integrated field in the calculation of the indicators. Library science, information science, and scientometrics journals are indicated by, respectively, red points, green crosses, and blue circles. (Color figure online)

Full size image

In addition to journals, we also consider research institutes in LIS. We restrict our analysis to the 86 institutes that have at least 25 publications in our data set. (Recall that publications are counted fractionally.) The top 10 institutes according to both the first-order MNCS indicator and the 20th-order MNCS indicator are listed in Table 5. Comparing the results of the two MNCS indicators, it is clear that institutes which are mainly active in the scientometrics subfield benefit a lot from the use of a higher-order MNCS indicator. For these institutes, the value of the 20th-order MNCS indicator tends to be much higher than the value of the first-order MNCS indicator. This is consistent with our above analysis for LIS journals, where we found that the two most important scientometrics journals (Journal of Informetrics and Scientometrics) benefit quite significantly from the use of a higher-order MNCS indicator.

Table 5 Top 10 research institutes according to both the first-order MNCS indicator and the 20th-order MNCS indicator
Full size table

Three separate LIS subfields

We now consider the case in which LIS is divided into three separate subfields (i.e., library science, information science, and scientometrics). In the calculation of the recursive MNCS indicator, a normalization is performed to correct for differences among the three subfields in the average number of citations per publication. Like in the above analysis, we focus mainly on comparing the first-order MNCS indicator with the 20th-order MNCS indicator.

In Table 6, the top 10 journals according to both the first-order MNCS indicator and the 20th-order MNCS indicator is shown. This table is similar to Table 4 above, except that in the calculation of the indicators LIS is treated as a field consisting of three separate subfields rather than as a single integrated field. Comparing Table 6 with Table 4, it can be seen that library science journals now play a much more prominent role, both in the case of the first-order MNCS indicator and in the case of the 20th-order MNCS indicator. As a consequence, the top 10 journals now looks much more balanced for both MNCS indicators. Also, unlike in Table 4, there are no journals in Table 6 with an extremely high value for the 20th-order MNCS indicator.

Table 6 Top 10 journals according to both the first-order MNCS indicator and the 20th-order MNCS indicator
Full size table

As can be seen in Table 6, the journal with the highest value for the 20th-order MNCS indicator is Interlending & Document Supply. We investigated this journal in more detail and found that each issue of the journal contains a review article entitled “Interlending and document supply: A review of the recent literature”. These review articles refer to other articles published in the same issue of the journal. Clearly, the practice of publishing these review articles is an important contributing factor to the journal’s top ranking in Table 6.

In Fig. 3, a comparison is presented of the first-order MNCS indicator with the second-order MNCS indicator (left panel) and the 20th-order MNCS indicator (right panel) for the 48 LIS journals in our data set. The figure shows that most of the differences between the first-order MNCS indicator and the higher-order MNCS indicators are not very large, especially when compared with the differences shown in Fig. 2. The journal that is most sensitive to the use of a higher-order MNCS indicator is Interlending & Document Supply. As pointed out above, this journal has quite special citation characteristics, which explains its sensitivity to the use of a higher-order MNCS indicator.

Fig. 3
figure 3

Comparison of the first-order MNCS indicator with the second-order MNCS indicator (left panel; ρ = 0.93) and the 20th-order MNCS indicator (right panel; ρ = 0.89) for 48 LIS journals. LIS is treated as a field consisting of three separate subfields in the calculation of the indicators. Library science, information science, and scientometrics journals are indicated by, respectively, red points, green crosses, and blue circles. (Color figure online)

Full size image

Results for research institutes in LIS are reported in Table 7. In the table, the top 10 institutes according to both the first-order MNCS indicator and the 20th-order MNCS indicator are shown. Like in Table 5, the top of the ranking is dominated by institutes with a strong focus on scientometrics research. This is the case both for the first-order MNCS indicator and for the 20th-order MNCS indicator. However, comparing Table 7 with Table 5, it can be seen that the MNCS values of the scientometrics institutes have decreased quite considerably, especially when looking at the 20th-order MNCS indicator. Hence, although the scientometrics institutes still occupy the top positions in the ranking, the differences with the other institutes have become smaller.

Table 7 Top 10 research institutes according to both the first-order MNCS indicator and the 20th-order MNCS indicator
Full size table

Comparison

In Fig. 4, we present a direct comparison of on the one hand the results obtained when LIS is treated as a single integrated field and on the other hand the results obtained when LIS is treated as a field consisting of three separate subfields. The comparison is made for the 48 LIS journals in our data set. Figure 4 clearly shows how treating LIS as a single integrated field benefits the scientometrics journals and harms the library science journals. In addition, the figure shows that this effect is strongly reinforced when instead of the first-order MNCS indicator the 20th-order MNCS indicator is used.

Fig. 4
figure 4

Comparison of the MNCS indicator when LIS is treated as a single integrated field with the MNCS indicator when LIS is treated as a field consisting of three separate subfields. The comparison is made for 48 LIS journals, and either the first-order MNCS indicator (left panel; ρ = 0.96) or the 20th-order MNCS indicator (right panel; ρ = 0.77) is used. Library science, information science, and scientometrics journals are indicated by, respectively, red points, green crosses, and blue circles. (Color figure online)

Full size image

Discussion and conclusion

Recursive bibliometric indicators are based on the idea that citations should be weighed differently depending on the source from which they originate. Citations from a prestigious journal, for instance, should have more weight than citations from an obscure journal. It is sometimes argued that by weighing citations differently depending on their source it is possible to measure not just the popularity of publications but also their prestige.

In this paper, we have combined the idea of recursive citation weighing with the idea of using a classification scheme to normalize citation counts for differences among fields. The combination of these two ideas has not been explored before. Although when used separately from each other the two ideas can be quite useful, our empirical analysis for the field of LIS indicates that the combination of the two ideas does not yield satisfactory results. The main observations from our analysis are twofold. First, our proposed recursive MNCS indicator is highly sensitive to the way in which fields are defined in the classification scheme that one uses. And second, if within a field there are subfields with significantly different citation characteristics, the recursive MNCS indicator will be strongly biased in favor of the subfields with the highest density of citations.

The sensitivity of bibliometric indicators to the field classification scheme that is used for normalizing citation counts has been investigated in various studies (Adams et al. 2008; Bornmann et al. 2008; Neuhaus and Daniel 2009; Van Leeuwen et al. 2009; Zitt et al. 2005). Our empirical results for the first-order MNCS indicator (i.e., the ordinary non-recursive MNCS indicator) are in line with earlier studies in which a significant sensitivity of bibliometric indicators to the classification scheme that is used has been reported. For the 20th-order MNCS indicator, this sensitivity even turns out to be much higher. Treating LIS as a single integrated field or as a field consisting of three separate subfields yields very different results for the 20th-order MNCS indicator.

If a field as defined in the classification scheme that one uses is heterogeneous in terms of citation characteristics, bibliometric indicators will have a bias that favors subfields with a higher citation density over subfields with a lower citation density. This is a general problem of bibliometric indicators that use a classification scheme to normalize citation counts. In the case of the recursive MNCS indicator, our empirical results show that the idea of recursive citation weighing strongly reinforces biases caused by the classification scheme. Within the field of LIS, the scientometrics subfield has the highest citation density, followed by the information science subfield. The library science subfield has the lowest citation density. When LIS is treated as a single integrated field, the differences in citation density among the three LIS subfields cause both the first-order MNCS indicator and the 20th-order MNCS indicator to be biased, where the bias favors the scientometrics subfield and harms the library science subfield. However, the bias is much stronger for the 20th-order MNCS indicator than for the first-order MNCS indicator. For instance, in the case of the 20th-order MNCS indicator, library science journals are completely dominated by journals in scientometrics and information science.

Based on the above observations, we advise against the introduction of recursiveness into bibliometric indicators that use a field classification scheme for normalizing citation counts (such as the MNCS indicator). Instead of providing more sophisticated measurements of citation impact (measurements of ‘prestige’ rather than ‘popularity’), the main effect of introducing recursiveness is to reinforce biases caused by the way in which fields are defined in the classification scheme that one uses. Although our negative results may partly relate to specific characteristics of the MNCS indicator, we expect our general conclusion to be valid for all field-normalized indicators.