Influence of omitted citations on the bibliometric statistics of the major Manufacturing journals

Abstract

Bibliometrics is a relatively young and rapidly evolving discipline. Essential for this discipline are bibliometric databases and their information content concerning scientific publications and relevant citations. Databases are unfortunately affected by errors, whose main consequence is represented by omitted citations, i.e., citations that should be ascribed to a certain (cited) paper but, for some reason, are lost. This paper studies the impact of omitted citations on the bibliometric statistics of the major Manufacturing journals. The methodology adopted is based on a recent automated algorithm—introduced in (Franceschini et al., J Am Soc Inf Sci Technol 64(10):2149–2156, 2013)—which is applied to the Web of Science (WoS) and Scopus database. Two important results of this analysis are that: (i) on average, the omitted-citation rate (p) of WoS is slightly higher than that of Scopus; and (ii) for both databases, p values do not change drastically from journal to journal and tend to slightly decrease with respect to the issue year of citing papers. Although it would seem that omitted citations do not represent a substantial problem, they may affect indicators based on citation statistics significantly. This paper analyses the effect of omitted citations on popular bibliometric indicators like the average citations per paper and its most famous variant, i.e., the ISI Impact Factor, showing that journal classifications based on these indicators may lead to questionable discriminations.

Appendix

Analysis of the distribution of omitted citations

Study at the level of the journal of cited papers

The dispersion related to the p _J value of each journal (defined in Sect. 4.1) can be roughly estimated through an expedient. Each p _J value can be expressed as:

$$ p_{J} = {{\sum\limits_{i = 1}^{{P_{J} }} {\left[ {\left( {p_{J} } \right)_{i} \cdot \left( {\gamma_{J} } \right)_{i} } \right]} } \mathord{\left/ {\vphantom {{\sum\limits_{i = 1}^{{P_{J} }} {\left[ {\left( {p_{J} } \right)_{i} \cdot \left( {\gamma_{J} } \right)_{i} } \right]} } {\sum\limits_{i = 1}^{{P_{J} }} {\left( {\gamma_{J} } \right)_{i} } }}} \right. \kern-0pt} {\sum\limits_{i = 1}^{{P_{J} }} {\left( {\gamma_{J} } \right)_{i} } }}, $$

(8)

being (p _J ) _i  = (ω _J ) _i /(ω _J ) _i (γ _J ) _i .(γ _J ) _i the percentage of citations omitted by the database of interest, referring to the i-th article published by J.

Equation 8 shows that (p _J ) _i can be seen as a weighted average of the omitted-citation rates of individual papers (i.e., (p _J ) _i values). These contributions have a variable weight, represented by the number of “theoretically overlapping” citations of each i-th article of interest (i.e., (γ _J ) _i ). Of course, articles with no citation will have a zero weight.

Being p _J a weighted quantity, one can represent the distribution of (p _J ) _i values by a special box-plot based on weighted quartiles, defined as ^w Q ⁽¹⁾ _J , ^w Q ⁽²⁾ _J and ^w Q ⁽³⁾ _J , i.e., the weighted first, second (or weighted median) and third quartile of the (p _J ) _i values. Weighted quartiles are reported in Table 9. These indicators are obtained by ordering in ascending order the (p _J ) _i values of the articles of interest and considering the values for which the cumulative of weights is equal to respectively the 25, 50 and 75 % of their sum.

Table 9 Weighted quartiles related to the distributions of the (p _J ) _i values, for the scientific journals listed in Table 3

The differences between the (p _J ) _i distributions of the Manufacturing journals seem insignificant for both WoS and Scopus. The reason is that the notches related to the majority of the journals are overlapped. In particular, we note that most of the notches are “collapsed” on the line corresponding to (p _J ) _i  = 0 and all ^w Q ⁽¹⁾ _J values are zero, as well as almost all of ^w Q ⁽²⁾ _J values, both for WoS and Scopus. This result is very interesting because it tells us that omitted citations are generally concentrated into a relatively small number of articles. To confirm this, we can see that—for each of the journals analyzed—the weighted median of the (p _J ) _i values (i.e. ^w Q ⁽²⁾ _J , in Table 9) is systematically lower than the weighted average, i.e. p _J .

Study at the level of the age of citing papers

The dispersion related to the p _Y values of each journal (defined in Sect. 4.2) can be roughly estimated through an expedient, similarly to that presented in Sect. 6.1.1. Each p _Y value can be expressed as:

$$ p_{Y} = {{\sum\limits_{i = 1}^{P} {\left[ {\left( {p_{Y} } \right)_{i} \cdot \left( {\gamma_{Y} } \right)_{i} } \right]} } \mathord{\left/ {\vphantom {{\sum\limits_{i = 1}^{P} {\left[ {\left( {p_{Y} } \right)_{i} \cdot \left( {\gamma_{Y} } \right)_{i} } \right]} } {\sum\limits_{i = 1}^{P} {\left( {\gamma_{Y} } \right)_{i} } }}} \right. \kern-0pt} {\sum\limits_{i = 1}^{P} {\left( {\gamma_{Y} } \right)_{i} } }}, $$

(9)

being (p _Y ) _i  = (ω _Y ) _i /(ω _Y ) _i (γ _Y ) _i .(γ _Y ) _i the percentage of citations omitted by the database of interest, among those obtained in the year Y, referring to the i-th article examined.

Equation 9 shows that the p _Y value relating to a database can be seen as a weighted average of the omitted-citation rates of individual papers ((p _Y ) _i ). These contributions have a variable weight, given by the number of theoretically overlapping citations ((γ _Y ) _i ).

The dispersion of the (p _Y ) _i values can be roughly estimated by examining the relevant weighted quartiles, defined as ^w Q ⁽¹⁾ _Y , ^w Q ⁽²⁾ _Y and ^w Q ⁽³⁾ _Y . The construction of these indicators is analogous to that described in Sect. 4.1.

The surprising result is that the totality of the weighed quartiles are zero for both databases. This result is not incompatible with the fact that the weighted quartiles seen for individual journals (in Table 9) were not necessarily all zero. In this new case, we used time-windows of a single year when counting the (omitted) citations of citing papers; the incidence of articles with zero omitted citations is therefore greater than in the previous case. The practical consequence is that all non-zero (p _Y ) _i values fall beyond the third weighted quartile of the corresponding (weighted) distribution. As an example, the graph in Fig. 7 represents the weighted cumulative distribution relating to the (p _Y ) _i values for the year 2012, according to WoS. It can be noticed that the first seventy-six weighed percentiles are all zeros. Similar results can be found considering the remaining years.

Fig. 7

“Weighted” box-plot of the (p _J ) _i values relating to the papers in each journal (J), according to the WoS database. ^w Q ⁽¹⁾ _J , ^w Q ⁽²⁾ _J and ^w Q ⁽³⁾ _J are the first, second and third weighted quartile of the distributions of interest. Journal abbreviations are reported in Table 3

Fig. 8

“Weighted” box-plot of the (p _J ) _i values relating to the papers in each journal (J), according to the Scopus database. ^w Q ⁽¹⁾ _J , ^w Q ⁽²⁾ _J and ^w Q ⁽³⁾ _J are the first, second and third weighted quartile of the distributions of interest. Journal abbreviations are reported in Table 3

Fig. 9

Weighted cumulative distribution relating to the (p _Y ) _i values for the year 2012, according to WoS

This result confirms the fact that, although the p _Y values of the two databases tend to decrease over time, these variations are quite weak from a statistical viewpoint.

Additional tables

See Table 10 and 11.

Table 10 Annual number of articles (P) issued by each of the journals analyzed and indexed by both WoS and Scopus

Table 11 CPP, CPP ^* and relevant statistics for each of the journals examined, in the years from 2008 to 2012