Bibliometric indicators of young authors in astrophysics: Can later stars be predicted?

Abstract

We test 16 bibliometric indicators with respect to their validity at the level of the individual researcher by estimating their power to predict later successful researchers. We compare the indicators of a sample of astrophysics researchers who later co-authored highly cited papers before their first landmark paper with the distributions of these indicators over a random control group of young authors in astronomy and astrophysics. We find that field and citation-window normalisation substantially improves the predicting power of citation indicators. The sum of citation numbers normalised with expected citation numbers is the only indicator which shows differences between later stars and random authors significant on a 1 % level. Indicators of paper output are not very useful to predict later stars. The famous h-index makes no difference at all between later stars and the random control group.

Appendix

Most cited papers of later stars

In Table 3 the most cited papers of later stars are compared to their most cited papers published within the first 5 years they have papers in WoS-journals. The citation numbers p.a. (cit/year) are determined with a citation window of 5 years for the most cited papers in all years. For the most cited papers of the first 5 years the citation window is restricted to these 5 years because we suppose an evaluation directly after this period.

Table 3 Papers of later stars with highest annual citation numbers in all years and in the first 5 years of each later star (s.a. text)

All authors in Table 3 have got more than twice the number of citations p.a. for their most cited paper than for their most cited paper in their first 5 years. The author on rank 8 has got more citations for the early most cited paper than the authors on ranks 2, 9, and 21 for their most cited paper in the whole period of analysis. Omitting these three later stars would result in a clearer picture but we cannot omit them without the risk of a bias with respect to subfields because differing citation numbers could be caused by different citation behaviours in different subfields (see footnote 4, p. 3). Our definition of later stars can only be relative: they have got substantially higher citation numbers for their most cited paper after their first 5 years than within these years.

Nonetheless, we repeated all calculations for the smaller sample of 24 later stars. This gives us an impression how stable the results are under small changes of the sample. All p values are higher now—what is expected for a smaller sample if it would be generated by random reduction—but, again, the normalised number of citations $c_{\rm norm}$ is the best indicator ($p = 1.8\,\%$) and the h-index the worst measure ($p=50.8\,\%$).

Descriptions of indicators

Productivity indicators

Number of papers: This elementary indicator of productivity belongs to a bygone era when co-authorship was the exception and not the rule. It has the unwanted adverse effects of multiple publishing of the same results and of honorary authorships.

Fractional score: Each paper i is divided into $a_i$ fractions where $a_i$ is the number of authors. These fractions are summed up for the papers of the evaluated author. We use the simplest variant where all fractions of a paper are equal: $f = \sum _i 1/a_i.$ This indicator penalises honorary authorships and takes into account that larger teams can be more productive.

Total influence

All indicators of total influence tend to increase with the author’s number of papers. That means, they are also indicating productivity.

Number of citations: Each citation of a paper indicates that it has influenced the citing author(s). The sum $c=\sum _i c_i$ of raw numbers $c_i$ of citations of an author’s papers is highly field dependent. The paper’s number of citations $c_i$ depends on the age of a paper at the time of evaluation. Highly cited papers have surely some quality but less cited ones can also be of high quality.

Normalised numbers of citations: We normalise each paper’s number of citations $c_i$ by an expected number of citations $\hbox{E}(c_i)$ which takes into account the paper’s age and the citation behaviour in astrophysics during the first 5 (calendar) years in the paper’s lifetime (cf. section “Expected citation numbers” of Appendix). After normalising each paper’s citation number we sum the ratios of observed and expected citation numbers:

$$\begin{aligned} c_{\rm norm}=\sum _{i=1}^n \frac{c_i}{{\rm E(c_i)}}. \end{aligned}$$

Some bibliometricians do not calculate the sum of ratios but the ratio of sums (Schubert and Braun 1986): $\sum _i c_i/\sum _i{\rm E}(c_i).$ This procedure is thought to evaluate the whole oeuvre of an author but has been criticised recently for being not “consistent” (Opthof and Leydesdorff 2010; Waltman et al. 2011).¹⁰

The j-index: The j-index is the sum of the square roots of citation numbers of the author’s papers

$$\begin{aligned} j=\sum _{i=1}^n \sqrt{c_i}. \end{aligned}$$

It was proposed by Levene et al. (2012) to downgrade the influence of highly cited papers in the sum of citation numbers.

Fractional citations: Analogously to the fractional score described above we distribute citations of each paper equally to its authors:

$$\begin{aligned} \sum _{i=1}^n \frac{c_i}{ a_i}. \end{aligned}$$

Fractional normalised citations: The normalised numbers of citations can also be distributed among the authors involved (Radicchi and Castellano 2011):

$$\begin{aligned} \sum _{i=1}^n \frac{ c_i}{{\rm E}(c_i)a_i}. \end{aligned}$$

Typical influence

Mean citation number: The arithmetic mean of citations of an author’s papers

$$\begin{aligned} \frac{1}{n}\sum _{i=1}^nc_i=\frac{c}{n} \end{aligned}$$

is the simplest indicator of influence which does not tend to increase with the author’s productivity.

Mean fractional citations: The arithmetic mean of fractionally counted citations of an author’s papers:

$$\begin{aligned} \frac{1}{n}\sum _{i=1}^n \frac{c_i}{a_i}. \end{aligned}$$

Median of fractional citations: The median of fractionally counted citations of an author’s papers ${\rm median}(c_i/a_i)$ is considered because citation distributions are skewed.

Maximum of fractional citations: We wondered whether for a later star a large maximum of (fractional) citations $\max (c_i/a_i)$ is more typical than a large value of any measure of central tendency of citation numbers (Lehmann et al. 2008, cf. p. 375).

Indices of h-type

Hirsch index: The h-index was introduced by Hirsch (2005) “to quantify an individual’s scientific research output.” It is defined as the maximum rank $r$ in a rank list of an author’s papers according to their citation numbers $c_i$ which is less than or equal to the citation number $c_r$ of the paper with rank r:$h = \max ( r | c_r \ge r).$ The h-index has been criticised for its arbitrariness (van Eck and Waltman 2008). It is arbitrary because in the definition Hirsch “assumes an equality between incommensurable quantities” (Lehmann et al. 2008, p. 377), namely a rank and a citation number. Hirsch himself stated that his index depends on field-specific citation and collaboration behaviour (Hirsch 2005, p. 16571).

Egghe’s g-index: Egghe (2006) criticised the h-index for being insensitive to the citation frequency of an author’s highly cited papers. His g-index can be defined as the maximum rank r which is less than or equal to the mean citation number $(\sum _i^r c_i)/r$ of papers till rank $r$ (Schreiber 2008b). This condition is equivalent to $\sum _i^r c_i \ge r^2.$ That means, $g$ can also be defined as

$$\begin{aligned} g = \max \left( r | \sum _{i=1}^r c_i \ge r^2\right) . \end{aligned}$$

Fractional indices of h-type

Schreiber’s h _m -index: Fractional counting of papers or of citations could be applied to define an h-index which takes multi-authorship into account (Egghe 2008; Schreiber 2008c). Schreiber (2008a) argued that fractionally counted citations could remove highly cited papers from the h-core if they have a lot of authors. This led him to define the $h_{\rm m}$-index as the maximal effective rank $r_{\rm eff}(r)= \sum _i^r 1/a_i$ which is less than or equal to the number of citations c _r :

$$\begin{aligned} h_{\rm m} = \max ( r_{\rm eff} | c_{r(r_{\rm eff}) } \ge r_{\rm eff}). \end{aligned}$$

Egghe’s $g_{\rm f}$ -index: Egghe (2008) proposed to define a fractional g-index $g_{\rm f}$ as

$$\begin{aligned} g_{\rm f} = \max \left( r | \sum _{i=1}^{r} \frac{c_i}{a_i} \ge r^2\right) . \end{aligned}$$

Here the citations are counted fractionally.

Schreiber’s $g_{\rm m}$ -index: Schreiber (2009) proposed a fractional g-index $g_{\rm m}$ where both, papers and citations, are counted fractionally:

$$\begin{aligned} g_{\rm m} = \max \left( r_{\rm eff} | \sum _{i=1}^{r(r_{\rm eff})} \frac{c_i}{a_i} \ge r_{\rm eff}^2\right) . \end{aligned}$$

Expected citation numbers

Usually, for field normalisation expected citation numbers of papers are calculated as arithmetic means of citation numbers of all papers (of the same document type) published in all journals of the field in the same year. There are two main technical problems with this method, the rough delineation of fields and the skewness of citation distributions.

We do not evaluate single authors but only want to show the influence of field normalisation on distributions of citation indicators of authors. Therefore we can use a random sample of papers (for which we have already the citation data) instead of all papers in the field. This sample contains papers published in the years 1991–2009 by all 331 random authors of our initial control sample. We only consider those 2,342 papers with at most 20 authors. Figure 2 shows the average cumulated citation numbers in the publication year, 1 year later, 2 years later etc. Due to the skewness of citation distributions these arithmetic means fluctuate. Therefore we made a linear regression for each of the five time series of citation numbers of papers (not of the averages) but restricted the analysis to the years 1995–2007 (coloured part of the regression lines) where we have more than 100 papers in each year. The interpolated citation numbers obtained by linear regression are used as expected citation numbers ${\rm E}(c_i)$ of papers published in the corresponding years.

From these data we estimate a doubling of citation numbers in astrophysics in the two decades around the millennium. Calculating expected citation numbers as field averages is problematic because the arithmetic mean is not a good measure for the central tendency of skewed citation distributions. Lundberg (2007) therefore proposed to determine expected citation numbers as geometric means of citation numbers of papers in the field. Because papers can have zero citations he adds 1 to be able to calculate the geometric mean. This can be justified by saying that publishing a paper is the first citation of the published results.

Fig. 3

The two indicators with best p values

Fig. 4

The indicators on rank 3 and 4 according to p values: $p < 5\,\%$

Fig. 5

The indicators on rank 5 and 6 according to p values

Fig. 6

The indicators on rank 7 and 8 according to p values

Fig. 7

The indicators on rank 9 and 10 according to p values

Fig. 8

The indicators on rank 11 and 12 according to p values

Fig. 9

The indicators on rank 13 and 14 according to p values

Fig. 10

The indicators on rank 15 and 16 according to p values