Long-term scientific impact revisited

Abstract

Citation-based measures are widely used as quantitative proxies for subjective factors such as the importance of a paper or even the worth of individual researchers. Here we analyze the citation histories of 4669 papers published in journals of the American Physical Society between 1960 and 1968 and argue that state-of-the-art models of citation dynamics and algorithms for forecasting nonstationary time series are very likely to fail to predict the long-term (50 years after publication) citation counts of highly-cited papers using citation data collected in a short period (say, 10 years) after publication. This is so because those papers do not exhibit distinctive short-term citation patterns, although their long-term citation patterns clearly set them apart from the other papers. We conclude that even if one accepts that citation counts are proxies for the quality of papers, they are not useful evaluative tools since the short-term counts are not informative about the long-term counts in the case of highly-cited papers.

GraphicAbstract

Appendices

Appendix A

The 4669 cumulative distribution functions exhibited in Fig. 1 can be grouped in a certain number K of classes (or clusters) according to their similarities. Here the dissimilarity or distance between the cumulative functions $c_i(t)$ and $c_j(t)$ of papers i and j is given by the sum of the squared errors

$$\begin{aligned} d_{ij} = \sum _{t=1}^T \left[ c_i(t) - c_j(t) \right] ^2. \end{aligned}$$

(A1)

Fig. 8

Mean quadratic error $\bar{d}$ that measures the mean of the squared distances of the cumulative distributions of the papers to the mean cumulative distributions of their classes as function of the number of classes K used as input to the K-means clustering algorithm

Fig. 9

Boxplots of the parameter $\mu _i$ of the WSB model (panel A) and of the time $\tau _i= \exp (\mu _i)$ necessary for paper i to reach the geometric mean of its final citations (panel B) for papers in classes $k=1,2,3,4$

Given the K classes obtained from the application of the K-means clustering algorithm [14] to the APS data set, we can readily determine the mean cumulative function $\bar{c}_k (t)$ for each class $k=1, \ldots ,K$,

$$\begin{aligned} \bar{c}_k (t) = \frac{1}{|\varOmega _k| }\sum _{i \in \varOmega _k } c_i(t) \end{aligned}$$

(A2)

where $\varOmega _k$ is the set of papers in class k and $| \varOmega _k |$ is the cardinality of $\varOmega _k$. Hence the squared distance of the cumulative distributions of the papers in class k to the mean cumulative distribution of class k is

$$\begin{aligned} \bar{d}_{k} = \sum _{i \in \varOmega _k } \sum _{t=1}^T \left[ c_i(t) - \bar{c}_k(t) \right] ^2. \end{aligned}$$

(A3)

Finally, we can define a measure of the goodness of the choice of the numbers of classes K used as input to the K-means clustering algorithm as the mean quadratic error $\bar{d} = \sum _{k=1}^K \bar{d}_{k}$, which is shown in Fig. 8 as a function of K. The idea of the elbow method is to pick the elbow of the curve $\bar{d} \times K$ as the number of classes to use, since adding another class does not result in a significant decrease of the error measure. Hence the choice $K=4$ used in the paper.

Appendix B

It is clear from Fig. 1 that different papers have very heterogeneous time scales. This heterogeneity is captured by the parameter $\mu _i$ of the WSB model (see Eq. 1), which determines the time $\tau _i$ necessary for paper i to reach the geometric mean of its final citations, viz, $\tau _i= \exp (\mu _i)$ [10]. Accordingly, in Fig. 9 we show the boxplots of $\mu _i$ and $\tau _i$ for papers in classes $k=1,2,3,4$, which demonstrate that papers in class $k=4$ have larger $\mu _i$ (and, consequently, larger $\tau _i$) values than papers in the other classes. In addition, these parameters exhibit a much larger variation for papers in class $k=4$ compared to the other classes.

Considering the remarkable distinctiveness of the parameters $\mu _i$ and $\tau _i$ for papers in class $k=4$, use of a same training period T for all papers, as usually done in the literature, may undermine the performance of the mechanistic models of citation dynamics for papers of extended longevity. A possible way to circumvent this issue is to introduce a paper-dependent training period $T_i = \tau ^* \tau _i$ for a fixed $\tau ^*$ so that papers of extended longevity (i.e., papers characterized by large $\tau _i$ values) are assigned long training periods. The training periods $T_i$ are shown in Fig. 10 for $\tau ^* = 0.001, 0.01$ and 0.1, where the white band indicates the region $T_i \in [1,600]$ months of training periods that are feasible to implement with the APS data set used in this paper. It is clear from this figure that there is no choice of $\tau ^*$ that includes all the 4669 papers considered in our study. Hence the paper-dependent training period cannot be implemented in practice, this being the reason we used the 10 years training period for all papers in the analysis of Sect. 5.

Fig. 10

Paper-dependent training periods $T_i = \tau ^* \tau _i$ for $\tau ^* = 0.001$ (panel A), 0.01 (panel B) and 0.1 (panel C). The white band indicates the region $T_i \in [1,600]$ months of training periods that are feasible to implement with the APS data set