Skip to main content
SpringerLink
Log in

The inner quality of an article: Will time tell?

  • Published:
Scientometrics Aims and scope Submit manuscript

Each period is dominated by a mood, with the result that most men fail to see the tyrant who rules over them.

Albert Einstein to Maurice Solovine in 1938 (see Einstein and Infeld, 1938, The Evolution of Physics, p. xxii).

Time is the best censor. Frédérique Chopin (letter to his family, 1846)

How many errors Time has patience for, W. H. Auden (first stanza of Our Bias).

Abstract

In this paper, we assess whether quality survives the test of time in academia by comparing up to 80 years of academic journal article citations from two top journals, Econometrica and the American Economic Review. The research setting under analysis is analogous to a controlled real world experiment in that it involves a homogeneous task (trying to publish in top journals) by individuals with a homogenous job profile (academics) in a specific research environment (economics and econometrics). Comparing articles published concurrently in the same outlet at the same time (same issue) indicates that symbolic capital or power due to institutional affiliation or connection does seem to boost citation success at the beginning, giving those educated at or affiliated with leading universities an initial comparative advantage. Such advantage, however, does not hold in the long run: at a later stage, the publications of other researchers become as or even more successful.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. For a quantitative analysis of the outside dimension, see Chan, Frey, Gallus, Schaffner, Torgler and Whyte (2013, 2014).

  2. For more recent studies, see Watts and Gilbert (2011) for an agent-based simulation, and Azoulay et al. (2014) and Chan, Frey, Gallus and Torgler (2014a) for the citation patterns of papers published before the bestowal of an award. Although both these latter construct synthetic counterfactuals with the same pre-award citation structure, Azoulay et al. (2014) observe only a small citation boost over a short period because of the award, while Chan, Frey, Gallus and Torgler (2014b) observe a very large and long-lasting effect.

  3. See, for example, Price (1963), Coles (1970), Allison and Stewart (1974), Allison (1980), Redner (1998, 2005). For one of the journals that we analyse, the American Economic Review, 80 % of the citations received within the 1911–2011 period are from 20 % of the articles (Torgler and Piatti 2013).

  4. Admittedly, authors who studied at or work at a leading university may not only have better connections or an ability to influence the subject/topic of publications but may also be able to amass substantial experience, gather feedback and inspiration, and be exposed to the type of training that may be used to develop research that increases the inner quality of a paper.

  5. Kalaitzidakis et al. (2011) rank Econometrica and AER second and first in economics journals, respectively. The two are also ranked third and fourth, respectively, in the “ambition-adjusted journal ranking” devised by Engemann and Wall (2009).

  6. We exclude the Papers and Proceedings.

  7. www.proquest.com/products-services/pqdtglobal.html.

  8. Here, the sample size is reduced due to a lack of observations.

  9. Results are available from the authors upon request.

References

  • Albert, R. S. (1975). Toward a behavioral definition of genius. American Psychologist, 30, 140–151.

    Article  Google Scholar 

  • Allison, P. D. (1980). Inequality and scientific productivity. Social Studies of Science, 10, 163–179.

    Article  Google Scholar 

  • Allison, P. D., & Stewart, J. A. (1974). Productivity differences among scientists: Evidence for accumulative advantage. American Review of Sociology, 39, 576–606.

    Article  Google Scholar 

  • Amir, R., & Knauff, M. (2008). Ranking economics departments worldwide on the basis of PhD placement. Review of Economics and Statistics, 90, 185–190.

    Article  Google Scholar 

  • Aversa, E. S. (1985). Citation patterns of highly cited papers and their relationship to literature aging: A study of the working literature. Scientometrics, 7(3–6), 383–389.

    Article  Google Scholar 

  • Axarloglou, K., & Theoharakis, V. (2003). Diversity in economics: An analysis of journal quality perceptions. Journal of the European Economic Association, 1(6), 1402–1423.

    Article  Google Scholar 

  • Azoulay, P., Stuart, T., & Wang, Y. (2014). Matthew: Effect or fable? Management Science, 60(1), 92–109.

    Article  Google Scholar 

  • Baldi, S. (1998). Normative versus social constructivist processes in the allocation of citations: A network-analytic model. American Sociological Review, 63, 829–846.

    Article  Google Scholar 

  • Bjork. S., Offer, A., & Söderberg, G. (2014). Time series citation data: the Nobel Prize in economics. Scientometrics, 98(1), 185–196.

  • Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behaviour. Journal of Documentation, 64, 45–80.

    Article  Google Scholar 

  • Bourdieu, P. (1989). Social space and symbolic power. Sociological Theory, 71(1), 14–25.

    Article  Google Scholar 

  • Chan, H. F., Frey, B. S., Gallus, J., Schaffner, M., Torgler, B., & Whyte, S. (2013). External influence as an indicator of scholarly importance. CREMA Working Paper Series, 2013–2016.

  • Chan, H. F., Frey, B. S., Gallus, J., Schaffner, M., Torgler, B., & Whyte, S. (2014a). Do the best scholars attract the highest speaking fees? An exploration of internal and external influence. Scientometrics, 101(1), 793–817.

  • Chan, H. F., Frey, B. S., Gallus, J., & Torgler, B. (2014b). Academic honors and performance. Labour Economics, 31, 188–204.

    Article  Google Scholar 

  •  Chubin, D. E., Porter, A. L., & Rossini, F. A. (1984). “Citation classics” analysis: An approach to characterizing interdisciplinary research. Journal of the Association for Information Science and Technology, 35, 360–368.

    Google Scholar 

  • Cole, S. (1970). Professional standing and the reception of scientific discoveries. American Journal of Sociology, 76, 286–306.

    Article  Google Scholar 

  • Cole, J., & Singer, B. (1991). A theory of limited differences: Explaining the productivity puzzle. In J. C. Harriet Zuckerman & J. Bauer (Eds.), The Outer Circle: Women in the Scientific Community (pp. 277–340). New York: W.W. Norton and Company.

    Google Scholar 

  • Ductor, L. (2014). Does co-authorship lead to higher academic productivity? Oxford Bulletin of Economics and Statistics. doi:10.1111/obes.12070.

    Google Scholar 

  • Einstein, A., & Infeld, L. (1938). The Evolution of Physics. Cambridge: Cambridge University Press.

  • Engemann, K., & Wall, H. J. (2009). A journal ranking for the ambitious economist. Federal Reserve Bank of St. Louis Review, 91, 127–139.

    Google Scholar 

  • Eom, Y.-H., & Fortunato, S. (2011). Characterizing and modeling citation dynamics. PLoS One, 6(9), e24926. doi:10.1371/journal.pone.0024926.

    Article  Google Scholar 

  • Garfield, E. (1989). More delayed recognition Part 1. Examples from the genetics of color blindness, the entropy of short-term memory, phosphoinositides, and polymer rheology. Current Contents, 38, 3–8.

    Google Scholar 

  • Garfield, E. (1990). More delayed recognition Part 2. From inhibin to scanning electron microscopy. Current Contents, 9, 3–9.

    Google Scholar 

  • Gilbert, G. N. (1977). Referencing as persuasion. Social Studies in Science, 7, 113–122.

    Article  Google Scholar 

  • Glänzel, W., Schlemmer, B., & Thijs, B. (2003). Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon. Scientometrics, 58(3), 571–586.

    Article  Google Scholar 

  • Gupta, H. M., Campanha, J. R., & Pesce, R. A. G. (2005). Power-law distributions for the citation index of scientific publications and scientists. Brazilian Journal of Physics, 35(4), 981–986.

    Article  Google Scholar 

  • Hjørland, B. (2013). Citation analysis: A social and dynamic approach to knowledge organization. Information Processing and Management, 49(6), 1313–1325.

    Article  Google Scholar 

  • Hsu, J.-W., & Huang, D.-W. (2011). Dynamics of citation distribution. Computer Physics Communications, 182(1), 185–187.

    Article  MathSciNet  Google Scholar 

  • Huang, W. (2015). Do abcs get more citations than xyzs? Economic Inquiry, 53, 773–789.

    Article  Google Scholar 

  • Hudson, J. (2007). Be known by the company you keep: Citations—quality or chance? Scientometrics, 71, 231–238.

    Article  Google Scholar 

  • Johnston, D. W., Piatti, M., & Torgler, B. (2013). Citation success over time: Theory or empirics? Scientometrics, 95, 1023–1029.

    Article  Google Scholar 

  • Kalaitzidakis, P., Mamuneas, T. P., & Stengos, T. (2011). An updated ranking of academic journals in economics. Canadian Journal of Economics/Revue canadienne d’économique, 44, 1525–1538.

    Article  Google Scholar 

  • Kalaitzidakis, P., Theofanis, P. M., & Thanasis, S. (2003). Rankings of academic journals and institutions in economics. Journal of the European Economic Association, 1, 1346–1366.

    Article  Google Scholar 

  • Kodrzycki, Y. K., & Yu, P. (2006). New approaches to ranking economics journals. The B.E. Journal of Economic Analysis & Policy, 5(1), 1–44.

    Article  Google Scholar 

  • Landes, W. M. (2003). The test of time: Does 20th century American art survive? In V. A. Ginsburgh (Ed.), Economics of art and culture invited papers at the 12th international conference of the association of cultural economics international (contributions to economic analysis, volume 260) (pp. 143–164). London: Emerald Group Publishing Limited.

    Chapter  Google Scholar 

  • Latour, B., & Woolgar, S. (1979). Laboratory life: The social construction of scientific facts. London: Sage.

    Google Scholar 

  • Lawani, S. M. (1986). Some bibliometric correlates of quality in scientific research. Scientometrics, 9(1–2), 13–25.

    Article  Google Scholar 

  • Levitt, J. M., & Thelwall, M. (2008). Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects. Scientometrics, 77(1), 41–60.

    Article  Google Scholar 

  • Lotka, A. J. (1926). The frequency distribution of scientific productivity. Journal of the Washington Academic of Sciences, 16, 317–323.

    Google Scholar 

  • Mayer, T. (2004). Comment on “Dry holes in economic research”. Kyklos, 57, 621–625.

    Article  Google Scholar 

  • Mazlish, B. (1982). The quality of “the quality of science”: An evaluation. In M. C. La Follette (Ed.), Quality in science (pp. 48–67). Cambridge, MA: MIT Press.

    Google Scholar 

  • Merton, R. K. (1957). Priorities in scientific discovery: A chapter in the sociology of science. American Sociological Review, 22, 635–659.

    Article  Google Scholar 

  • Merton, R. K. (1968). The Matthew effect in science. Science, 159, 56–63.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigation. Chicago: Chicago University Press.

    Google Scholar 

  • Merton, R. K. (1988). The Matthew effect in science II: Cumulative advantage and the symbolism of intellectual property. Isis, 79, 606–623.

    Article  Google Scholar 

  • Mingers, J. (2008). Exploring the dynamics of journal citations: Modelling with s-curves. Journal of the Operational Research Society, 59, 1013–1025.

    Article  MATH  Google Scholar 

  • Ohba, N., & Nakao, K. (2012). Sleeping beauties in ophthalmology. Scientometrics, 93(2), 253–264.

    Article  Google Scholar 

  • Ponomarev, I. V., Williams, D. E., Hackett, C. J., Schnell, J. D., & Haak, L. L. (2014). Predicting highly cited papers: A method for early detection of candidate breakthroughs. Technological Forecasting & Social Change, 81, 49–55.

  • Price, D. D. S. (1963). Little science, big science. New York: Columbia University Press.

    Google Scholar 

  • Price, D. D. S. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the Association for Information Science and Technology, 27, 292–306.

    Google Scholar 

  • Putnam, L. L. (2009). Symbolic capital and academic fields: An alternative discourse on journal rankings. Management Communication Quarterly, 23, 127–134.

    Article  Google Scholar 

  • Redner, S. (1998). How popular is your paper? An empirical study of the citation distribution. The European Physical Journal B: Condensed Matter and Complex Systems, 4(2), 131–134.

    Article  Google Scholar 

  • Redner, S. (2005). Citation statistics from 110 years of physical review. Physics Today, 58(6), 49–53.

    Article  Google Scholar 

  • Roth, C., Wu, J., & Lozano, S. (2012). Assessing impact and quality from local dynamics of citation networks. Journal of Informetrics, 6(1), 111–120.

    Article  Google Scholar 

  • Sangwal, K. (2013). Growth dynamics of citations of cumulative papers of individual authors according to progressive nucleation mechanism: Concept of citation acceleration. Information Processing and Management, 49(4), 757–772.

    Article  Google Scholar 

  • Simmons, L. M, Jr, & West, G. B. (1981). Physics, philosophy, leadership, policy: An interview with Peter Carruthers. Los Alamos Science, 2, 133–143.

    Google Scholar 

  • Stack, S. (2004). Gender, children and research productivity? Research in Higher Education, 45, 891–920.

    Article  Google Scholar 

  • Stigler, G. J., Stigler, S. M., & Friedland, C. (1995). The journals of economics. Journal of Political Economy, 103, 331–359.

    Article  Google Scholar 

  • Torgler, B., & Piatti, M. (2013). A century of American economic review: Insights on critical factors in journal publishing. Basingstoke, UK: Palgrave/MacMillan.

    Book  Google Scholar 

  • Wall, H. (2009). Journal rankings in economics: handle with care. Working paper 2009-014A, Federal Reserve Bank of St. Louis.

  • Wallace, M. L., Larivière, V., & Gingras, Y. (2009). Modeling a century of citation distributions. Journal of Informetrics, 3(4), 296–303.

    Article  Google Scholar 

  • Wang, D., Song, C., & Barabási, A.-L. (2013). Quantifying long-term scientific impact. Science, 342(6154), 127–132.

    Article  Google Scholar 

  • Watts, C., & Gilbert, N. (2011). Does cumulative advantage affect collective learning in science? An agent-based simulation. Scientometrics. 89(1), 437–463.

    Article  Google Scholar 

Download references

Acknowledgments

For outstanding help thanks are due to Marco Piatti. For advice and suggestions thanks are due to two anonymous referees. We acknowledge financial support from the Australian Research Council (FT110100463).

Author information

Authors and Affiliations

  1. Queensland Behavioural Economics Group (QuBE), School of Economics and Finance, Queensland University of Technology, GPO Box 2434, Brisbane, QLD, 4001, Australia

    Ho Fai Chan, Lionel Page & Benno Torgler

  2. Ecole Nationale Statistique et de l’Administration Economique (ENSAE), 3 Avenue Pierre Larousse, Paris, France

    Malka Guillot

  3. Center for Research in Economics, Management and the Arts (CREMA), Zurich, Switzerland

    Benno Torgler

Authors
  1. Ho Fai Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Malka Guillot
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lionel Page
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Benno Torgler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benno Torgler.

Appendix

Appendix

See Tables 4 and 5 and Figs. 4, 5, 6, 7, 8.

Table 4 Institutional ranking
Full size table
Table 5 Results of random-effects GLS regression models (top 20 university and Ph.D.)
Full size table
Fig. 4
figure 4

Citation rank difference over time for authors belonging or not to a top 20 university

Full size image
Fig. 5
figure 5

Citation rank difference over time, by decade of publication (top 20 university)

Full size image
Fig. 6
figure 6

Citation rank difference over time for authors obtaining a Ph.D. in a top ten university

Full size image
Fig. 7
figure 7

Citation rank difference over time, by decade of publication (top ten Ph.D.)

Full size image
Fig. 8
figure 8

Contrasts of predictive margins (by top 20 university and Ph.D.)

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chan, H.F., Guillot, M., Page, L. et al. The inner quality of an article: Will time tell?. Scientometrics 104, 19–41 (2015). https://doi.org/10.1007/s11192-015-1581-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-015-1581-y

Keywords

Search

Navigation