, Volume 43, Issue 1, pp 27–43 | Cite as

The use and misuse of citation analysis in research evaluation

  • R. N. Kostoff


The present paper addresses some of the many possible uses of citations, including bookmark, intellectual heritage, impact tracker, and self-serving purposes. The main focus is on the applicability of citation analysis as an impact or quality measure. If a paper's bibliography is viewed as consisting of a directed (research impact or quality) component related to intellectual heritage and random components related to specific self-interest topics, then for large numbers of citations from many different citing paper, the most significant intellectual heritage (research impact or quality) citations will aggregate and the random author-specific self-serving citations will be scattered and not accumulate. However, there are at least two limitations to this model of citation analysis for stand-alone use as a measure of research impact of quality. First, the reference to intellectual heritage could be positive or negative. Second, there could be systemic biases which affect the aggregate results, and one of these, the “Pied Piper Effect”, is described in detail. Finally, the results of a short citation study comparing Russian and American papers in different technical fields are presented. The questions raised in interpreting this data highlight a few of the difficulties in attempting to interpret citation results without supplementary information.

Leydesdorff (Leydesdorff, 1998) addresses the history of citations and citation analysis, and the transformation of a reference mechanism into a purportedly quantitive measure of research impact/quality. The present paper examines different facets of citations and citation analysis, and discusses the validity of citation analysis as a useful measure of research impact/quality.


Science Citation Index Citation Analysis Citation Count Hypersonic Flow Research Impact 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Kostoff, R. N. (1994), Research impact quantification,R&D Management, 24: 3, July.Google Scholar
  2. Kostoff, R. N. (1995). Federal research impact assessment: Axioms, approaches, applications,Scientometrics, 34: 2.CrossRefGoogle Scholar
  3. Kostoff, R. N. (1997a),The Handbook of Research Impact Assessment, Seventh Edition. DTIC Report Number ADA296021. Also, available at Scholar
  4. Kostoff, R. N. (1997b), Peer review: The appropriate GPRA metric for research,Science, Vol. 277. 1 August. p. 651–652.CrossRefGoogle Scholar
  5. Kostoff, R. N. (1997c), Use and misuse of metrics in research evaluation,Science and Engineering Ethics, 3:2.Google Scholar
  6. Kostoff, R. N. (1989a), Hypersonic and supersonic flow roadmaps using bibliometrics and database tomography, (submitted for publication).Google Scholar
  7. Kostoff, R. N. (1998b), Database tomography for technical intelligence: A roadmap of the near-earth space science and technology literature,Information Processing and Management, 34:1.CrossRefGoogle Scholar
  8. Leydesdorff, L. (1998), Theories of Citation,Scientometrics, 43 (1) 5–25.CrossRefGoogle Scholar
  9. Macroberts, M. andMacroberts, B. (1996), Problems of citation analysis,Scientometrics, 36 (3) 435. Top 10 U. S. universities in clinical medicine research, 1990–1994,Science. Vol. 269. 1 September, 1995. p. 1223.CrossRefGoogle Scholar
  10. Scientists who fund themselves,Science, Vol. 279. 9 January, 1998, p. 179.Google Scholar

Copyright information

© Akadémiai Kiadó 1998

Authors and Affiliations

  • R. N. Kostoff
    • 1
  1. 1.Office of Naval ResearchArlington(USA)

Personalised recommendations