Skip to main content
Log in

On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Aristotle (undated/1959), Über die Seele (De anima) [= E. Gruber (Hrsg.), Werke Aristoteles, Bd. 13], Wissenschaftliche Buchgesellschaft, Darmstadt.

    Google Scholar 

  • Bagby, R. M., Parker, J. D., Bury, A. S. (1990), A comparative citation analysis of attribution theory and the theory of cognitive dissonance, Personality and Social Psychology Bulletin, 16: 274–283.

    Article  Google Scholar 

  • Bonzi, S., Snyder, H. W. (1991), Motivations for citation: A comparison of self citation and citation of others, Scientometrics, 21: 245–254.

    Article  Google Scholar 

  • Brooks, T. A. (1986), Evidence of complex citer motivations, Journal of the American Society for Information Science, 37: 34–36.

    Article  Google Scholar 

  • Brooks, T. A. (1988), Citer motivations. In: A. Kent (Ed.), Encyclopedia of Library and Information Science, Volume 43. Marcel Dekker, New York, NY, pp. 48–59.

    Google Scholar 

  • Brožek, J. (1980), The echoes of Wundt’s work in the United States, 1887–1977: A quantitative citation analysis, Psychological Research, 42: 103–107.

    Article  Google Scholar 

  • Chubin, D. E., Moitra, S. D. (1975), Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5: 423–441.

    Article  Google Scholar 

  • Cole, J., Cole, S. (1971), Measuring the quality of sociological research: Problems in the use of the Science Citation Index, American Sociologist, 6: 23–29.

    Google Scholar 

  • Cronin, B. (1984), The Citation Process: The Role and Significance of Citations in Scientific Communication, Taylor Graham, London, Great Britain.

    Google Scholar 

  • Endler, N. S., Rushton, J. P., Roediger, H. L. (1978), Productivity and scholary impact (citations) of British, Canadian, and U.S. Departments of Psychology (1975), American Psychologist, 33: 1064–1082.

    Article  Google Scholar 

  • Garfield, E. (1979), Citation Indexing: Its Theory and Application in Science, Technology, and Humanities, Wiley, New York, NY.

    Google Scholar 

  • Gray, P. H. (1983), Using science citation analysis to evaluate administrative accountability for salary variance, American Psychologist, 38: 116–117.

    Article  Google Scholar 

  • Liu, M. (1993), Progress in documentation — The complexities of citation practice: A review of citation studies, Journal of Documentation, 49: 370–408.

    Google Scholar 

  • May, R. M. (1997), The scientific wealth of nations, Science, 275: 793–796.

    Article  Google Scholar 

  • McPherson, A., Bonem, M., Green, G., Osborne, J. G. (1984), A citation analysis of the influence on research of Skinner’s Verbal Behavior, Behavior Analyst, 7: 157–167.

    Google Scholar 

  • Moravsik, M. J., Murugesan, P. (1975), Some results on the function and quality of citations, Social Studies of Science, 5: 86–92.

    Google Scholar 

  • Peritz, B. (1983), A classification of citation roles for the social sciences and related fields, Scientometrics, 5: 303–312.

    Article  Google Scholar 

  • Perlman, D. (1984), Recent developments in personality and social psychology: A citation analysis, Personality and Social Psychology Bulletin, 10: 493–501.

    Article  Google Scholar 

  • Schui, G., Krampen, G. (2006), Möglichkeiten und Grenzen der Anwendung bibliometrischer Indikatoren im Rahmen von Evaluationsvorhaben in der deutschsprachigen Psychologie (Possibilities and limits of applications of bibliometrical indicators within evaluations of psychology). In: G. Krampen, H. Zayer (Eds), Didaktik und Evaluation in der Psychologie (Teaching methods and evaluation in psychology). Hogrefe, Göttingen (Germany), pp. 11–26.

    Google Scholar 

Text material of authored and edited textbooks and encyclopaedias

  • Amelang, M., Bartussek, D. (2001), Differentielle Psychologie und Persönlichkeitsforschung (Differential psychology and personality research) (5th ed.), Kohlhammer, Stuttgart (Germany).

    Google Scholar 

  • Baumann, U., Perrez, M. (Eds) (1998), Lehrbuch Klinische Psychologie — Psychotherapie (Textbook clinical psychology — psychotherapy) (2nd ed.), Huber, Bern (Switzerland).

    Google Scholar 

  • Birbaumer, N., Schmidt, R. (2003), Biologische Psychologie (Biological psychology) (5th ed.), Springer, Berlin (Germany).

    Google Scholar 

  • Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1994–1997). Enzyklopädie der Psychologie: Pädagogische Psychologie (Encyclopedia of psychology: Educational psychology) (Serie D/I, Bd. 1–4), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2000), Enzyklopädie der Psychologie: Differentielle Psychologie und Persönlichkeitsforschung (Encyclopedia of Psychology: Differential psychology and personality research) (Serie C/VIII, Bd. 1–4), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2001), Enzyklopädie der Psychologie: Klinische Psychologie (Encyclopedia of psychology: Clinical psychology) (Serie D/II, Bd. 1–4), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2002), Enzyklopäde der Psychologie: Biologische Psychologie (Encyclopedia of psychology: Biological psychology) (Serie C/I, Bd. 1–7), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Davison, G., Neale, J., Hautzinger, M. (2002), Klinische Psychologie (Clinical psychology) (6th ed.), Beltz, Weinheim (Germany).

    Google Scholar 

  • Ehlert, U. (Ed.) (2003), Verhaltensmedizin (Behavioral medicine), Springer, Berlin (Germany).

    Google Scholar 

  • Frey, D., Irle, M. (Eds) (1998/2002), Theorien der Sozialpsychologie (Theories of social psychology) (3 Vol.), Huber, Bern (Switzerland).

    Google Scholar 

  • Graumann, C. G. (Ed.) (1972/1975), Sozialpsychologie (Social psychology) [= Gottschaldt, K., Lersch, P., Sander, F., Thomae, H. (Eds) Handbuch der Psychologie in 12 Bänden, Bd. 7(1) und Bd. 7(2)], Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Herkner, W. (2001), Lehrbuch Sozialpsychologie (Textbook social psychology) (2nd ed.), Huber, Bern (Switzerland).

    Google Scholar 

  • Herrmann, T., Lantermann, E.-D. (Eds) (1985), Persönlichkeitspsychologie (Psychology of personality), Urban & Schwarzenberg), München (Germany).

    Google Scholar 

  • Krapp, A., Weidenmann, B. (Eds) (2001), Pädagogische Psychologie (Educational psychology) (4th ed.), Beltz, Weinheim (Germany).

    Google Scholar 

  • Oerter, R., Montada, L. (Eds) (2002), Entwicklungspsychologie (Developmental psychology) (5th ed.), Beltz, Weinheim (Germany).

    Google Scholar 

  • Tausch, R., Tausch, A. (1998), Erziehungspsychologie (Psychology of education) (11th ed.), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Thomae, H. (Ed.) (1959), Entwicklungspsychologie (Developmental psychology) [= Gottschaldt, K., Lersch, P., Sander, F., Thomae, H. (eds) Handbuch der Psychologie in 12 Bänden, Bd. 3], Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Trautner, H. M. (1992), Lehrbuch der Entwicklungspsychologie (Textbook developmental psychology) (2 Vol., 2nd ed.), Hogrefe, Göttingen (Germany).

    Google Scholar 

Text material of test-manuals

  • Borkenau, P., Ostendorf, F. (1993), Neo-Fünf-Faktoren-Inventar (Neo-Five-Factors-Inventory) NEO-FFI), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Fahrenberg, J., Hampel, R., Selg, H. (2001), Freiburger Persönlichkeitsinventar (Freiburg Personality-Inventory) (FPI-R; 7th ed.), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Franke, G. H. (2002), Die Symptom-Checkliste von L.R. Derogatis (Symptom-Checklist) (SCL-90-R; 2nd ed.), Beltz, Weinheim (Germany).

    Google Scholar 

  • Grimm, H., Schöler, H. (1991), Heidelberger Sprachentwicklungstest (Heidelberg language development test) (HSET; 2nd ed.), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Kastner-Koller, V., Deimann, P. (1998), Der Wiener Entwicklungstest (Vienna development test) (WET), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Krohne, H.-W., Pulsack, A. (1995), Erziehungsstilinventar (Education-style inventory) (ESI; 2nd ed.), Beltz, Weinheim (Germany).

    Google Scholar 

  • Müller, R. (1980), Diagnostisches Soziogramm (Diagnostic sociogram), Westermann, Braunschweig (Germany).

    Google Scholar 

  • Rosenzweig, S., Hörmann, H., Moog, W. (1957), Rosenzweig P-F Test, Form für Erwachsene (Rosenzweig P-F Test, Adults) (PFT), Hogrefe, Göttingen (Germany).

    Google Scholar 

  • Tewes, U., Rossmann, P., Schallberger, U. (2000), Hamburg-Wechsler-Intelligenztest für Kinder (Hamburg-Wechsler-Intelligence-Test for Children) (HAWIK-III; 3rd ed.), Huber, Bern (Switzerland).

    Google Scholar 

  • Wittchen, H.-U., Zaudig, M., Fydrich, T. (1997), Strukturiertes Klinisches Interview für DSM IV — Achsen I und II (Structured clinical interview for DSM-IV, Axis I and II) (SKID), Hogrefe, Göttingen (Germany).

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Günter Krampen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Krampen, G., Becker, R., Wahner, U. et al. On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications. Scientometrics 71, 191–202 (2007). https://doi.org/10.1007/s11192-007-1659-2

Download citation

  • Received:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-007-1659-2

Keywords

Navigation