Abstract
In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aristotle (undated/1959), Über die Seele (De anima) [= E. Gruber (Hrsg.), Werke Aristoteles, Bd. 13], Wissenschaftliche Buchgesellschaft, Darmstadt.
Bagby, R. M., Parker, J. D., Bury, A. S. (1990), A comparative citation analysis of attribution theory and the theory of cognitive dissonance, Personality and Social Psychology Bulletin, 16: 274–283.
Bonzi, S., Snyder, H. W. (1991), Motivations for citation: A comparison of self citation and citation of others, Scientometrics, 21: 245–254.
Brooks, T. A. (1986), Evidence of complex citer motivations, Journal of the American Society for Information Science, 37: 34–36.
Brooks, T. A. (1988), Citer motivations. In: A. Kent (Ed.), Encyclopedia of Library and Information Science, Volume 43. Marcel Dekker, New York, NY, pp. 48–59.
Brožek, J. (1980), The echoes of Wundt’s work in the United States, 1887–1977: A quantitative citation analysis, Psychological Research, 42: 103–107.
Chubin, D. E., Moitra, S. D. (1975), Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5: 423–441.
Cole, J., Cole, S. (1971), Measuring the quality of sociological research: Problems in the use of the Science Citation Index, American Sociologist, 6: 23–29.
Cronin, B. (1984), The Citation Process: The Role and Significance of Citations in Scientific Communication, Taylor Graham, London, Great Britain.
Endler, N. S., Rushton, J. P., Roediger, H. L. (1978), Productivity and scholary impact (citations) of British, Canadian, and U.S. Departments of Psychology (1975), American Psychologist, 33: 1064–1082.
Garfield, E. (1979), Citation Indexing: Its Theory and Application in Science, Technology, and Humanities, Wiley, New York, NY.
Gray, P. H. (1983), Using science citation analysis to evaluate administrative accountability for salary variance, American Psychologist, 38: 116–117.
Liu, M. (1993), Progress in documentation — The complexities of citation practice: A review of citation studies, Journal of Documentation, 49: 370–408.
May, R. M. (1997), The scientific wealth of nations, Science, 275: 793–796.
McPherson, A., Bonem, M., Green, G., Osborne, J. G. (1984), A citation analysis of the influence on research of Skinner’s Verbal Behavior, Behavior Analyst, 7: 157–167.
Moravsik, M. J., Murugesan, P. (1975), Some results on the function and quality of citations, Social Studies of Science, 5: 86–92.
Peritz, B. (1983), A classification of citation roles for the social sciences and related fields, Scientometrics, 5: 303–312.
Perlman, D. (1984), Recent developments in personality and social psychology: A citation analysis, Personality and Social Psychology Bulletin, 10: 493–501.
Schui, G., Krampen, G. (2006), Möglichkeiten und Grenzen der Anwendung bibliometrischer Indikatoren im Rahmen von Evaluationsvorhaben in der deutschsprachigen Psychologie (Possibilities and limits of applications of bibliometrical indicators within evaluations of psychology). In: G. Krampen, H. Zayer (Eds), Didaktik und Evaluation in der Psychologie (Teaching methods and evaluation in psychology). Hogrefe, Göttingen (Germany), pp. 11–26.
Text material of authored and edited textbooks and encyclopaedias
Amelang, M., Bartussek, D. (2001), Differentielle Psychologie und Persönlichkeitsforschung (Differential psychology and personality research) (5th ed.), Kohlhammer, Stuttgart (Germany).
Baumann, U., Perrez, M. (Eds) (1998), Lehrbuch Klinische Psychologie — Psychotherapie (Textbook clinical psychology — psychotherapy) (2nd ed.), Huber, Bern (Switzerland).
Birbaumer, N., Schmidt, R. (2003), Biologische Psychologie (Biological psychology) (5th ed.), Springer, Berlin (Germany).
Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1994–1997). Enzyklopädie der Psychologie: Pädagogische Psychologie (Encyclopedia of psychology: Educational psychology) (Serie D/I, Bd. 1–4), Hogrefe, Göttingen (Germany).
Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2000), Enzyklopädie der Psychologie: Differentielle Psychologie und Persönlichkeitsforschung (Encyclopedia of Psychology: Differential psychology and personality research) (Serie C/VIII, Bd. 1–4), Hogrefe, Göttingen (Germany).
Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2001), Enzyklopädie der Psychologie: Klinische Psychologie (Encyclopedia of psychology: Clinical psychology) (Serie D/II, Bd. 1–4), Hogrefe, Göttingen (Germany).
Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2002), Enzyklopäde der Psychologie: Biologische Psychologie (Encyclopedia of psychology: Biological psychology) (Serie C/I, Bd. 1–7), Hogrefe, Göttingen (Germany).
Davison, G., Neale, J., Hautzinger, M. (2002), Klinische Psychologie (Clinical psychology) (6th ed.), Beltz, Weinheim (Germany).
Ehlert, U. (Ed.) (2003), Verhaltensmedizin (Behavioral medicine), Springer, Berlin (Germany).
Frey, D., Irle, M. (Eds) (1998/2002), Theorien der Sozialpsychologie (Theories of social psychology) (3 Vol.), Huber, Bern (Switzerland).
Graumann, C. G. (Ed.) (1972/1975), Sozialpsychologie (Social psychology) [= Gottschaldt, K., Lersch, P., Sander, F., Thomae, H. (Eds) Handbuch der Psychologie in 12 Bänden, Bd. 7(1) und Bd. 7(2)], Hogrefe, Göttingen (Germany).
Herkner, W. (2001), Lehrbuch Sozialpsychologie (Textbook social psychology) (2nd ed.), Huber, Bern (Switzerland).
Herrmann, T., Lantermann, E.-D. (Eds) (1985), Persönlichkeitspsychologie (Psychology of personality), Urban & Schwarzenberg), München (Germany).
Krapp, A., Weidenmann, B. (Eds) (2001), Pädagogische Psychologie (Educational psychology) (4th ed.), Beltz, Weinheim (Germany).
Oerter, R., Montada, L. (Eds) (2002), Entwicklungspsychologie (Developmental psychology) (5th ed.), Beltz, Weinheim (Germany).
Tausch, R., Tausch, A. (1998), Erziehungspsychologie (Psychology of education) (11th ed.), Hogrefe, Göttingen (Germany).
Thomae, H. (Ed.) (1959), Entwicklungspsychologie (Developmental psychology) [= Gottschaldt, K., Lersch, P., Sander, F., Thomae, H. (eds) Handbuch der Psychologie in 12 Bänden, Bd. 3], Hogrefe, Göttingen (Germany).
Trautner, H. M. (1992), Lehrbuch der Entwicklungspsychologie (Textbook developmental psychology) (2 Vol., 2nd ed.), Hogrefe, Göttingen (Germany).
Text material of test-manuals
Borkenau, P., Ostendorf, F. (1993), Neo-Fünf-Faktoren-Inventar (Neo-Five-Factors-Inventory) NEO-FFI), Hogrefe, Göttingen (Germany).
Fahrenberg, J., Hampel, R., Selg, H. (2001), Freiburger Persönlichkeitsinventar (Freiburg Personality-Inventory) (FPI-R; 7th ed.), Hogrefe, Göttingen (Germany).
Franke, G. H. (2002), Die Symptom-Checkliste von L.R. Derogatis (Symptom-Checklist) (SCL-90-R; 2nd ed.), Beltz, Weinheim (Germany).
Grimm, H., Schöler, H. (1991), Heidelberger Sprachentwicklungstest (Heidelberg language development test) (HSET; 2nd ed.), Hogrefe, Göttingen (Germany).
Kastner-Koller, V., Deimann, P. (1998), Der Wiener Entwicklungstest (Vienna development test) (WET), Hogrefe, Göttingen (Germany).
Krohne, H.-W., Pulsack, A. (1995), Erziehungsstilinventar (Education-style inventory) (ESI; 2nd ed.), Beltz, Weinheim (Germany).
Müller, R. (1980), Diagnostisches Soziogramm (Diagnostic sociogram), Westermann, Braunschweig (Germany).
Rosenzweig, S., Hörmann, H., Moog, W. (1957), Rosenzweig P-F Test, Form für Erwachsene (Rosenzweig P-F Test, Adults) (PFT), Hogrefe, Göttingen (Germany).
Tewes, U., Rossmann, P., Schallberger, U. (2000), Hamburg-Wechsler-Intelligenztest für Kinder (Hamburg-Wechsler-Intelligence-Test for Children) (HAWIK-III; 3rd ed.), Huber, Bern (Switzerland).
Wittchen, H.-U., Zaudig, M., Fydrich, T. (1997), Strukturiertes Klinisches Interview für DSM IV — Achsen I und II (Structured clinical interview for DSM-IV, Axis I and II) (SKID), Hogrefe, Göttingen (Germany).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Krampen, G., Becker, R., Wahner, U. et al. On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications. Scientometrics 71, 191–202 (2007). https://doi.org/10.1007/s11192-007-1659-2
Received:
Issue Date:
DOI: https://doi.org/10.1007/s11192-007-1659-2