, Volume 15, Issue 3–4, pp 257–282 | Cite as

Journal-to-journal citation data: Issues of validity and reliability

  • R. E. Rice
  • Christine L. Borgman
  • Diane Bednarski
  • P. J. Hart


Citation analysis is a useful method for studying a wide range of topics in bibliometrics and the sociology of science. However, many challenges have been made to the validity and reliability of the underlying assumptions, the data, and the methods used in citation studies. This article addresses these issues in three parts. First is a brief review of validity and reliability issues in citation research. Next we explore measurement error in a principal source of journal-to-journal citation data, the Institute for Scientific Information'sJournal Citation Reports. Possible sources of measurement error include discrepancies between citing and cited data, changed or deleted journal titles, aberrant abbreviations, and listing algorithms. The last section is a detailed description of ways to overcome some of the measurement errors. The data and examples are drawn from a journal-to-journal citation study in the fields of Communication, Information Science, and Library Science.


Measurement Error Scientific Information Citation Analysis Principal Source Citation Data 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. J. BENIGER (1988). Information and communication: The new convergence.Communication Research, 15 (2), 198–218.Google Scholar
  2. R. W. BOHANNON (1986). Letter to the editor.Physical Therapy, 66(9), 1431–1432.Google Scholar
  3. C. L. BORGMAN & J. SCHEMENT (1988). Information science and communication science: An essay on convergence. In: PRENTICE, A. (Ed.),Information science as a discipline, (in press) NY: Schuman.Google Scholar
  4. B. BOYCE & C. BANNING (1979). Data accuracy in citation studies. RQ, 18, 349–150.Google Scholar
  5. R. BROADUS (1987). Early approaches to bibliometrics.Journal of the American Society for Information Science, 38(2), 127–129.Google Scholar
  6. L. D. BROWN & J. C. GARDNER (1985). Using citation analysis to assess the impact of journals and articles on contemporary accounting research.Journal of Accounting Research, 23(1), 84–109.Google Scholar
  7. R. S. BURT (1978). Stratification and prestige among elite experts in methodological and mathematical sociology circa 1975.Social Networks, 1, 105–158.Google Scholar
  8. R. S. BURT & P. DOREIAN (1982). Testing a structural model of perception: Conformity and deviance with respect to journal norms in elite sociological methodology.Quality and Quantity, 16, 109–150.Google Scholar
  9. A. R. BUSS & J. R. McDERMOTT (1976). Ratings of psychology journals compared to objective measures of journal impact.American Psychologist, 31(9), 675–678.Google Scholar
  10. M. CARPENTER & F. NARIN (1973). Clustering of scientific journals.Journal of the American Society for Information Science, 23, 425–436.Google Scholar
  11. H. CASON & M. LUBOTSKY (1936). The influence and dependence of psychological journals on each other.Psychological Bulletin, 33(2), 95–103.Google Scholar
  12. A. CAWKELL (1978). Evaluating scientific journals with Journal Citation Reports: A case study in acoustics.Journal of American Society for Information Science, 29, 41–46.Google Scholar
  13. D. CHUBIN (1983).Sociology of sciences: An annotated bibliography on invisible colleges, 1972–1981, NY: Garland Publishing.Google Scholar
  14. J. COLE & S. COLE (1971). Measuring the quality of sociological research: Problems in the use of the Science citation index.American Sociologist, 6, 23–29.Google Scholar
  15. D. CRANE (1972).Invisible colleges: Diffusion of knowledge in scientific communities. Chicago: University of Chicago Press.Google Scholar
  16. D. CRANE (1974). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals.The American Sociologist, 2, 195–201.Google Scholar
  17. L. CUMMINGS & P. FROST (Eds), (1985).Publishing in the organizational sciences. Homewood, ILL: Irwin.Google Scholar
  18. J. DICKMAN (1984). How do secondary services select journals?Cleveland Clinic Quarterly, 51, 575–578.Google Scholar
  19. J. DICKMAN & G. PLATEAU (1983). Foreign literature challenges to Chemical Abstracts Service.Journal of Chemical Information and Computer Science, 23(2), 70–73.Google Scholar
  20. P. DOREIAN (1985). Structural equivalence in a psychology journal network.Journal of the American Society for Information Science, November, 411–417.Google Scholar
  21. P. DOREIAN (1985). A measure of standing of journals in stratified networks.Journal of the American Society for Information Science, 8(5/6), 341–363.Google Scholar
  22. P. DOREIAN & T. FARARO (1985). Structural equivalence in a journal network.Journal of the American Society for Information Science, 36(1), 28–37.Google Scholar
  23. D. EDGE (1979). Quantitative measures of communication in science: A critical review.History of Science, 17, 102–133.Google Scholar
  24. E. GARFIELD (1979).Citation indexing: Its theory and application in science, technology and humanities. NY: Wiley.Google Scholar
  25. E. GARFIELD (1984). The 100 most-cited papers ever and how we select citation classics.Current Contents, 23, 3–9.Google Scholar
  26. W. GARVEY (1979).Communication: The essence of science. Facilitating information exchange among librarians, scientists, engineers, and students. NY: Pergamon Press.Google Scholar
  27. J. GOODRICH & C. ROLAND (1977). Accuracy of published medical reference citations.Journal of Technical Writing and Communication, 7, 15–19.Google Scholar
  28. M. GORDON (1982). Citation ranking versus subjective evaluation in the determination of journal hierarchies in the social sciences.Journal of the American Society for Information Science, 33, 55–57.Google Scholar
  29. W. HAGSTROM (1965).The scientific community. NY: Basic Books.Google Scholar
  30. P. J. HART (1987).The use and role of commercial online databases in four organizations. Unpublished PhD dissertation. Los Angeles: University of Southern California, Annenberg School of Communications.Google Scholar
  31. G. HIRST (1978). Discipline impact factors: A method for determining core journal lists.Journal of the American Society for Information Science, 29, 171–172.Google Scholar
  32. R. HJERPPE (1980).A bibliography of bibliometrics and citationindexing and analysis. (TRITA-LIB-2013). Stockholm: Royal Institute of Technology Library.Google Scholar
  33. Institute for Scientific Information. (1985).Journal Citation Reports. Philadelphia: Institute for Scientific Information.Google Scholar
  34. D. LINDSEY (1978).The scientific publication system in social science. San Francisco: Josey-Bass.Google Scholar
  35. M. LINE (1985). Changes in rank lists of serials over time: Interluding versus citation data.College & Research Libraries, 46(1), 77–79.Google Scholar
  36. S. LOCK (1986).A difficult balance: Editorial peer review in medicine. Philadelphia, PA: Institute for Scientific Information.Google Scholar
  37. T. LUCE & D. JOHNSON (1978). Rating of education and psychological journals.Educational Researcher, 7, 8–10.Google Scholar
  38. K. MACE & H. WARNER (1973). Ratings of psychology journals.American Psychologist, 28(2), 184–186.Google Scholar
  39. P. McALLISTER; R. ANDERSON & F. NARIN (1980). Comparison of peer and citation assessment of the influence of scientific journals.Journal of the American Society for Information Science, 31, 147–152.Google Scholar
  40. N. MIDORIKAWA (1983). Citation analysis of physics journals: Comparison of subfield of physics.Scientometrics, 5(6), 361–374.Google Scholar
  41. F. NARIN; M. CARPENTER & N. BERLT (1972). Inter-relationships of scientific journals.Journal of the American Society for Information Science, 23, 323–331.Google Scholar
  42. F. NARIN & D. GARSIDE (1972). Journal relationships in special education.Exceptional Children, 38(9), 695–703.Google Scholar
  43. W. NORD (1985). Looking at ourselves as we look at others: An exploration of the publication system for organizational research. In: L. CUMMINGS & P. FROST (Eds),Publishing in the organizational sciences. Homewood, ILL: Irwin.Google Scholar
  44. W. PAISLEY (1984). Communication in the communication sciences. In: B. DERVIN & M. VOIGT (Eds),Progress in the communication sciences, Vol. 5 (pp. 1–43.) Norwood, NJ: Ablex.Google Scholar
  45. D. PETERS & S. CECI (1982). Peer review practices of psychological journals: The fate of published articles, submitted again.Behavioral and Brain Science, 2(5), 187–195. (Cited inLock, 1985).Google Scholar
  46. A. PORTER (1977). Citation analysis: Queries and caveats.Social Studies of Science, 7, 257–267.Google Scholar
  47. W. POWELL (1985).Getting into print: The decision-making process in scholarly publishing. Chicago: University of Chicago Press.Google Scholar
  48. B. REEVES & C. L. BORGMAN (1983). A bibliometric evaluation of core journals in communication research.Human Communication Research, 10(1), 119–136.Google Scholar
  49. R. E. RICE; C. L. BORGMAN & B. REEVES (1988). Citation networks of communication journals, 1977–1985.Human Communicatiin Research, 15(2), 256–283.Google Scholar
  50. J. RUSHTON (1984). The validity of citation counts.Bulletin of the British Psychological Society, 37, 33–36.Google Scholar
  51. W. SATARIANO (1978). Journal use in sociology: Citation analysis versus readership patterns.Library Quarterly, 48(3), 293–300.Google Scholar
  52. A. SINGLETON (1976). Journal ranking and selection: A review in physics.Journal of Documentation, 32(4), 258–289.Google Scholar
  53. W. SCOTT (1974). Interreferee agreement on some characteristics of manuscripts submitted to theJournal of Personality and Social Psychology.American Psychologist, 29, 698–702. (Cited in Bohannon, 1986).Google Scholar
  54. H. SMALL (1981). The relationship of information science to the social sciences: A co-citation analysis.Information Processing and Management, 17, 39–50.Google Scholar
  55. J. SMART (1983). Perceived quality and citation rates of education journals.Research in Higher Education, 19(2), 175–182.Google Scholar
  56. L. SMITH (1981). Citation analysis.Library Trends, Summer, 83–106.Google Scholar
  57. N. SMITH & D. CAULLEY (1981). The evaluation of education journals through the study of citations.Educational Researcher, 10, 11–12.Google Scholar
  58. C. SO (1988). Citation networks of communication journals.Human Communication Research, 15(2), 236–255.Google Scholar
  59. SSCI Journal citation reports; a bibliometric analysis of social sciences journals in the ISI data base. Social Sciences Citation Index 1981 annual, vol. 7. (1981, 1984, 1985, 1986). Philadelphia: Institute for Scientific Information.Google Scholar
  60. F. THORNE (1977). The Citation Index: Another case of spurious validity.Journal of Clinical Psychology, October, 33, 1157–1161.Google Scholar
  61. Ulrich's International Periodicals Guide; A classified Guide to a Selected List of Current Periodicals, Foreign and Domestic. 25th ed, 1986–87. (1986). NY: R. R. Bowker.Google Scholar
  62. H. WHITE & B. GRIFFITH (1982). Authors as markers of intellectual space: Co-citation on studies of science, technology and society.Journal of Documentation, 38(4), 255–272.Google Scholar
  63. M. WHITE & K. WHITE (1977). Citation analysis of psychology journals.American Psychologist, 32, 301–305.Google Scholar
  64. N. WHITMAN & B. EYRE (1985). The pattern of publishing previously rejected articles in selected journals.Family Medicine, 17(1), 26–28.Google Scholar
  65. S. Jr. WIBERLEY (1982). Journal rankings from citation studies: A comparison of national and local data from social work.Library Quarterly, 52(4), 348–359.Google Scholar
  66. M. E. WILLIAMS (Ed.), (1985).Computer-Readable Databases: A Directory and Sourcebook, Vols 1–2. Chicago: American Library Association.Google Scholar
  67. L. ZHIGNESSE & C. OSGOOD (1967). Bibliographic citation characteristics of the psychological journal network in 1950 and 1960.American Psychologist, 22, 778–791.Google Scholar
  68. J. ZIMAN (1968).Public knowledge. Cambridge, England: Cambridge University Press.Google Scholar
  69. H. ZUCKERMAN & R. MERTON (1971). Patterns of evaluation in science: Institutionalization, structure, and functions of the referee system.Minerva, 9, 66–100. (Cited in BOHANNON 1986.)Google Scholar

Copyright information

© Akadémiai Kiadó 1989

Authors and Affiliations

  • R. E. Rice
    • 1
  • Christine L. Borgman
    • 2
  • Diane Bednarski
    • 3
  • P. J. Hart
    • 4
  1. 1.School of Communication, Information and Library StudiesRutgers UniversityNew Brunswick(USA)
  2. 2.Graduate School of Library and Information Science University of CaliforniaLos Angeles(USA)
  3. 3.OCLC Pacific Network 250 West First StClaremont(USA)
  4. 4.Dept. of Computer ScienceUniversity of Southern CaliforniaLos Angeles(USA)

Personalised recommendations