Scientometrics

, Volume 92, Issue 2, pp 429–441 | Cite as

Journal impact evaluation: a webometric perspective

Article

Abstract

In theory, the web has the potential to provide information about the wider impact of academic research, beyond traditional scholarly impact. This is because the web can reflect non-scholarly uses of research, such as in online government documents, press coverage or public discussions. Nevertheless, there are practical problems with creating metrics for journals based on web data: principally that most such metrics should be easy for journal editors or publishers to manipulate. Nevertheless, two alternatives seem to have both promise and value: citations derived from digitised books and download counts for journals within specific delivery platforms.

References

  1. Aguillo, I. F. (2009). Measuring the institution’s footprint in the web. Library Hi Tech, 27(4), 540–556.CrossRefGoogle Scholar
  2. Bar-Ilan, J. (2011). Articles tagged by ‘bibliometrics’ on Mendeley and CiteULike. Paper presented at the Metrics 2011 Symposium on Informetric and Scientometric Research.Google Scholar
  3. Barjak, F., & Thelwall, M. (2008). A statistical analysis of the web presences of European life sciences research teams. Journal of the American Society for Information Science and Technology, 59(4), 628–643.CrossRefGoogle Scholar
  4. Becher, T., & Trowler, P. (2001). Academic tribes and territories (2nd ed.). Milton Keynes: Open University press.Google Scholar
  5. Bogers, T., Bosch, A. V. D. (2008). Recommending scientific articles using citeulike. In: Proceedings of the 2008 ACM conference on Recommender Systems (RecSys ‘08), New York, pp. 287–290.Google Scholar
  6. Bollen, J., & Van de Sompel, H. (2008). Usage impact factor: the effects of sample characteristics on usage-based impact metrics. Journal of the American Society for Information Science and Technology, 59(1), 136–149.CrossRefGoogle Scholar
  7. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behaviour. Journal of Documentation, 64(1), 45–80.CrossRefGoogle Scholar
  8. Brin, S., & Page, L. (1998). The anatomy of a large scale hyper textual web search engine. Computer Networks and ISDN Systems, 30(1–7), 107–117.CrossRefGoogle Scholar
  9. Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology, 57(8), 1060–1072.CrossRefGoogle Scholar
  10. Brooks, T. A. (1986). Evidence of complex citer motivations. Journal of the American Society for Information Science, 37, 34–36.Google Scholar
  11. Chakrabarti, S., Dom, B., Raghavan, P., Rajagonpalan, S., Gibson, D., Kleinberg, J. M. (1998). Automatic resource compilation by analyzing hyperlink structure and associated text. Paper presented at the 7th International World Wide Web Conference, April 1998.Google Scholar
  12. Chakrabarti, S., Joshi, M. M., Punera, K., Pennock, D. M. (2002). The structure of broad topics on the web. http://www2002.org/CDROM/refereed/338.
  13. Chakrabarti, S., van den Berg, M., Dom, B. (1999). Focused crawling: a new approach to topic-specific web resource discovery. Paper presented at the 8th International World Wide Web Conference, May 1999.Google Scholar
  14. Cole, S., & Cole, J. R. (1967). Scientific output and recognition: a study in the operation of the reward system in science. American Sociological Review, 32(3), 377–390.CrossRefGoogle Scholar
  15. Coleman, A. (2007). Assessing the value of a journal beyond the impact factor. Journal of the American Society for Information Science and Technology, 58(8), 1148–1161.CrossRefGoogle Scholar
  16. COUNTER. (2011a). Counting online usage of networked electronic resources. http://www.projectcounter.org/. Accessed 14 December 2011.
  17. COUNTER. (2011b). Register of vendors providing usage reports compliant with release three of the code of practice for journals and databases. http://www.projectcounter.org/r2013/R2013CV_Dec2011.pdf. Accessed 14 December 2011.
  18. Cronin, B. (1984). The citation process: the role and significance of citations in scientific communication. London: Taylor Graham.Google Scholar
  19. Davis, P. (2011). Gaming the impact factor puts journal in time-out. http://scholarlykitchen.sspnet.org/2011/2010/2017/gaming-the-impact-factor-puts-journal-in-time-out/. Accessed 14 December 2011.
  20. De Costa, C. M. (2002). “The contagiousness of childbed fever”: a short history of puerperal sepsis and its treatment. The Medical Journal of Australia, 177(11/12), 668–671.Google Scholar
  21. Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4), e123.CrossRefGoogle Scholar
  22. Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science, 178(4060), 471–479.CrossRefGoogle Scholar
  23. Garfield, E. (2005). The agony and the ecstasy: the history and the meaning of the journal impact factor. Fifth International Congress on Peer Review in Biomedical Publication, Chicago. USA. http://garfield.library.upenn.edu/papers/jifchicago2005.pdf. Accessed 27 September 2007.
  24. Gyongyi, Z., Garcia-Molina, H., Pedersen, J. (2004). Combating web spam with TrustRank. Proceedings of the thirtieth international conference on very large data bases, 30, 576–587.Google Scholar
  25. Han, S., Ahn, Y.-Y., Moon, S., Jeong, H. (2006). Collaborative blog spam filtering using adaptive percolation search, WWW2006 Workshop. http://www.blogpulse.com/www2006-workshop/papers/collaborative-blogspam-filtering.pdf. Accessed 5 May 2006.
  26. Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457.Google Scholar
  27. Henning, V., Reichelt, J. (2008). Mendeley—A last.fm for research? In: IEEE Fourth International Conference on eScience (eScience ‘08), Los Alamitos, pp. 327–328.Google Scholar
  28. Huizenga, J. R. (1992). Cold fusion: the scientific fiasco of the century. Rochester: University of Rochester press.Google Scholar
  29. Ingwersen, P. (1998). The calculation of web impact factors. Journal of Documentation, 54(2), 236–243.CrossRefGoogle Scholar
  30. Jacsó, P. (2005). Google Scholar: the pros and the cons. Online Information Review, 29(2), 208–214.CrossRefGoogle Scholar
  31. Kleinberg, J. M. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5), 604–632.MathSciNetMATHCrossRefGoogle Scholar
  32. Kousha, K., & Horri, A. (2004). The relationship between scholarly publishing and the counts of academic in links to Iranian university web sites: exploring academic link creation motivations. Journal of Information Management and Scientometrics, 1(2), 13–22.Google Scholar
  33. Kousha, K., & Thelwall, M. (2007). Google Scholar citations and Google Web/URL citations: a multi-discipline exploratory analysis. Journal of the American Society for Information Science and Technology, 58(7), 1055–1065.CrossRefGoogle Scholar
  34. Kousha, K., & Thelwall, M. (2008). Assessing the impact of disciplinary research on teaching: an automatic analysis of online syllabuses. Journal of the American Society for Information Science and Technology, 59(13), 2060–2069.CrossRefGoogle Scholar
  35. Kousha, K., & Thelwall, M. (2009). Google Book search: citation analysis for social science and the humanities. Journal of the American Society for Information Science and Technology, 60(8), 1537–1549.CrossRefGoogle Scholar
  36. Kousha, K., Thelwall, M., & Rezaie, S. (2010). Using the web for research evaluation: the integrated online impact indicator. Journal of Informetrics, 4(1), 124–135.CrossRefGoogle Scholar
  37. Larson, R. R. (1996). Bibliometrics of the world wide web: an exploratory analysis of the intellectual structure of cyberspace. ASIS 59th annual meeting, Baltimore.Google Scholar
  38. Li, X., Thelwall, M., Giustini, D. (2011). Validating online reference managers for scholarly impact measurement. Scientometrics.Google Scholar
  39. Li, X., Thelwall, M., Wilkinson, D., & Musgrove, P. B. (2005). National and international university departmental web site interlinking, part 2: link patterns. Scientometrics, 64(2), 187–208.CrossRefGoogle Scholar
  40. Maxmen, A. (2010). Science networking gets serious. Cell, 141(3), 387–389.CrossRefGoogle Scholar
  41. Mayr, P., & Walter, A. K. (2007). An exploratory study of Google Scholar. Online Information Review, 31(6), 814–830.CrossRefGoogle Scholar
  42. Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: web of science vs. scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125.CrossRefGoogle Scholar
  43. Merton, R. K. (1973). The sociology of science. Theoretical and empirical investigations. Chicago: University of Chicago press.Google Scholar
  44. Meyer, M. (2003). Academic patents as an indicator of useful research? A new approach to measure academic inventiveness. Research Evaluation, 12(1), 17–27.CrossRefGoogle Scholar
  45. Moed, H. F. (2005). Statistical relationships between downloads and citations at the level of individual documents within a single journal. Journal of the American Society for Information Science and Technology, 56(10), 1088–1097.CrossRefGoogle Scholar
  46. Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.CrossRefGoogle Scholar
  47. Newton, I. (1676; 1992 republication). Letter from Isaac Newton to Robert Hooke, 5 February 1676. In: J.-P. Maury (Ed.), Newton: Understanding the Cosmos. New York: New Horizons.Google Scholar
  48. NISO. (2011). ANSI/NISO Z39.93-2007 the standardized usage statistics harvesting initiative (SUSHI) protocol. http://www.niso.org/apps/group_public/project/details.php?project_id=2021. Accessed 14 December 2011.
  49. Oppenheim, C. (2000). Do patent citations count? In B. Cronin & H. B. Atkins (Eds.), The web of knowledge: a festschrift in honor of Eugene Garfield (pp. 405–432). Metford: Information today Inc. ASIS monograph series.Google Scholar
  50. Priem, J., Costello, K. L. (2010). How and why scholars cite on Twitter. In: Proceedings of the American Society for Information Science and Technology (ASIST 2010), vol 47. pp. 1–4).Google Scholar
  51. Priem, J., Hemminger, B. M. (2010). Scientometrics 2.0: toward new metrics of scholarly impact on the social web. http://www.uic.edu/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2874/2570. First Monday, 15(7), Accessed 7 December 2011.
  52. Priem, J., Taraborelli, D., Groth, P., Neylon, C. (2011). Altmetrics: a manifesto. http://altmetrics.org/manifesto/. Accessed 14 December 2011.
  53. Provos, N., Mavrommatis, P., Rajab, M. A., Monrose, F. (2008). All your iFRAMEs point to us. http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/provos-2008a.pdf. Accessed 13 December 2011.
  54. Qiu, J. P., Chen, J. Q., & Wang, Z. (2004). An analysis of back link counts and web impact factors for Chinese university websites. Scientometrics, 60(3), 463–473.CrossRefGoogle Scholar
  55. RCUK. (2011). Types of impact. http://www.rcuk.ac.uk/documents/impacts/TypologyofResearchImpacts.pdf. Accessed 12 December 2011.
  56. Rodríguez, I., & Gairín, J. M. (1997). Valorando el impacto de la información en Internet: AltaVista, el “Citation Index” de la Red (Evaluating the impact of internet information: Altavista, the “Citation Index” of the web). Revista Española de Documentación Científica, 20(2), 175–181.CrossRefGoogle Scholar
  57. Rousseau, R. (1997). Sitations: an exploratory study. http://www.cindoc.csic.es/cybermetrics/articles/v2001i2001p2001.html. Cybermetrics, 1(1), Accessed 25 July 2006.
  58. Shepherd, P. (2011). Journal usage factor: results, recommendations and next steps. http://www.uksg.org/sites/uksg.org/files/JournalUsageFactorReport080711.pdf. Accessed 14 December 2011.
  59. Smith, A. G. (1999). A tale of two web spaces; comparing sites using web impact factors. Journal of Documentation, 55(5), 577–592.Google Scholar
  60. Thelwall, M. (2001). Extracting macroscopic information from web links. Journal of American Society for Information Science and Technology, 52(13), 1157–1168.CrossRefGoogle Scholar
  61. Thelwall, M. (2002). Conceptualizing documentation on the web: an evaluation of different heuristic-based models for counting links between university web sites. Journal of American Society for Information Science and Technology, 53(12), 995–1005.CrossRefGoogle Scholar
  62. Thelwall, M., & Harries, G. (2004). Do the web sites of higher rated scholars have significantly more online impact? Journal of the American Society for Information Science and Technology, 55(2), 149–159.CrossRefGoogle Scholar
  63. Thelwall, M., & Kousha, K. (2008). Online presentations as a source of scientific impact?: an analysis of powerpoint files citing academic journals. Journal of the American Society for Information Science and Technology, 59(5), 805–815.CrossRefGoogle Scholar
  64. Thomas, O., & Willett, P. (2000). Webometric analysis of departments of librarianship and information science. Journal of Information Science, 26(6), 421–428.CrossRefGoogle Scholar
  65. van Raan, A. F. J. (2001). Bibliometrics and internet: some observations and expectations. Scientometrics, 50(1), 59–63.CrossRefGoogle Scholar
  66. Vanclay, J. K. (2012). Impact factor: outdated artefact or stepping-stone to journal certification? Scientometrics.Google Scholar
  67. Vaughan, L., & Hysen, K. (2002). Relationship between links to journal web sites and impact factors. ASLIB Proceedings, 54(6), 356–361.CrossRefGoogle Scholar
  68. Vaughan, L., & Shaw, D. (2003). Bibliographic and web citations: what is the difference? Journal of the American Society for Information Science and Technology, 54(14), 1313–1322.CrossRefGoogle Scholar
  69. Vaughan, L., & Shaw, D. (2005). Web citation data for impact assessment: a comparison of four science disciplines. Journal of the American Society for Information Science and Technology, 56(10), 1075–1087.CrossRefGoogle Scholar
  70. Vaughan, L., & Shaw, D. (2008). A new look at evidence of scholarly citation in citation indexes and from web sources. Scientometrics, 74(2), 317–330.CrossRefGoogle Scholar
  71. Vaughan, L., & Thelwall, M. (2003). Scholarly use of the web: what are the key inducers of links to journal web sites? Journal of American Society for Information Science and Technology, 54(1), 29–38.CrossRefGoogle Scholar
  72. Vaughan, L., & Thelwall, M. (2005). A modeling approach to uncover hyperlink patterns: the case of Canadian universities. Information Processing and Management, 41(2), 347–359.CrossRefGoogle Scholar
  73. Weller, K., Dröge, E., Puschmann, C. (2011). Citation analysis in Twitter: approaches for defining and measuring information flows within tweets during scientific conferences. 1st Workshop on Making Sense of Microposts. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-2718/paper_2004.pdf. Accessed 16 June 2011.
  74. Weller, K., Puschmann, C. (2011). Twitter for scientific communication: how can citations/references be identified and measured? In: Proceedings of the poster session at the Web Science Conference 2011, Koblenz, Germany. http://journal.webscience.org/2500/). Accessed 21 October 2011.
  75. Whitley, R. (2000). The intellectual and social organization of the sciences (2nd ed.). Oxford: Oxford University press.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2012

Authors and Affiliations

  1. 1.Statistical Cybermetrics Research Group, School of TechnologyUniversity of WolverhamptonWolverhamptonUK

Personalised recommendations