Advertisement

Innovative Higher Education

, Volume 37, Issue 5, pp 359–371 | Cite as

Rigor, Impact and Prestige: A Proposed Framework for Evaluating Scholarly Publications

  • Richard E. WestEmail author
  • Peter J. Rich
Article

Abstract

As publication pressure has increased in the world of higher education, more journals, books, and other publication outlets have emerged. Thus it is critical to develop clear criteria for effectively evaluating the quality of publication outlets. Without such criteria funding agencies and promotion committees are forced to guess at how to evaluate a scholar’s portfolio. In this article, we explore the perils of evaluating journals based on a single quantitative measure (e.g., the Impact Factor rating of the Institute for Science Information). We then discuss key considerations for evaluating scholarship, including three main criteria: rigor, impact, and prestige. Finally, we conclude with examples of how these criteria could be applied in evaluating scholarship.

Keywords

Academia Scholarship Research Publication Standards 

References

  1. Amin, M., & Mabe, M. A. (2003). Impact factors: Use and abuse. Medicina (Buenos Aires), 63(4), 347–354.Google Scholar
  2. Barbui, C., Cipriani, A., Malvini, L., & Tansella, M. (2006). Validity of the impact factor of journals as a measure of randomized controlled trial quality. The Journal of Clinical Psychiatry, 67(1), 37–40.CrossRefGoogle Scholar
  3. Bekhradnia, B. (2009). Proposals for the research excellence framework—a critique. Retrieved from the Higher Education Policy Institute website: http://www.hepi.ac.uk
  4. Bird, H. A. (2007). Presenting and publishing research data. In H. R. H. Patel, M. Arya, & I. S. Shergill (Eds.), Basic science techniques in clinical practice (pp. 118–125). New York, NY: Springer Science + Business Media.Google Scholar
  5. Blyth, E., Shardlow, S. M., Masson, H., Lyons, K., Shaw, I., & White, S. (2010). Measuring the quality of peer-reviewed publications in social work: Impact factors—liberation or liability? Social Work Education, 29(2), 120–136.CrossRefGoogle Scholar
  6. Boor, M. (1982). The citation impact factor: Another dubious index of journal quality. American Psychologist, 37(8), 975–977.CrossRefGoogle Scholar
  7. Corby, K. (2001). Method or madness? educational research and citation prestige. Portal: Libraries and the Academy, 1(3), 279–288. doi: 10.1353/pla.2001.0040 CrossRefGoogle Scholar
  8. D’Odorico, L. (2001). The citation impact factor in developmental psychology. Cortex, 37(4), 578–579.CrossRefGoogle Scholar
  9. Depken, C. A., & Ward, M. R. (2009). Sited, sighted, and cited: The effect of JSTOR in economic research. Unpublished manuscript. University of Texas at Arlington Working Paper.Retrieved from SSRN: http://ssrn.com/abstract=1472063
  10. Fairbairn, H., Holbrook, A., Bourke, S., Preston, G., Cantwell, R., & Scevak, J. (2009). A profile of education journals. In P. Jeffrey (ed.) AARE 2008 Conference Papers Collection [Proceedings]. Available at http://www.aare.edu.au/08pap/fai08605.pdf. Accessed 17 Jan 2011.
  11. Foley, J. A., & Sala, S. D. (2010). The impact of self-citation. Cortex, 46(6), 802–810.CrossRefGoogle Scholar
  12. Furr, L. A. (1995). The relative influence of social work journals: Impact factors versus core influence. Journal of Social Work Education, 31(1), 38–45.Google Scholar
  13. González-Pereira, B., Guerrero-Bote, V. P., Moya-Anegón, F. (2011). The SJR indicator: A new indicator of journals’ scientific prestige. Retrieved from http://arxiv.org/ftp/arxiv/papers/0912/0912.4141.pdf
  14. Gordge, R. (2011). Decision on assessing research impact. Higher Education Funding Council for England. Retrieved from http://www.hefce.ac.uk/research/ref/pubs/2011/01_11/
  15. Harzing, A. (2011). Publish or perish, version 3.1.4004. Available at http://www.harzing.com/pop.htm
  16. Haslam, N., & Laham, S. M. (2010). Quality, quantity, and impact in academic publication. European Journal of Social Psychology, 40(2), 216–220.Google Scholar
  17. Hirst, G. (1978). Discipline impact factors: A method for determining core journal lists. Journal of the American Society for Information Science, 29(4), 171–172.CrossRefGoogle Scholar
  18. Holcomb, T. L., Bray, K. E., & Dorr, D. L. (2003). Publications in educational/instructional technology: Perceived values of educational technology professionals. Educational Technology, 43(5), 53–57.Google Scholar
  19. Lavie, P. (2009). The race for the impact factor. Journal of Sleep Research, 18(3), 283–284.CrossRefGoogle Scholar
  20. Lee, Y., Driscoll, M. P., & Nelson, D. W. (2004). The past, present, and future of research in distance education: Results of a content analysis. American Journal of Distance Education, 18(4), 225–241.CrossRefGoogle Scholar
  21. Levine, T. R. (2010). Rankings and trends in citation patterns of communication journals. Communication Education, 59(1), 41–51.CrossRefGoogle Scholar
  22. Melby, C. S. (Ed). (2005). Examining the future of professional journals. Nursing &Health Sciences, 7(4), 219-220.Google Scholar
  23. Nkomo, S. M. (2009). The seductive power of academic journal rankings: Challenges of searching for the otherwise. The Academy of Management Learning and Education, 8(1), 106–112.CrossRefGoogle Scholar
  24. Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33(1), 1.CrossRefGoogle Scholar
  25. Orey, M., Jones, S. A., & Branch, R. M. (2010). Educational media and technology yearbook. Vol. 35 (illustrated ed.). New York, NY: Springer.Google Scholar
  26. Rey-Rocha, J., Martín-Sempere, M. J., Martínez-Frías, J., & López-Vera, F. (2001). Some misuses of journal impact factor in research evaluation. Cortex, 37(4), 595–597.CrossRefGoogle Scholar
  27. Ritzhaupt, A. D., Sessums, C., Johnson, M. (2011, November). Where should educational technologists publish? An examination of journals within the field. Paper presented at the Association of Educational Communications and Technology, Jacksonville, Fl., USA.Google Scholar
  28. Rousseau, R., & Hooydonk, G. V. (1996). Journal production and journal impact factors. Journal of the American Society for Information Science, 47(10), 775–780.CrossRefGoogle Scholar
  29. Sala, S. D., & Brooks, J. (2008). Multi-authors’ self-citation: A further impact factor bias? Cortex, 44(9), 1139–1145.CrossRefGoogle Scholar
  30. SciVerse Scopus. (2011). Content coverage guide. Retrieved from http://www.info.sciverse.com/UserFiles/sciverse_scopus_content_coverage_0.pdf
  31. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498–502.CrossRefGoogle Scholar
  32. Sievert, M., & Haughawout, M. (1989). An editor’s influence on citation patterns: A case study of Elementary School Journal. Journal of the American Society for Information Science, 40(5), 334–341.CrossRefGoogle Scholar
  33. Taris, T. (2006). Review of “citation analysis in research evaluation.”. European Journal of Work and Organizational Psychology, 15(3), 378–382.CrossRefGoogle Scholar
  34. Tyrer, P. (2008). Practical impact? Author’s reply. The British Journal of Psychiatry, 192(1), 69.CrossRefGoogle Scholar
  35. van Driel, M. L., Maier, M., & Maeseneer, J. D. (2007). Measuring the impact of family medicine research: Scientific citations or societal impact? Family Practice, 24(5), 401–402.CrossRefGoogle Scholar
  36. Wagner, A. B. (2009, Spring). Percentile-based journal impact factors: A neglected collection development metric. Issues in Science and Technology Librarianship, 57. Retrieved from http://www.istl.org/09-spring/refereed1.html
  37. Weingart, P. (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics, 62(1):117–131.Google Scholar
  38. Wicks, D. (2004). The institution of tenure: Freedom or discipline. Management Decision, 42(5), 619–627.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Instructional Psychology and Technology DepartmentBrigham Young UniversityProvoUSA

Personalised recommendations