Skip to main content
SpringerLink
Log in

Rigor, Impact and Prestige: A Proposed Framework for Evaluating Scholarly Publications

  • Published:
Innovative Higher Education Aims and scope Submit manuscript

Abstract

As publication pressure has increased in the world of higher education, more journals, books, and other publication outlets have emerged. Thus it is critical to develop clear criteria for effectively evaluating the quality of publication outlets. Without such criteria funding agencies and promotion committees are forced to guess at how to evaluate a scholar’s portfolio. In this article, we explore the perils of evaluating journals based on a single quantitative measure (e.g., the Impact Factor rating of the Institute for Science Information). We then discuss key considerations for evaluating scholarship, including three main criteria: rigor, impact, and prestige. Finally, we conclude with examples of how these criteria could be applied in evaluating scholarship.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Amin, M., & Mabe, M. A. (2003). Impact factors: Use and abuse. Medicina (Buenos Aires), 63(4), 347–354.

    Google Scholar 

  • Barbui, C., Cipriani, A., Malvini, L., & Tansella, M. (2006). Validity of the impact factor of journals as a measure of randomized controlled trial quality. The Journal of Clinical Psychiatry, 67(1), 37–40.

    Article  Google Scholar 

  • Bekhradnia, B. (2009). Proposals for the research excellence framework—a critique. Retrieved from the Higher Education Policy Institute website: http://www.hepi.ac.uk

  • Bird, H. A. (2007). Presenting and publishing research data. In H. R. H. Patel, M. Arya, & I. S. Shergill (Eds.), Basic science techniques in clinical practice (pp. 118–125). New York, NY: Springer Science + Business Media.

    Google Scholar 

  • Blyth, E., Shardlow, S. M., Masson, H., Lyons, K., Shaw, I., & White, S. (2010). Measuring the quality of peer-reviewed publications in social work: Impact factors—liberation or liability? Social Work Education, 29(2), 120–136.

    Article  Google Scholar 

  • Boor, M. (1982). The citation impact factor: Another dubious index of journal quality. American Psychologist, 37(8), 975–977.

    Article  Google Scholar 

  • Corby, K. (2001). Method or madness? educational research and citation prestige. Portal: Libraries and the Academy, 1(3), 279–288. doi:10.1353/pla.2001.0040

    Article  Google Scholar 

  • D’Odorico, L. (2001). The citation impact factor in developmental psychology. Cortex, 37(4), 578–579.

    Article  Google Scholar 

  • Depken, C. A., & Ward, M. R. (2009). Sited, sighted, and cited: The effect of JSTOR in economic research. Unpublished manuscript. University of Texas at Arlington Working Paper.Retrieved from SSRN: http://ssrn.com/abstract=1472063

  • Fairbairn, H., Holbrook, A., Bourke, S., Preston, G., Cantwell, R., & Scevak, J. (2009). A profile of education journals. In P. Jeffrey (ed.) AARE 2008 Conference Papers Collection [Proceedings]. Available at http://www.aare.edu.au/08pap/fai08605.pdf. Accessed 17 Jan 2011.

  • Foley, J. A., & Sala, S. D. (2010). The impact of self-citation. Cortex, 46(6), 802–810.

    Article  Google Scholar 

  • Furr, L. A. (1995). The relative influence of social work journals: Impact factors versus core influence. Journal of Social Work Education, 31(1), 38–45.

    Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V. P., Moya-Anegón, F. (2011). The SJR indicator: A new indicator of journals’ scientific prestige. Retrieved from http://arxiv.org/ftp/arxiv/papers/0912/0912.4141.pdf

  • Gordge, R. (2011). Decision on assessing research impact. Higher Education Funding Council for England. Retrieved from http://www.hefce.ac.uk/research/ref/pubs/2011/01_11/

  • Harzing, A. (2011). Publish or perish, version 3.1.4004. Available at http://www.harzing.com/pop.htm

  • Haslam, N., & Laham, S. M. (2010). Quality, quantity, and impact in academic publication. European Journal of Social Psychology, 40(2), 216–220.

    Google Scholar 

  • Hirst, G. (1978). Discipline impact factors: A method for determining core journal lists. Journal of the American Society for Information Science, 29(4), 171–172.

    Article  Google Scholar 

  • Holcomb, T. L., Bray, K. E., & Dorr, D. L. (2003). Publications in educational/instructional technology: Perceived values of educational technology professionals. Educational Technology, 43(5), 53–57.

    Google Scholar 

  • Lavie, P. (2009). The race for the impact factor. Journal of Sleep Research, 18(3), 283–284.

    Article  Google Scholar 

  • Lee, Y., Driscoll, M. P., & Nelson, D. W. (2004). The past, present, and future of research in distance education: Results of a content analysis. American Journal of Distance Education, 18(4), 225–241.

    Article  Google Scholar 

  • Levine, T. R. (2010). Rankings and trends in citation patterns of communication journals. Communication Education, 59(1), 41–51.

    Article  Google Scholar 

  • Melby, C. S. (Ed). (2005). Examining the future of professional journals. Nursing &Health Sciences, 7(4), 219-220.

    Google Scholar 

  • Nkomo, S. M. (2009). The seductive power of academic journal rankings: Challenges of searching for the otherwise. The Academy of Management Learning and Education, 8(1), 106–112.

    Article  Google Scholar 

  • Opthof, T. (1997). Sense and nonsense about the impact factor. Cardiovascular Research, 33(1), 1.

    Article  Google Scholar 

  • Orey, M., Jones, S. A., & Branch, R. M. (2010). Educational media and technology yearbook. Vol. 35 (illustrated ed.). New York, NY: Springer.

    Google Scholar 

  • Rey-Rocha, J., Martín-Sempere, M. J., Martínez-Frías, J., & López-Vera, F. (2001). Some misuses of journal impact factor in research evaluation. Cortex, 37(4), 595–597.

    Article  Google Scholar 

  • Ritzhaupt, A. D., Sessums, C., Johnson, M. (2011, November). Where should educational technologists publish? An examination of journals within the field. Paper presented at the Association of Educational Communications and Technology, Jacksonville, Fl., USA.

  • Rousseau, R., & Hooydonk, G. V. (1996). Journal production and journal impact factors. Journal of the American Society for Information Science, 47(10), 775–780.

    Article  Google Scholar 

  • Sala, S. D., & Brooks, J. (2008). Multi-authors’ self-citation: A further impact factor bias? Cortex, 44(9), 1139–1145.

    Article  Google Scholar 

  • SciVerse Scopus. (2011). Content coverage guide. Retrieved from http://www.info.sciverse.com/UserFiles/sciverse_scopus_content_coverage_0.pdf

  • Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498–502.

    Article  Google Scholar 

  • Sievert, M., & Haughawout, M. (1989). An editor’s influence on citation patterns: A case study of Elementary School Journal. Journal of the American Society for Information Science, 40(5), 334–341.

    Article  Google Scholar 

  • Taris, T. (2006). Review of “citation analysis in research evaluation.”. European Journal of Work and Organizational Psychology, 15(3), 378–382.

    Article  Google Scholar 

  • Tyrer, P. (2008). Practical impact? Author’s reply. The British Journal of Psychiatry, 192(1), 69.

    Article  Google Scholar 

  • van Driel, M. L., Maier, M., & Maeseneer, J. D. (2007). Measuring the impact of family medicine research: Scientific citations or societal impact? Family Practice, 24(5), 401–402.

    Article  Google Scholar 

  • Wagner, A. B. (2009, Spring). Percentile-based journal impact factors: A neglected collection development metric. Issues in Science and Technology Librarianship, 57. Retrieved from http://www.istl.org/09-spring/refereed1.html

  • Weingart, P. (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics, 62(1):117–131.

  • Wicks, D. (2004). The institution of tenure: Freedom or discipline. Management Decision, 42(5), 619–627.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Instructional Psychology and Technology Department, Brigham Young University, Provo, UT, 84602, USA

    Richard E. West & Peter J. Rich

Authors
  1. Richard E. West
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter J. Rich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard E. West.

Rights and permissions

Reprints and permissions

About this article

Cite this article

West, R.E., Rich, P.J. Rigor, Impact and Prestige: A Proposed Framework for Evaluating Scholarly Publications. Innov High Educ 37, 359–371 (2012). https://doi.org/10.1007/s10755-012-9214-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10755-012-9214-3

Keywords

Search

Navigation