Skip to main content
SpringerLink
Log in

Peer Review in Social Sciences and Humanities. Addressing the Interpretation of Quality Criteria

  • Chapter
  • First Online:
The Evaluation of Research in Social Sciences and Humanities

Abstract

In SSH the definition of commonly agreed research quality criteria is problematic. Their implementation in the drafting of questions addressed to anonymous referees in the peer review process may create unintended distortions. The chapter reports on an initiative of ANVUR, following the publication of the first research assessment exercise (VQR 2004–2010), based on the creation of several expert groups in SSH. These groups were left free to formulate recommendations and criticisms about the evaluation process, in particular with respect to the peer review process. It turns out that many recommendations were in fact received and led to the formulation of the new exercise (2011–2014). When they were not received, the motivations were spelled out. It is argued that this conversational dynamics may greatly favour the convergence of scientific communities in SSH towards shared criteria.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Ahlgren, P., Colliander, C., & Persson, O. (2012). Field normalized rates, field normalized journal impact and Norwegian weights for allocation of university research funds. Scientometrics, 92(3), 767–780.

    Article  Google Scholar 

  • Archambault, E., Vignola-Gagné, E., Côté, G., Larivière, V., & Gingras, Y. (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases. Scientometrics, 68(3), 329–342. 1007.

    Google Scholar 

  • Ardanuy, J. (2013). Sixty years of citation analysis studies in the humanities (1995–2010). Journal of the American Society for Information Science and Technology, 64(8), 1751–1755.

    Article  Google Scholar 

  • Ardanuy, J., Urbano, C., & Quintana, L. (2009). Citation analysis of Catalan literary studies (1974–2003): Towards a bibliometrics of humanities studies in minority languages. Journal of the American Society for Information Science and Technology, 81, 347–366.

    Google Scholar 

  • Becher, T., & Trowler, P. R. (2001). Academics tribes and territories: Intellectual enquiry and the cultures of disciplines. Buckingham: Open University Press.

    Google Scholar 

  • Bonaccorsi, A. (2012). Potenzialità e limiti dell’analisi bibliometrica nelle discipline umanistiche e sociali. Nota di lavoro ANVUR. Available at www.anvur.org/attachments/article/44/valutazione_aree_umanistiche_e_sociali.pdf

  • Bonaccorsi, A. (2015). La valutazione possibile. Teoria e pratica nel mondo della ricerca. Il Mulino: Bologna.

    Google Scholar 

  • Bonaccorsi, A., Cicero, T., Ferrara, A., & Malgarini, M. (2015). Journal ratings as predictors of articles quality in Arts, Humanities and Social Sciences: An analysis on the Italian Research Evaluation Exercise. F1000Res, 4(196). 10.12688/f1000research.6478.1.

    Google Scholar 

  • Bourke, P., & Butler, L. (1996). Publication types, citation rates and evaluation. Scientometrics, 37(3), 473–494.

    Article  Google Scholar 

  • Chubin, D., & Hackett, E. (1990). Peerless science: Peer review and U.S. science policy. Albany: State University of New York Press.

    Google Scholar 

  • Committee on the National Plan for the Future of the Humanities. (2009). Sustainable humanities: Report from the committee on the national plan for the future of the humanities. Amsterdam: Amsterdam University Press. http://www.oapen.org/search?identifier=339995.

  • Cullars, J. M. (1989). Citation characteristics of French and German literary monographs. Library Quarterly, 59, 305–325.

    Article  Google Scholar 

  • Cullars, J. M. (1998). Citation characteristics of English-language monographs in philosophy. Library and Information Science Research, 20(1), 41–68.

    Google Scholar 

  • Engels, T. C., Ossenblok, T. L., & Spruyt, E. H. (2012). Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics, 93(2), 373–390.

    Article  Google Scholar 

  • Ferrara, A., & Bonaccorsi, A. (2016). How robust is journal rating in Humanities and Social Sciences? Evidence from a large-scale, multi-method exercise. Research Evaluation, 25(3), 279–291.

    Google Scholar 

  • Finkenstaedt, T. (1990). Measuring research performance in the humanities. Scientometrics, 19(5–6), 409–417.

    Article  Google Scholar 

  • Georgas, H., & Cullars, J. M. (2005). A citation study of the characteristics of the linguistics literature. College and Research Libraries, 66(6), 496–515.

    Article  Google Scholar 

  • Giménez-Toledo, E., & Román-Román, A. (2009). Assessment of humanities and social sciences monographs through their publishers: A review and a study towards a model of evaluation. Research Evaluation, 18(3), 201–213.

    Article  Google Scholar 

  • Giménez-Toledo, E., Tejada-Artigas, C., & Mañana-Rodriguez, J. (2013). Evaluation of scientific books’ publishers in social sciences and humanities: Results of a survey. Research Evaluation, 22(1), 64–77.

    Article  Google Scholar 

  • Giménez-Toledo, E., Mañana-Rodriguez, J., Engels, T. C., Ingwersen, P., Pölönen, J., Sivertsen, G., & Zuccala, A. A. (2015). The evaluation of scholarly books as research output. Current developments in Europe. In A. A. A. Salah, Y. Tonta, A. A. Akdag Salah, C. Sugimoto, & U. Al (Eds.), Proceedings of the 15th International Society for Scientometrics and Informetrics Conference, Istanbul, Turkey, 29th June to 4th July, 2015 (pp. 469–476). Istanbul: Bogazici University.

    Google Scholar 

  • Glänzel, W., & Schoepflin, U. (1999). A bibliometric study of reference literature in the sciences and social sciences. Information Processing & Management, 35(1), 31–44.

    Article  Google Scholar 

  • Gorraiz, J., Purnell, P. J., & Glänzel, W. (2013). Opportunities for and limitations of the book citation index. Journal of the American Society for Information Science and Technology, 64(7), 1388–1398.

    Article  Google Scholar 

  • Gruppo di ricerca ANVUR sulla Valutazione nell’area giuridica (GRAVAG). Verso la nuova VQR: Idee e proposte per la ricerca giuridica. Presentation to the Association of learned societies in legal disciplines. Rome, 26 June, 2015.

    Google Scholar 

  • Gruppo di ricerca sulla Valutazione in Architettura. (2015). Relazione finale. Available at http://www.anvur.it/index.php?option=com_content&view=article&id=871&Itemid=626&lang=it

  • Gruppo di ricerca sulla Valutazione nelle Scienze sociali e politiche. (2015). Relazione finale. Available at http://www.anvur.it/index.php?option=com_content&view=article&id=871&Itemid=626&lang=it

  • Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is originality in the social sciences and the humanities? American Sociological Review, 69(2), 190–212.

    Article  Google Scholar 

  • Hammarfelt, B. (2011). Interdisciplinary and the intellectual base of literature studies: Citation analysis of highly cited monographs. Scientometrics, 86(3), 705–725.

    Article  Google Scholar 

  • Hammarfelt, B. (2012). Harvesting footnotes in a rural field: citation patterns in Swedish literary studies. Journal of Documentation, 68(4), 536–558.

    Article  Google Scholar 

  • Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics, 101(2), 1419–1430.

    Article  Google Scholar 

  • Hammarfelt, B., & de Rijcke, S. (2015). Accountability in context: Effects of research evaluation systems on publication practices, disciplinary norms, and individual working routines in the faculty of Arts at Uppsala University. Research Evaluation, 24(1), 63–77.

    Article  Google Scholar 

  • Harley, D., & Krzys, A. S. (2011). Peer review in academic promotion and publishing: Its meaning, locus, and future. UC Berkeley: Center for Studies in Higher Education. http://escholarship.org/uc/item/1xv148c8.

    Google Scholar 

  • Hellqvist, B. (2010). Referencing in the humanities and its implications for citation analysis. Journal of the American Society for Information Science and Technology, 61(2), 310–318.

    Google Scholar 

  • Hemlin, S. (1996). Social studies of the humanities. A case study of research conditions and performance in ancient history and classical archaeology and English. Research Evaluation, 6(1), 53–61.

    Google Scholar 

  • Hicks, D. (2004). The four literatures of social science. In H. F. Moed, W. Glänzel, & U. Schmoch (Eds.), Handbook of quantitative science and technology research: The use of publication and patent statistics in studies of S&T systems (pp. 476–496). Dordrecht: Kluwer Academic Publishers.

    Google Scholar 

  • Hicks, D., & Wang, J. (2009). Towards a bibliometric database for the social sciences and humanities-a European scoping project (Annex 1 of the report ‘Towards a bibliometric database for the social sciences and humanities-a European scoping project’). Sussex: Science and Technology Policy Research Unit.

    Google Scholar 

  • Huang, M., & Chang, Y. (2008). Characteristics of research output in social sciences and humanities: From a research evaluation perspective. Journal of the American Society for Information Science and Technology, 59(11), 1819–1828.

    Article  Google Scholar 

  • Hug, S. E., Ochsner, M., & Daniel, H. D. (2013). Criteria for assessing research quality in the humanities: A Delphi study among scholars of English literature, German literature and art history. Research Evaluation, 22(5), 369–383.

    Article  Google Scholar 

  • Hug, S. E., Ochsner, M., & Daniel, H. D. (2014). A framework to explore and develop criteria for assessing research quality in the humanities. International Journal for Education Law and Policy, 10(1), 55–64.

    Google Scholar 

  • Knievel, J. E., & Kellsey, C. (2005). Citation analysis for collection development: A comparative study of eight humanities fields. The Library Quarterly, 75(2), 142–168.

    Article  Google Scholar 

  • Knorr-Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Kousha, K., Thelwall, M., & Rezaie, S. (2011). Assessing the citation impact of books: The role of google books, google scholar and scopus. Journal of the American Society for Information Science and Technology, 62(11), 2147–2164.

    Google Scholar 

  • Kousha, K., & Thelwall, M. (2009). Google book search: Citation analysis for social science and the humanities. Journal of the American Society for Information Science and Technology, 60(8), 1537–1549.

    Article  Google Scholar 

  • Kyvik, S. (2003). Changing trends in publishing behavior among university faculty, 1980–2000. Scientometrics, 58(1), 35–48.

    Article  Google Scholar 

  • Lamont, M. (2009). How professors think: Inside the curious world of academic judgment. Cambridge, MA: Harvard University Press.

    Book  Google Scholar 

  • Lamont, M., & Huutoniemi, K. (2011). Comparing customary rules of fairness: Evidence of evaluative practices in peer review panels. In C. Camic, N. Gross, & M. Lamont (Eds.), Social science in the making (pp. 209–232). Chicago: University of Chicago Press.

    Google Scholar 

  • Larivière, L., Archambault, È., Gringas, Y., & Vignola-Gagné, È. (2006). The place of serials in referencing practices: Comparing the natural sciences and engineering with social sciences and humanities. Journal of the American Society for Information Science and Technology, 57(8), 997–1004.

    Article  Google Scholar 

  • Leydesdorff, L., & Felt, U. (2012). ‘Books’ and ‘book chapters’ in the book citation index (BKCI) and science citation index (SCI, SoSCI, A&HCI). In Proceedings of the ASIS&T Annual Meeting, Vol. 49.

    Google Scholar 

  • Leydesdorff, L., Hammarfelt, B., & Salah, A. A. A. (2011). The structure of arts and humanities citation index: A mapping on the basis of aggregated citations among 1157 journals. Journal of the American Society for Information Science and Technology, 62(12), 2414–2426.

    Article  Google Scholar 

  • Lindholm-Romantschuk, Y., & Warner, J. (1996). The role of monographs in scholarly communication: An empirical study of philosophy, sociology and economics. Journal of Documentation, 52(4), 389–404.

    Article  Google Scholar 

  • Linmans, J. A. M. (2010). Why with bibliometrics the humanities does not need to be the weakest link. Indicators for research evaluation based on citations, libarary holdings and productivity measures. Scientometrics, 83(2), 337–354.

    Article  Google Scholar 

  • Lo Sasso, M. (2011). Valutazione della ricerca e progetto: Intervista ad Andrea Bonaccorsi. Techne, 02, 86–91.

    Google Scholar 

  • Mallard, G., Lamont, M., & Guetzkow, J. (2009). Fairness as appropriateness: Negotiating epistemological differences in peer review. Science, Technology and Human Values, 34(5), 573–606.

    Article  Google Scholar 

  • Martin, B., Tang, P., Morgan, M., Glänzel, W., Hornbostel, S., Lauer, G., & Zic-Fuchs, M. (2010). Towards a bibliometric database for the social sciences and humanities-A European scoping project (A report produced for DFG, ESRC, AHRC, NWO, ANR and ESF). Sussex: Sience and Technology Policy Research Unit.

    Google Scholar 

  • Moed, H. F. (2005). Citation analysis in research evaluation. Dordrecht: Springer.

    Google Scholar 

  • Moed, H. F., Luwel, M., & Nederhof, A. J. (2002). Towards research performance in the humanities. Library Trends, 50(3), 498–520.

    Google Scholar 

  • Moed, H. F., Linmans, J. A. M., Nederhof, A. J., Zuccala, A., López Illescas, C., & de Moya Anegón, F. (2010). Options for a comprehensive database of research outputs in social sciences and humanities (Annex 2 of the report ‘Towards a bibliometric database for the social sciences and humanities-a European scoping project’). Sussex: Science and Technology Policy Research Unit.

    Google Scholar 

  • Nederhof, A. J. (2006). Bibliometric monitoring of research performance in the social sciences and the humanities: A review. Scientometrics, 66(1), 81–100.

    Article  Google Scholar 

  • Nederhof, A. J. (2011). A bibliometric study of productivity and impact of modern language and literature research. Research Evaluation, 20(2), 117–129.

    Article  Google Scholar 

  • Nederhof, A. J., & Zwaan, R. (1991). Quality judgments of journals as indicators of research performance in the humanities and the social and behavioral sciences. Journal of the American Society for Information Science, 42(5), 332–340.

    Article  Google Scholar 

  • Nederhof, A. J., Zwaan, R. A., De Bruin, R. E., & Dekker, P. (1989). Assessing the usefulness of bibliometric indicators for the humanities and the social sciences: A comparative study. Scientometrics, 15(5–6), 423–435.

    Article  Google Scholar 

  • Novick, P. (1988). That noble dream: The ‘Objectivity Question’ and the American historical profession. Cambridge, MA: Cambridge University Press.

    Book  Google Scholar 

  • Ochsner, M., Hug, S. E., & Daniel, H.-D. (2012). Indicators for research quality in the humanities: opportunities and limitations. Bibliometrie—Praxis und Forschung, 1(4).

    Google Scholar 

  • Ochsner, M., Hug, S. E., & Daniel, H.-D. (2013). Four types of research in the humanities: Setting the stage for research quality criteria in the humanities. Research Evaluation, 22(4), 79–92.

    Google Scholar 

  • Ochsner, M., Hug, S. E., & Daniel, H.-D. (2014). Setting the stage for the assessment of research quality in the humanities: Consolidating the results of four empirical studies. Zeitschrift für Erziehungswissenschaft, 17(6 Supplement), 111–132.

    Article  Google Scholar 

  • Ochsner, M., Hug, S. E., & Daniel, H.-D. (2016). Research assessment in the Humanities. Towards criteria and procedures. Dordrecht: Springer.

    Book  Google Scholar 

  • Ossenblok, T. L., Engels, T. C., & Sivertsen, G. (2012). The representation of the social sciences and humanities in the Web of Science. A comparison of publication patterns and incentive structures in Flanders and Norway (2005–9). Research Evaluation, 21(4), 280–290.

    Article  Google Scholar 

  • Piro, F. N., Aksnes, D. W., & Rorstad, K. (2013). A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing. Journal of the American Society for Information Science and Technology, 64(2), 307–320.

    Article  Google Scholar 

  • Royal Netherlands Academy of Arts and Sciences. (2011). Quality indicators for research in the humanities. Amsterdam: Royal Netherlands Academy of Arts and Sciences. Available at https://www.knaw.nl/shared/resources/actueel/publicaties/pdf/qualityindicators-for-research-in-the-humanities.

    Google Scholar 

  • Seeber, M., & Bacchelli, A. (2017). Does single blind peer review hinder newcomers? Scientometrics, 113(1), 567–585.

    Google Scholar 

  • Sivertsen, G. (2010). A performance indicator based on complete data for the scientific publication output at research institutions. ISSI Newsletter, 6(1), 22–28.

    Google Scholar 

  • Sivertsen, G. (2014). Scholarly publication patterns in the social sciences and humanities and their coverage in Scopus and Web of Science. In E. Noyons (Ed.), Proceedings of the science and technology indicators conference 2014 Leiden (pp. 598–604). Leiden: Centre for Science and Technology Studies.

    Google Scholar 

  • Sivertsen, G., & Larsen, B. (2012). Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: An empirical analysis of the potential. Scientometrics, 91(2), 567–575.

    Article  Google Scholar 

  • Tang, M., Wang, C., Chen, K., & Hsiang, J. (2012). Exploring alternative cyber bibliometrics for evaluation of scholarly performance in the social sciences and humanities in Taiwan. In Proceedings of the ASIS&T Annual Meeting, Vol. 49.

    Google Scholar 

  • Thompson, J. W. (2002). The death of the scholarly monograph in the humanities? Citation patterns in literary scholarship. Libri, 52(3), 121–136.

    Article  Google Scholar 

  • Torres-Salinas, D., & Moed, H. F. (2009). Library catalogue analysis as a tool in studies of social sciences and humanities: An exploratory study of published book titles in economics. Journal of Informetrics, 3(1), 9–26.

    Article  Google Scholar 

  • Travis, G. D. L., & Collins, H. M. (1991). New light on old boys cognitive and institutional particularism in the peer review system. Science, Technology and Human Values, 16(3), 322–341.

    Article  Google Scholar 

  • Van Leeuwen, T. (2006). The application of bibliometric analyses in the evaluation of social science research. Who benefits from it, and why it is still feasible. Scientometrics, 66(1), 133–154.

    Article  Google Scholar 

  • Volkswagen Stiftung. (2014). What is intellectual quality in the humanities? Some guidelines. Hannover: Volkswagen Stiftung. Available at http://www.volkswagenstiftung.de/uploads/media/Humanities_Quality_Guidelines.pdf.

  • White, H. D., Boell, S. K., Yu, H., Davis, M., Wilson, C. S., & Cole, F. T. H. (2009). Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences. Journal of the American Society for Information Science and Technology, 60(6), 1083–1096.

    Google Scholar 

  • Whitley, R. (2000). The intellectual and social organization of the sciences (2nd ed.). Oxford: Oxford University Press.

    Google Scholar 

  • Zuccala, A. (2012). Quality and influence in literary work: Evaluating the ‘educated imagination’. Research Evaluation, 21(3), 229–241.

    Article  Google Scholar 

  • Zuccala, A., & van Leeuwen, T. (2011). Book reviews in humanities research evaluations. Journal of the American Society for Information Science and Technology, 62(10), 1979–1991.

    Article  Google Scholar 

  • Zuccala, A., van Someren, M., & van Bellen, M. (2014). A machine-learning approach to coding book reviews as quality indicators: Toward a theory of megacitation. Journal of the American Society for Information Science and Technology, 65(11), 1643–2330.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DESTEC, University of Pisa, Pisa, Italy

    Andrea Bonaccorsi

  2. IRVAPP-FBK, Trento, Italy

    Andrea Bonaccorsi

Authors
  1. Andrea Bonaccorsi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrea Bonaccorsi .

Editor information

Editors and Affiliations

  1. DESTEC, University of Pisa, Pisa, Italy

    Andrea Bonaccorsi

Appendix 1

Appendix 1

  • Architecture evaluation research group

  • Members: Cristina Bianchetti, Marco Biraghi, Marco Gaiani, Francesco Garofalo (coordinator), Sergio Poretti.

  • Social and Political Sciences evaluation research group

  • Members: Franco Garelli, Mauro Magatti, Leonardo Morlino (coordinator), Marino Regini, Irma Taddia, Salvatore Veca.

  • Humanities evaluation research group

  • Stefano Poggi (coordinator) Francesco Aceto, Giovanna Brogi, Alberto Camplani, Michele Corsi, Flavia Cuturi, Elena Dell’Agnese, Keir Douglas Elam, Vittorio Formentin, Filippo Maria Pontani, Adriano Roccucci, Mirella Schino, Giovanni Solimine, Roberto Tottoli.

  • Legal Studies evaluation research group

  • Vincenzo Militello (coordinator), Mauro Bussani, Cosimo Cascione, Giuseppe Conte, Pasquale De Sena, Carlo Fusaro, Diana-Urania Galetta, Tommaso Greco, Raffaello Lupi, Cinzia Motti, Lorenzo Zoppoli.

  • Business Studies evaluation research group

  • Stefano Brusoni (coordinator), Paola Adinolfi, Arnaldo Camuffo, Corrado Cerruti, Vittorio Coda, Alfonso Gambardella, Gianni Lorenzoni, Riccardo Palumbo, Alberto Quagli, Sandro Sandri, Elita Schillaci, Paola Schwizer, Alessandro Ruggieri, Alessandro Zattoni.

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Bonaccorsi, A. (2018). Peer Review in Social Sciences and Humanities. Addressing the Interpretation of Quality Criteria. In: Bonaccorsi, A. (eds) The Evaluation of Research in Social Sciences and Humanities. Springer, Cham. https://doi.org/10.1007/978-3-319-68554-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-68554-0_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-68553-3

  • Online ISBN: 978-3-319-68554-0

  • eBook Packages: Social SciencesSocial Sciences (R0)

Publish with us

Policies and ethics

Search

Navigation