Skip to main content
SpringerLink
Log in

The visible hand of research performance assessment

  • Published:
Higher Education Aims and scope Submit manuscript

Abstract

Far from allowing a governance of universities by the invisible hand of market forces, research performance assessments do not just measure differences in research quality, but yield themselves visible symptoms in terms of a stratification and standardization of disciplines. The article illustrates this with a case study of UK history departments and their assessment by the Research Assessment Exercise (RAE) and the Research Excellence Framework (REF), drawing on data from the three most recent assessments (RAE 2001, 2008, REF 2014). Symptoms of stratification are documented by the distribution of memberships in assessment panels, of research active staff, and of external research grants. Symptoms of a standardization are documented by the publications submitted to the assessments. The main finding is that the RAEs/REF and the selective allocation of funds they inform consecrate and reproduce a disciplinary center that, in contrast to the periphery, is well-endowed with grants and research staff, decides in panels over the quality standards of the field, and publishes a high number of articles in high-impact journals. This selectivity is oriented toward previous distributions of resources and a standardized notion of “excellence” rather than research performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Assessments of the evolution of the RAE/REF over the years are abound (Bence and Oppenheim 2005; Martin and Whitley 2010). While the RAE 2001 is above all characterized by a grade inflation and a subsequent much more concentrated funding policy by the HEFCE, the main change in the RAE 2008a, b, c was the introduction of research profiles for each department, based on what proportion of its publications was judged to be of national or international quality. The most important novelty of the REF is that “output quality” (now weighed at 60 %) is supplemented with “impact” (25 %) and “research environment” (15 %) (REF 2011).

  2. The list of publishers is based on previous insights into the discipline of history (Hamann 2014; Gengnagel and Hamann 2014), has been inductively expanded and revised during the investigation, and is widely congruent with the results of Zuccala et al. (2014).

  3. A complete list of journals can be requested from the author.

  4. The flow of research staff should always be seen in proportion to absolute research positions. While a 55 % increase in research staff for the “bottom 6” of 2008 corresponds to an absolute growth of 15.8 FTE research positions, the 17 % increase of research staff at the “top 6” departments in the same period equals an absolute growth of 35.5 FTE research positions. Although the differences in relative staff increase (55 and 17 %) may indicate the contrary, the gap between both rank groups still grows in favor of the top rank group.

  5. The extraordinary gap between the top and bottom groups in the RAE 2008 is caused by the financial position of the history department at UCL described in footnote 6.

  6. The peculiar financial position of the history department at UCL in Fig. 1 is caused by an institutional exception: from 1966 to 2012, the UCL Centre for the History of Medicine was primarily funded by the Wellcome Trust. Accordingly, during the period in question the trust awarded the Centre two grants, which explain the exceptional position of UCL in terms of external research funds (RAE 2008c).

  7. A less pronounced relation could be found between book chapters and rank group, no distinct relation could be found between edited books and rank groups. The data for chapters and edited volumes can be requested from the author.

  8. This is particularly remarkable since the history panel of the REF 2014 vowed not to “privilege any journal or conference rankings/lists, the perceived standing of the publisher or the medium of publication, or where the research output is published.” (REF 2012: 87).

References

  • Archambault, É., Vignola Gagné, É., Côté, G., Larivière, V., & Gingras, Y. (2006). Benchmarking scientific output in the social sciences and humanities: the limits of existing databases. Scientometrics, 68(3), 329–342.

    Article  Google Scholar 

  • Bence, V., & Oppenheim, C. (2005). The evolution of the UK’s Research Assessment Exercise: Publications, performance and perceptions. Journal of Educational Administration and History, 37(2), 137–155.

    Article  Google Scholar 

  • Benner, M., & Sandström, U. (2000). Institutionalizing the triple helix: research funding and norms in the academic system. Research Policy, 29(2000), 291–301.

    Article  Google Scholar 

  • Bourdieu, P. (1986). The forms of capital. In J. G. Richardson (Ed.), Handbook of Theory and Research for the Sociology of Education (pp. 241–258). New York: Greenwood Press.

    Google Scholar 

  • Bourdieu, P. (1988). Homo Academicus. Cambridge: Polity Press.

    Google Scholar 

  • Brown, R., & Carasso, H. (2013). Everything for Sale? The marketisation of UK higher education. London: Routledge.

    Google Scholar 

  • Burris, V. (2004). The academic caste system: Prestige hierarchies in PhD exchange networks. American Sociological Review, 69(2), 239–264.

    Article  Google Scholar 

  • Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90.

    Article  Google Scholar 

  • Campbell, K., Vick, D. W., Murray, A. D., & Little, G. F. (1999). Journal publishing, journal reputation, and the United Kingdom’s Research Assessment Exercise. Journal of Law and Society, 26(4), 470–501.

    Article  Google Scholar 

  • Cole, J. R., & Cole, S. (1973). Social Stratification in Science. Chicago: Chicago University Press.

    Google Scholar 

  • Davis, K., & Moore, W. E. (1944). Some principles of stratification. American Sociological Review, 10(2), 242–249.

    Article  Google Scholar 

  • Deem, R., Hillyard, S., & Reed, M. (2008). Knowledge, higher education, and the new managerialism: The changing management of UK universities. Oxford: Oxford University Press.

    Google Scholar 

  • Elton, L. (2000). The UK Research Assessment Exercise: Unintended consequences. Higher Education Quarterly, 54(3), 274–283.

    Article  Google Scholar 

  • Espeland, W. N., & Sauder, M. (2007). Rankings and reactivity. How public measures recreate social worlds. American Journal of Sociology, 113(1), 1–40.

    Article  Google Scholar 

  • Foucault, M. (2010). The government of self and others. Lectures at the Collège de France 1982–1983. New York: Palgrave Macmillan.

    Book  Google Scholar 

  • Gengnagel, V., & Hamann, J. (2014). The making and persisting of modern german humanities. Balancing acts between autonomy and social relevance. In R. Bod, J. Maat, & T. Weststeijn (Eds.), The making of the humanities III. The modern humanities (pp. 641–654). Amsterdam: Amsterdam University Press.

    Google Scholar 

  • Geuna, A., & Martin, B. R. (2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277–304.

    Article  Google Scholar 

  • Hamann, J. (2014). Die Bildung der Geisteswissenschaften. Zur Genese einer sozialen Konstruktion zwischen Diskurs und Feld. Konstanz: UVK.

    Google Scholar 

  • Hare, P. G. (2003). The United Kingdom’s Research Assessment Exercise: Impact on institutions, departments, individuals. Higher Education Management and Policy, 15(2), 43–61.

    Article  Google Scholar 

  • Harley, S. (2002). The impact of research selectivity on academic work and identity in UK universities. Studies in Higher Education, 27(2), 187–205.

    Article  Google Scholar 

  • Harley, S., & Lee, F. S. (1997). Research selectivity, managerialism, and the academic labor process: The future of nonmainstream economics in UK universities. Human Relations, 50(11), 1427–1460.

    Google Scholar 

  • Harman, G. (2005). Australian social scientists and transition to a more commercial university environment. Higher Education Research & Development, 24(1), 79–94.

    Article  Google Scholar 

  • Hazelkorn, E. (2007). The Impact of league tables and ranking systems on higher education decision making. Higher Education Management and Policy, 19(2), 1–24.

    Article  Google Scholar 

  • Henkel, M. (1999). The modernisation of research evaluation: The case of the UK. Higher Education, 38, 105–122.

    Article  Google Scholar 

  • Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261.

    Article  Google Scholar 

  • Kehm, B. M., & Leišytė, L. (2010). Effects of new governance on research in the humanities--. The example of medieval history. In D. Jansen (Ed.), Governance and performance in the German Public Research Sector. Disciplinary differences (pp. 73–90). Berlin: Springer.

    Chapter  Google Scholar 

  • Laudel, G. (2005). Is external research funding a valid indicator for research performance? Research Evaluation, 14(1), 27–34.

    Article  Google Scholar 

  • Lee, F. S. (2007). The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities. Cambridge Journal of Economics, 31(2), 309–325.

    Article  Google Scholar 

  • Lee, F. S., & Harley, S. (1998). Peer review, the Research Assessment Exercise and the demise of non-mainstream economics. Capital & Class, 22(3), 23–51.

    Article  Google Scholar 

  • Lee, F. S., Pham, X., & Gu, G. (2013). The UK Research Assessment Exercise and the narrowing of UK economics. Cambridge Journal of Economics, 37(4), 693–717.

    Article  Google Scholar 

  • Leišytė, L., & Westerheijden, D. (2014). Research Evaluation and Its Implications for Academic Research in the United Kingdom and the Netherlands. Discussion Papers des Zentrums für HochschulBildung, Technische Universität Dortmund, 2014(1), 3–32.

    Google Scholar 

  • Lucas, L. (2006). The Research Game in Academic Life. Maidenhead: Open University Press.

    Google Scholar 

  • Martin, B. R., & Whitley, R. D. (2010). The UK Research Assessment Exercise. A case of regulatory capture? In R. D. Whitley, J. Gläser, & L. Engwall (Eds.), Reconfiguring knowledge production. Changing authority relationships in the sciences and their consequences for intellectual innovation (pp. 51–80). Oxford: Oxford University Press.

    Chapter  Google Scholar 

  • Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56–63.

    Article  Google Scholar 

  • Merton, R. K. (1973). The sociology of science. Theoretical and empirical investigations. Chicago: University of Chicago Press.

    Google Scholar 

  • Moed, H. F. (2008). UK Research Assessment Exercises: Informed judgments on research quality or quantity? Scientometrics, 74(1), 153–161.

    Article  Google Scholar 

  • Morgan, K. J. (2004). The research assessment exercise in English universities, 2001. Higher Education, 48, 461–482.

    Article  Google Scholar 

  • Morrissey, J. (2013). Governing the academic subject: Foucault, governmentality and the performing university. Oxford Review of Education, 39(6), 797–810.

    Article  Google Scholar 

  • Münch, R. (2008). Stratifikation durch Evaluation. Mechanismen der Konstruktion und Reproduktion von Statushierarchien in der Forschung. Zeitschrift für Soziologie, 37(1), 60–80.

    Article  Google Scholar 

  • Münch, R., & Schäfer, L. O. (2014). Rankings, diversity and the power of renewal in science. A comparison between Germany, the UK and the US. European Journal of Education, 49(1), 60–76.

    Article  Google Scholar 

  • Power, M. (1997). The Audit Society. Rituals of verification. Oxford: Oxford University Press.

    Google Scholar 

  • RAE. (1992). Universities Funding Council. Research Assessment Exercise 1992: The Outcome. Circular 26/92 Table 62, History. http://www.rae.ac.uk/1992/c26_92t62.html. Accessed 08 Aug 2015.

  • RAE. (1996). 1996 Research Assessment Exercise. Unit of assessment: 59 History. http://www.rae.ac.uk/1996/1_96/t59.html. Accessed 08 Aug 2015.

  • RAE. (2001a). 2001 Research Assessment Exercise. Unit of Assessment: 59 History. http://www.rae.ac.uk/2001/results/byuoa/uoa59.htm. Accessed 08 Aug 2015.

  • RAE. (2001b). Panel list history. http://www.rae.ac.uk/2001/PMembers/Panel59.htm. Accessed 08 Aug 2015.

  • RAE. (2001c). Section III: Panels’ criteria and working methods. http://www.rae.ac.uk/2001/pubs/5_99/ByUoA/Crit59.htm. Accessed 08 Aug 2015.

  • RAE. (2001d). Submissions, UoA history. http://www.rae.ac.uk/2001/submissions/Inst.asp?UoA=59. Accessed 08 Aug 2015.

  • RAE. (2001e). What is the RAE 2001? http://www.rae.ac.uk/2001/AboutUs/. Accessed 08 Aug 2015.

  • RAE. (2008a). RAE 2008 panels. http://www.rae.ac.uk/aboutus/panels.asp. Accessed 08 Aug 2015.

  • RAE. (2008b). RAE 2008 quality profiles UOA 62 history. http://www.rae.ac.uk/results/qualityProfile.aspx?id=62&type=uoa. Accessed 08 Aug 2015.

  • RAE. (2008c). RAE 2008 submissions, UOA 62 history. http://www.rae.ac.uk/submissions/submissions.aspx?id=62&type=uoa. Accessed 08 Aug 2015.

  • REF. (2011). Assessment framework and guidance on submissions. http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf. Accessed 08 Aug 2015.

  • REF. (2012). Panel criteria and working methods, Part 2D: Main panel D criteria. http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12_2D.pdf. Accessed 08 Aug 2015.

  • REF. (2014a). Panel membership, Main panel D and sub-panels 27-36. http://www.ref.ac.uk/media/ref/content/expanel/member/Main%20Panel%20D%20membership%20%28Sept%202014%29.pdf. Accessed 08 Aug 2015.

  • REF. (2014b). REF 2014 results & submissions, UOA 30—History. http://results.ref.ac.uk/Results/ByUoa/30. Accessed 08 Aug 2015.

  • Royal Society. (2009). Journals under threat: a joint response from history of science, technology and medicine authors. Qualität in der Wissenschaft, 34(4), 62–63.

    Google Scholar 

  • Sayer, D. (2014). Rank Hypocrisies. The Insult of the REF. New York et al.: Sage.

    Google Scholar 

  • Sharp, S., & Coleman, S. (2005). Ratings in the Research Assessment Exercise 2001—The patterns of university status and panel membership. Higher Education Quarterly, 59(2), 153–171.

    Article  Google Scholar 

  • Slaughter, S., & Leslie, L. L. (1999). Academic Capitalism: Politics, Policies, and the Entrepreneurial University. Baltimore: John Hopkins University Press.

    Google Scholar 

  • Strathern, M. (1997). ’Improving ratings’: audit in the British University system. European Review, 5(3), 305–321.

    Article  Google Scholar 

  • Talib, A. A. (2001). The Continuing behavioural modification of academics since the 1992 Research Assessment Exercise. Higher Education Review, 33(3), 30–46.

    Google Scholar 

  • Tapper, T., & Salter, B. (2002). The external pressures on the internal governance of universities. Higher Education Quarterly, 56(3), 245–256.

    Article  Google Scholar 

  • Tapper, T., & Salter, B. (2004). Governance of higher education in Britain: The significance of the Research Assessment Exercise for the founding council model. Higher Education Quarterly, 58(1), 4–30.

    Article  Google Scholar 

  • Teixeira, P., Jongbloed, B. W., Dill, D., & Amaral, A. (Eds.). (2004). Markets in higher education. Rhetoric or reality? Dordrecht: Springer.

  • The Past Speaks. (2011). Rankings of history journals. http://pastspeaks.com/2011/06/15/erih-rankings-of-history-journals/. Accessed 08. Aug 2015.

  • Times Higher Education. (2008). Historians decry journal rankings. (2008, 04. Jan 2008).

  • Weber, M. (1978). Economy and society, 2 vol. Berkeley, Los Angeles: University of California Press.

    Google Scholar 

  • Whitley, R. D., Gläser, J., & Engwall, L. (Eds.). (2010). Reconfiguring knowledge production. Changing authority relationships in the sciences and their consequences for intellectual innovation. Oxford: Oxford University Press.

    Google Scholar 

  • Willmott, H. (2011). Journal list fetishism and the perversion of scholarship: reactivity and the ABS list. Organization, 18(4), 429–442.

    Article  Google Scholar 

  • Wooding, S., van Leeuwen, T. N., Parks, S., Kapur, S., & Grant, J. (2015). UK doubles its “World-Leading” research in life sciences and medicine in six years: Testing the claim? PLoS One, 10(7), e0132990.

    Article  Google Scholar 

  • Yokoyama, K. (2006). The effect of the research assessment exercise on organisational culture in English universities: collegiality versus managerialism. Tertiary Education and Management, 12(4), 311–322.

    Article  Google Scholar 

  • Zuccala, A., Guns, R., Cornacchia, R., & Bod, R. (2014). Can we rank scholarly book publishers? A bibliometric experiment with the field of history. Journal of the American Society for Information Science and Technology, 66(7), 1333–1347.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Forum Internationale Wissenschaft, Bonn University, Heussallee 18-24, 53113, Bonn, Germany

    Julian Hamann

Authors
  1. Julian Hamann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Julian Hamann.

Ethics declarations

Conflict of interest

The author declares that he has no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hamann, J. The visible hand of research performance assessment. High Educ 72, 761–779 (2016). https://doi.org/10.1007/s10734-015-9974-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10734-015-9974-7

Keywords

Search

Navigation