Our evidence base for this paper comprises REF2014 data published by the UK Higher Education ‘trade paper’, Times Higher Education. We consulted this data to identify the extent to which Russell GroupFootnote 5 universities as a collective sub-grouping of research-led institutions within the UK’s HE sector, failed to submit members of FTE eligible staff to the REF2014 evaluation process. We also reproduce data relevant to these institutions’ Grade Point Average (GPA)Footnote 6 and GPA weighted with ‘research intensity’Footnote 7 scores to locate divergence and commonalities between these metrics and as a way of illuminating the gamesmanship undertaken by many research-intensive universities in the pursuit of research excellence.
What we have not done nor claim is a sophisticated statistical analysis of evaluation data. This is neither our intention nor interest. Instead, we have made simple calculations from existing data sets related to numbers and percentages of FTE eligible staff submitted to REF2014 to determine the extent of FTE eligible staff not submitted. We use the simplicity of these calculations, which are essentially basic subtractions, to evince what we perceive the patently obvious, yet largely ignored mass exclusion and derogation of the academic rank and file.
Finally, it is important to note that whilst the focus of our discussion is primarily with Russell Group institutions as the UK’s research elite, we would not wish to be seen to dismiss similar forms of exclusion brought about by REF2014 and suffered by academics populating non-Russell Group institutions. Furthermore, we recognize that the risk of exclusion through performance evaluation is not only a concern for academic researchers but academic teachers, who may be prospectively subjected in the UK to a Teaching Excellence Framework (TEF) (cf. Johnson 2015).
The ‘University of Excellence’
Writing in 1996, the late Bill Readings spoke of how the emergence of the “university of excellence” challenged the legitimacy of the university and those that worked within it; where Enlightenment grand narratives of truth and freedom had been superseded by a grand narrative of the market. Other commentators, like Jon Nixon (2008), have similarly bemoaned a state of crisis in the university where a struggle for competitive advantage between higher education institutions in national and international contexts has caused the corrosion and increasing evisceration of the idea of the university as a space of cultural and scientific critique and creativity. The marketization of higher education and an increased emphasis for universities to rationalize their activity and justify their public patronage on the terms of their economic and societal contributions, calls for a new kind of academic or academic specialization. Academics predisposed to performing in innovative and entrepreneurial ways are those most able to provide authoritative and compelling accounts of the ways with which they are excellentFootnote 8. These are, to borrow from Becher and Trowler (2001) a new “tribe and territory” of academics, more market conscious and savvy, more flexible in terms of adapting their identity and practice to a market logic, or alternatively more cognizant of their vulnerability in the context of a precarious and unforgiving labour market. These are academics that arguably are rated not for the excellence of teaching or research but their excellence of administration (Rolfe 2013). Beyond this cohort of academic entrepreneurs and largely capitalist elites lies a significant population at risk (cf. Jump 2014), unable, or perhaps unwilling, to comply, or in many cases gain equal entry and license to participate, with the standards and protocol demanded by the university of excellence.
Evaluating Excellence: Excluding Researchers
The attribution of excellence as a value is problematic because its meaning is polysemic; without consensus; and susceptible to producing radical uncertainty. More troubling is that beyond conceptual variance and disparateness is a misassumption that excellence is an achievement or goal when it is instead a largely redundant and meaningless qualifier (Rolfe 2013). Indeed, in our reckoning, an aspiration of excellence seems as much amorphous as futile; a truly Sisyphean pursuit. This is, we rationalize, why, where a universal qualification of excellence is elusive, its quantification assumes precedence. The quantification of excellence provides a way out of the incertitude of its qualification and according to NPM proselytes, enables articulation and measurement of efficiency and profitability. It is ostensibly, therefore, well purposed to ‘describe’Footnote 9 a market economy of knowledge and education. In our estimation, however, a shift towards the quantification of excellence provides no greater advantage than a U-turn into a cul-de-sac.
Significantly, not all research active staff were submitted by their institutions for consideration in REF2014. Instead, researchers were selected or ‘cherry-picked’ by institutional managers on the basis of ‘calculations’Footnote 10 or rather, safe-bets that their research would return high evaluation scores. Determining REF eligibility or ‘returnability’Footnote 11 necessitated most institutions administering an internal peer-review process that preceded the REF’s own ‘independent’ review of publications and impact case studies. The results of REF2014 confirmed considerable variation in the numbers of eligible staff submitted by their universities and an opportunity for universal submission being bypassed by institutional strategists in favour of ‘playing-the-game’ of Grade Points Average (GPA). Whilst GPA in simple terms is designed to reflect the ‘excellence’ of research outputs and impacts submitted for evaluation and comprises, therefore, the main indicator of the quality of research within an institution, large numbers of outputs and impacts that might have been considered within REF2014 were ignored and were failed to be included in many universities’ submissions. For institutions failing to return a majority of eligible academic staff, their research intensity, which reflects the depth of research quality across a whole institution, lessened. In some instances, the gap between GPA and research intensity was pronounced and provided firm indication of institutions playing the game, albeit at the expense of large numbers of academic staff.
When the initial REF2014 results were published, the trade press such as the Times Higher Education (THE) was without access to data, subsequently provided by the UK’s Higher Education Statistics Agency (HESA), covering numbers of staff returned against those not by institutions. An initial organization of scores by GPA reported in THE revealed a top ten populated in descending order by the Institute of Cancer Research, Imperial College London, London School of Economics and Political Science, University of Oxford, University of Cambridge, Cardiff University, King’s College London, University College London, University of Warwick and the London School of Hygiene and Tropical Medicine. However, only a week later when research intensity data was made available, THE published a new set of results and league positions that incorporated and were weighted according to research intensity. This altered the picture somewhat, with some new entrants to the top ten, some dropping out, and an overall reordering for those that remained. Most dramatically, one institution, Cardiff University, moved from a position of 6th in the initial GPA rankings to 50th when its submission was recalculated according to research intensity.
Of course, league tables may be interpreted and reported by their stakeholders in a variety of ways and there is perhaps no clear sense of which constitutive data set is most reliable or offers the most authoritative, complete or honest declaration of excellence. Notwithstanding, institutional claims of excellence may only be considered partial where attribution is made not to the entirety of an academic body but sub-set of it. The case of Cardiff University in REF2014 provides one such example of selective interpretation where institutional excellence was claimed but only linked to less than two-thirds (or 62%) of its eligible staff. What then for the remaining 38% perceived unfit to represent the university? Cardiff University was not alone in such game-playing. Other universities, particularly in Wales, were guilty of returning far fewer in REF2014. Neighbouring institutions such as the University of South Wales returned only 714 or 16% of its eligible staff, while Cardiff Metropolitan University returned a meagre 381 or 9% of its eligible staff. The extreme paucity of participation by academics from both of these institutions in REF2014 reflects an even more aggressive policy of selectivity operating within newer and less research-intensive, non-Russell Group institutionsFootnote 12, which necessitates further scrutiny. Slightly bucking the trend, Swansea University, as Wales’ second-highest ranking institution in both GPA and research intensity frameworks, submitted 71% of its eligible staff, moving it in the latter framework eight places above Cardiff University to 42nd.
From a sociological perspective, the most alarming yet less reported aspect of research evaluation is the failure to universalize the research evaluation process and the blatantly, exclusionary policies pursued by some institutions. When we look across the UK’s Russell Group institutions, the purported heartland of the UK’s research excellence, the cumulative figures are even more troubling. In Table 1 we provide an alternative ranking for the REF based on the number of eligible staff excluded from the evaluation process.
An examination of the figures published by THE reveal not only significant variance between the ranking position of institutions by GPA and intensity weighted GPA but a significant number of eligible academics not returned by Russell Group institutions. Our calculations are that while, 28,251 FTE eligible staff were submitted across the Russell Group, 5,941 FTE eligible staff were not; which corresponds to 21% of Russell Group ‘eligible’ academics not featuring within REF2014. From another perspective, the number of eligible ‘Russell Group’ academics not participating in REF2014 would be greater than the combined submissions of Cambridge, Oxford and King’s College London. Whilst this in itself is something of a hyperbolic analogy, it is nevertheless a calculation that disrupts the claims of many Russell Group institutions to research greatness.
In our alternative ranking of Russell Group institutions not by GPA or research intensity weighted GPA but number of staff not participating in REF2014, Cardiff University leads the way while the Universities of Cambridge and Queen’s University Belfast in joint last are arguably the most inclusive of the Russell Group institutions, both omitting ‘only’ 5% each of their eligible staff. Nonetheless the entirety of the Russell Group fraternity appears to claim the same kind of research excellence. How can this be the case? While the claims of Cambridge appear strongest where the percentage representation of staff and concurrently high GPA and intensity weighted scores evidence slight fluctuation, divergence between GPA and intensity weighted GPA by and across institutions suggests that institutional claims of research evidence are rather ambiguous. Furthermore, without universal submission, whole-institutional claims of research excellence are rendered less than persuasive. Ultimately, what these configurations and various value matrices demonstrate is not only elasticity in the interpretation and application of REF2014 results, but that as an evaluation exercise, issues of comparability and, therefore, ranking were fraught by gulfs in the institutional treatment of academics as either essential or expendable. Furthermore, these results may be interpreted as revealing, at either end, the extent to which institutions’ embraced either a model of neoliberal managerialism or collegial democracy.
While many institutions claimed success and evidence of their competitiveness through REF2014, there exist reports of individuals whose exclusion from it has resulted not only in a sense of professional failure and degradation but fear of and subjection to intimidation and bullying. Even before the publication of REF2014 results, a poll conducted by the UK’s Guardian newspaper into work-place bullying revealed that 73% of a cohort of 1,366 respondents felt that the REF was further contributing to a bullying culture in UK universities. This figure rose to 81% among those working in Russell Group institutions and further still among Researcher (85%) and PhD candidates (91%). The UK University and College UnionFootnote 13 (UCU) (2013) also reported on the particular challenge of the REF for early career academics and its influence in accentuating requirements for entry-level academic positions; predominantly a demand for applicants to demonstrate four ‘returnable’ or ‘REF-able’ outputs (Bishop 2015). This kind of criterion, however, we would suggest, is an unreasonable if not entirely unrealistic expectation where many, if not most early career academics will suffer the precariousness of insecure and short-term appointments that do not permit independent research and may, furthermore, result in applicants deviating from and abandoning the focus of their doctoral work. Only a few benefit from funded post-doctoral fellowships that provide time and finance with which to translate doctoral theses into the pre-requisite quartet of outputs. Yet even then, ‘returnability’ is far from guaranteed where the quality threshold demanded by the REF is dizzying and where only outputs considered three to four star quality attract Government remuneration. To provide greater context to this, outputs considered to be of two-star quality, defined in REF2014 as “quality that is recognized internationally in terms of originality, significance and rigour” (www.ref.ac.uk), would not have been considered ‘REF-able’ and, therefore, would likely have been excluded from institutional submissions. So, in other words, in the terms of REF2014, to conduct research that is internationally recognized is to fall some way short. Taking this a step further, four-star quality, which was defined as “quality that is world-leading in terms of originality, significance and rigour” (ibid.), would demand in all probability significant cash flow and ability on the part of researchers to leverage-in large research funds. The likelihood of this eventuality among academics just starting out and without the necessary reputational capital is, even in the most optimistic prognosis, slim.
The path to ‘returnability’ would appear to be made all the more perilous where institutional REF administration and administrators are considered by some to be respectively, inconsistentFootnote 14 and lubricious. UCU (2013a), for instance, has reported “a number of deficiencies in institutional procedures established to select which outputs and individual researchers [were] submitted to the REF”. UCU reported skepticism apropos the competency and neutrality of REF managers and reviewers; complaints of a lack of transparency; and intentional obfuscation in communicating, if at all, the justification for academics’ inclusion or exclusion within institutional REF2014 submissions; and the use of journal rankings in deciding whether to include outputs even though doing so contradicts assurances made by funding councils that such rankings would not be used as an evaluative criterion. More troubling, the UCU (ibid.) also reported on grievous malpractice in universities and threats made by some institutions of punitive action for academics that were not included in their REF2014 submissions. Threats reported (UCU 2013b) include academics having research privileges rescinded; contracts redrawn to ‘teaching-focused’; and being placed under ‘capability’ measuresFootnote 15. The implication of these threats is not only that the value of teaching is undermined but that academics’ career prospects are irreversibly damaged (cf. Fazackerley 2013). Furthermore, maligning university teachers as second-class citizens instead of treating them in a parity of esteem with researchers, is as Kelly (2013) argues, foolish not only on ideological but pragmatic grounds where the student marketplace is increasingly competitive and nevermore so important to universities’ solvency.
Evaluating Excellence: Excluding Research
At the same time that researchers are being excluded by the competition fetish, research agendas are increasingly streamlined, instrumentalized and adapted to fit the strategic priorities of research councils (Burawoy 2011; Docherty 2011). The organization of research by researchers – and top-down by research managers and administrators – is also now much more strategic and designed to follow the timeframes and requirements imposed by evaluation systems like the REF. For instance, academics are now inclined, or more realistically, compelled, to organize their research activity into the window of time in-between REFs, typically four-to-five years. This time-period represents the opportunity with which to conduct research; organize for its impact; and ensure its culmination with at least four ‘excellent’ outputs necessary for consideration within a university’s submission. In no shape or form is any account made for the perambulatory and serendipitous nature of research as a process of discovery.
Where academics appear increasingly to follow a ‘4 × 4’ route to research excellence – four ‘outstanding’Footnote 16 publications within a four-year period – the production of knowledge appears increasingly constrained by the process of its evaluation. Indeed, the terms of evaluation appear increasingly to dictate the ways with which academics conduct research (cf. Moriarty 2011). Some academic managers, for instance, now argue that in responding to the terms of research evaluation, when a book has equal weighting with a journal publication, academics should abandon the first and focus on the latter because the cost-benefit ratio is more favourable. We also have experience of institutional managers who argue that academics should only write-up research which is the recipient of external funding, as if funding is the only barometer of the merit and quality of research. In this context, this very article, much as a multitude of book chapters would be no longer possible. What also of ‘research’, which requires no funding and does not feature as a Research Councils, UK (RCUK) funded work-package, but is the consequence of curiosity and blue-skies thinking, and for research which is unlikely to attract funding, not because of any concern regarding quality or value, but because it does not conform to the strategic plans and prioritizations of research funders and their reviewers?
In the era of what Gornall and Salisbury (2012) term academic ‘hyperprofessionality’, the extent to which academics might return on a 4x4 model is compromised by the various and arguably excessive demands of mechanisms such as work allocation models and key performance indicators. Where such managerial devices are intended to increase the productivity, profitability and efficiency of academic staff, they are resisted and bemoaned for the extent to which they unnecessarily obfuscate and impede academics’ attempts to perform as teachers and researchers. They are also responsible for ‘burn-out’ and attrition, as academics leave the Academy no longer able to cope or deliver on institutional demands (cf. Warner 2013). Work allocation models and key performance indicators are another manifestation of the quantification of excellence, which appears, contrary and antagonistic to the terms of academic endeavour, when grounded and led by creativity. The new terms of competition in HE, informed by its marketization and a surrender to neoliberal axioms or what Michael Burawoy (2011) refers to as the imposition of “Soviet planning” are arguably, therefore, responsible for the denouement of academic creativity, reflexive knowledge, criticality and innovation. They are certainly entirely at odds with traditional liberal and pre-liberal models of academic professionalism and Merton (1942) norms of universalism, communalism, disinterestedness and skepticism.
The ascent of managerialist cultures in HE and the growing ubiquity of performance-metrics into every crevice of academic life is blamed for the erosion of creative flair and to borrow from Allan Bloom (1987), the closing of the intellectual mind and degeneration of the university as a knowledge incubator. Whilst competition, if proportionate or rather non-excessive, may inspire creative thinking, in the current HE landscape it features not as a catalyst of creativity but conformity. Competition in this context is not implemented for the purpose of challenging the limits of academic imaginations and adventures but in constraining these to a uniform expression of what counts. Any idea of academia as a portfolio of diverse undertakings would seem to be excluded tout court. In addition, as we have already established, demonstrating what counts and moreover what counts as excellence is an entirely arbitrary process that is prone to radical uncertainty and polarized opinion (cf. Lamont 2009). Yet the way with which knowledge is evaluated and excellence attributed is entirely dependent upon academics forcing knowledge into physical and measurable manifestations or in REF parlance, units-of-assessment, which academics attempt to satisfy through research outputs and impacts.
To our mind, a belief that research can be unproblematically converted into a product that can be measured and controlled in purely market terms is false. Market competition is necessarily zero-sum and produces an endless spiral of winners and losers. Of course, research is not a market product in this sense. It is a human process. While some sensible absolute criterion of accountability is clearly important in order to establish reasonable minimum standards, and ensure a fair days work for a fair days pay (as it were), it would seem hardly necessary to colonize the entire academic research agenda organized in five/six-year cycles via a bureaucratic revolution of nightmarish proportions for this modest goal to be achieved. One could be forgiven for thinking that the initial audit trail, initiated by Thatcher in 1986, drawing on a library of neoliberal thinkers, contained within its rationale, a deep distrust of academics and a pervasive anti-intellectualism. If evaluation of research should be focused on anything, it might be in determining how academics can improve a process that is constantly evolving, instead of providing a false sense of accomplishment.
The compartmentalization of research into a 4 × 4 framework, and similarly the modularization and slenderization of university education (cf. McGettigan 2013), suggests that knowledge may be contained and stored in neat boxes. It also intimates ownership and a sense that academic outputs are an individual or institution’s property, where it might be more helpful and profitable to think of ideas not as academic capital, but what Hannah Arendt (1958) calls, a means for “public action”. Surely then, as Cooper (2015) argues, this orientation provides a more meaningful basis from which to organize and achieve academic visibility and impact. Regrettably, however, the most profound contradiction of impact evaluation in the REF is that it actually curtails the possibility for ideas to translate into public action, where it restricts the kinds of conversations and collaborations between academics and the publics to those that might more easily be claimed to have impact.
For all the investment in UK HE in knowledge exchange and transfer (cf. Watermeyer 2011, 2016) there appears an explicit bias towards knowledge not for social action but for either commercial exploitation or what could be termed as functionalist societal accommodation. In other words, the REF as presently constituted, would appear to incorporate those exemplars that enable universities to include what they perceive to be their strongest, most marketable impact case studies. This, it can be argued cogently, is likely to be the outcome of a preparatory process internal to individual universities where the academic promotion by departments or individuals of impact case studies and other auditable content must necessarily compete for inclusion. The kinds of public or community engagement that generate ‘soft’ impact which are less well-represented in the REF are those at risk, when they are not only not endorsed but actively discouraged by senior academic and institutional managers (cf. Watermeyer 2015). The Bunyanesque shadow of hard impacts, those engineered through tightly defined and regulated stakeholder interactions, may likely relegate the merits of public interactions to obscurity. In a race to embrace the triple-helix (cf. Leydesdorff and Etzkowitz 1996), universities seem to be denying academics the opportunity with which to engage a broader public in a genuinely critical sense. Concurrently, notions of a ‘quadruple helix’, which might incorporate the public as an integral member of the knowledge society, seem, at least for the immediate future, fantastical.