Hyping the REF: promotional elements in impact submissions

Abstract

The evaluation of research to allocate government funding to universities is now common across the globe. The Research Excellence Framework, introduced in the UK in 2014, marked a major change by extending assessment beyond the ‘quality’ of published research to include its real-world ‘impact’. Impact submissions were a key determinant of the £4 billion allocated to universities following the exercise. The case studies supporting claims for impact are therefore a high stakes genre, with writers keen to make the most persuasive argument for their work. In this paper we examine 800 of these ‘impact case studies’ from disciplines across the academic spectrum to explore the rhetorical presentation of impact. We do this by analysing authors’ use of hyperbolic and promotional language to embroider their presentations, discovering substantial hyping with a strong preference for boosting the novelty and certainty of the claims made. Chemistry and physics, the most abstract and theoretical disciplines of our selection, contained the most hyping items with fewer as we move along the hard/pure – soft/applied continuum as the real-world value of work becomes more apparent. We also show that hyping varies with the type of impact, with items targeting technological, economic and cultural areas the most prolific.

Introduction

The evaluation of academic research to decide government funding to universities is now common throughout the world and dates back to 1986 in the UK. Several iterations of the Research Assessment Exercise have occurred every 5 years since then with various modifications, but 2014 marked a major shift in policy. Renamed the Research Excellence Framework, the new structure extended the assessment of scientific ‘quality’ beyond published research to include its real-world ‘impact’ (Watermeyer, 2019). For the first time, universities were required to show that their work had

‘... an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’¹.

This, then, is the apotheosis of a movement towards the accountability of publicly funded research. More intrusive, subjective and complex than judging the contribution of published outputs, the ‘impact agenda’ seeks to ensure that funded research offers taxpayers social benefits.

These impact submissions are a key determinant of the almost £4 billion allocated by the UK Government to universities through funding councils. The case studies supporting claims for impact therefore comprise a high stakes genre, and writers are keen to make the most persuasive argument they can for their work. This involves rhetorical decisions which best position their work to appeal to a panel of assessors who may not be specialists in the topic in order to demonstrate its broad economic and societal impact (Khazragui & Hudson, 2015). Moreover, as Silvertsen and Meijer (2020) have recently argued, assessment decisions tend to reward ‘extraordinary’ impact while downplaying the importance of ‘normal’ everyday impacts that organisations need to make use of useful new knowledge. Like Silversten and Meijer, we see this as incentivising submissions which claim ground-breaking results with unexpected wide-reaching societal impact. It is not surprising, then, that impact statements might see some embellishment, and we focus here on the most obvious ways this is achieved by analysing Hyland and Jiang’s (2021a) use of hyperbolic and promotional language to glamorise, publicise, embroider and/or exaggerate aspects of their research—a phenomenon Millar et al. (2019) refer to as ‘hype’.

In this paper we examine hyping phenomena in 800 of these ‘impact case studies’ and explore how academics promote the economic and social relevance of research from disciplines across the academic spectrum. In doing so we address a gap in the literature on how impact criteria have been interpreted by academics and influenced their self-report, narrative-based submissions. The results indicate how the current methodology for evaluating research impact through individual, evidence-based case studies, not only promotes the selection of particularly impressive examples, but encourages the use of hype in presenting them. The study therefore adds to our current understanding of disciplinary persuasion in the high-stake context of performance-based assessment and raises questions about the usefulness and reliability of impact narratives as a basis for university funding.

Impact and the REF

Over the last 40 years the methods used to assess research have become more sophisticated and, inevitably, more burdensome (e.g. Penfield et al., 2014). In the UK, the process began during the 1980s, when Margaret Thatcher’s government pronounced that all areas of public expenditure should demonstrate ‘value for money’. The RAE was introduced as a way of trying to hold funded institutions to account for the public money they receive, and the UK was one of the first countries both to institutionalise university research assessment and link it to financial allocations. With stringent government funding cuts imposed on universities, the University Grants Council, after several attempts, created a general mechanism to determine the allocation of this reduced funding (Martin & Whitley, 2010). Treated with suspicion and hostility by universities from the start (e.g. Chiang, 2019), a structure evolved, largely independent of government intervention, to rank and reward universities on the basis of the quality of their research outputs.

Over time, the assessments have become more rigorous, costly and time consuming (Pedersen et al., 2019; Chiang, 2019) with the pressures to do well demanding greater levels of effort by submitting departments and by assessment panels. With the introduction of the REF in 2014, the peer review element of the old system is combined with assessment of the impact of research. While much of the rest of the exercise remained unchanged, the added criterion of ‘impact’ meant that what Watermeyer (2019) calls ‘the noose of competitive accountability’, grew tighter.

As part of the 2014 exercise, each institution was not only required to submit details of publications produced by selected staff between January 2008 and December 2013, but also to provide claims for its impact. This took the form of asking institutions to complete a template describing their approach to facilitating the impact of its research on wider society and case studies describing specific examples of this. In all, UK universities submitted 6975 impact case studies. Each participating university had to submit a minimum of two cases with the actual figure depending on the number of staff in a subject area or ‘unit of assessment’ (UOA). Larger universities, such as Cambridge, UCL and Oxford, submitted over 250 cases each, while smaller institutions, like the University of the Creative Arts and Royal Agricultural University, entered just two. Submissions were made in 36 subject areas.

The cases comprised a four-page document with the following recommended format:

1Summary of the impact100 words

2Underpinning research500 words

3References to the researchSix references

4Details of the impact750 words

5Sources to corroborate the impact10 references

In some case studies the sections vary considerably from the indicative word length, but all were restricted to four pages. The submissions were assessed by an expert sub-panel for each UOA, working under four main panels.

Needless to say, the explicit evaluation of ‘impact’ has not been uncontroversial. Impact itself, while obviously important, is a hard term to pin down as research communities, and perhaps individual researchers, have their own interpretations so that engineers will view impact differently to art historians, for example. While assessment panels have some flexibility to draw on disciplinary understandings, they need to follow broad criteria of ‘reach and significance’ specified by the government funding agency (HEFCE, 2011: 5). There is also the problem that impact is scaler and has different magnitudes so that some impacts may be huge while most are likely to be modest. There is also the difficulty that some impacts may actually be negative, as in later re-evaluations of some vaccines for example. Overall, while researchers might acknowledge they have a responsibility to contribute to society in some way, the evaluation of impact can only be partial, indirect and long term (e.g. Chubb & Reed, 2018; Pedersen et al., 2019).

REF impact assessment has also drawn criticism from observers in the social studies of science regarding three broad areas: expert review, the construction of impact case studies and the stipulation that impact ‘builds on’ research (Smith et al., 2011).

First, some academics believe that the establishment of expert panels to judge impact risks threatening the integrity of disciplines and the autonomy of academics by including funders, commercial representatives and other end-users. They are concerned that this variety of assessors may have different ways of defining, measuring and weighing impact and that this puts greater pressure on academics to engage in rapidly produced and immediately relevant outcomes (e.g. UCU, 2013). Second, the creation of impact cases also has the potential to remove the control which academics have over their research agendas and contribute to a rigidity in how outputs are formatted (Heinze et al., 2009). Finally, the assumption that impact ‘builds on research’ and is traceable back to a distinct research unit in a single institution is also questionable. Often impacts from research do not stem directly from excellent research and can involve numerous centres and multiple strands of research conducted over a long period.

This last point is crucial for those working in the social sciences and humanities where the creation of knowledge follows less linear paths and has less obviously tangible impacts. Impact in these fields is far more likely to be one of gradual ‘knowledge creep’ rather than a simple, direct and instantaneous causative effect (Martin, 2011; Watermeyer, 2019).

The difficulties of defining and ‘capturing’ impact in a rigorous and consistent way across all fields of inquiry and over 100 universities thus pose serious problems for those who write, support and evaluate the submitted cases. This is an extremely competitive and high stakes genre which also carries a great deal of uncertainty for writers. As a consequence, institutions invest considerable time and resources in producing the case studies to ensure that impact is appropriately highlighted and made visible. In this paper we investigate whether writers press their arguments for impact in rhetorical ways, going beyond the real-world value of the research using discoursal embellishment and the use of hype.

Hype and academic persuasion

Academics have always presented their research in a favourable light, carving a recognisable and valued niche for themselves so their work has more chance of being published, cited and used. With career pressures on academics to publish and more than 3 million new peer-reviewed articles appearing each year (Johnson, et al., 2018), there is even more encouragement to rhetorically promote results and professional visibility itself.

Thus Martin and Leon Perez (2014) found writers in both health sciences and political sciences often underscore the contribution of their research in article introductions while writers have been observed to explicitly highlight the novel contribution of their work in literary studies (Lindeberg, 2004) and applied linguistics (Wang & Yang, 2015). Some scientists believe that such hyping practices have now reached a level where objectivity has been replaced by sensationalism and manufactured excitement (e.g. Rinaldi, 2012; Scott & Jones, 2017). Fraser and Martin (2009), for example, found a significant increase in 21 ‘biassed’ adjectives’ such as important, critical and original in clinical research journals between 1985 and 2005. Similarly, hyperbole has risen in medical journals with the frequency of 25 positive-sounding words such as novel, amazing, innovative and unprecedented increased almost nine-fold in the titles and abstracts of papers published in PubMed between 1974 and 2014 (Vinkers, Tijdink & Otte, 2015).

By exaggerating the importance of findings, writers are seen to undermine the impartiality of science, fuel scepticism and alienate readers (e.g. Horgan, 2015). The editor of Cell Biology International, for example, bemoaned an increase in ‘drama words’ such as drastic decrease, new and exciting evidence and remarkable effect which he believed had turned science into a ‘theatrical business’ (Wheatley, 2014: 14). However, many of these studies are largely arbitrary and ad hoc, restricted to a few words that the reporter finds most offensive.

Recent studies by applied linguists, however, have employed greater rigour. Millar, Salager-Meyer and Budgell (2019), for example, manually annotated a small corpus of Randomised Control Trials (RTCs) for hyping items. RTCs are employed to measure the effectiveness of new treatments in medical research, yet the authors found 6.7 occurrences of hype words per-paper, or 2.0 per 1000 words. In a follow-up interview with seven authors, Millar, Budgell and Salager-Meyer (2020) found that motives for hyping were reinforced by pressures to publish.

More recently, Hyland and Jiang (2021a) examined 400 candidate hype terms in the 200 most highly cited scientific papers dealing with the Covid19 virus and found 35.9 items per 10,000 words. This was not only significantly more cases than in a reference corpus from the same fields, but the results showed a significant increase in hyping over each month of the study (January to July 2020). The authors argue that the intense, high-stakes competition to understand the virus and discover a vaccine encouraged researchers to promote their results. In another study Hyland and Jiang (2021b) explored hype in research articles in a wider range of disciplines, a larger corpus of texts and over a longer period of time. They found that these promotional features have increased by 19% over the last 50 years when adjusted for length of papers—indicating an authorial repositioning in response to the changing circumstances of institutional evaluation and metrics-driven career trajectories.

It would, we surmise, be surprising if the new genre of impact case studies were unaffected by this movement, and we test this assumption here. Indeed, the intensification of this audit-based culture, and the rewards which accrue to those who are successful, are likely to encourage authors to boost the potential payoffs of their work. Using the same inventory of 400 hyping terms as Hyland and Jiang, we seek to determine if academics seek to hype the value of their REF submissions and how they do so.

Data and analysis

To explore hyping practices in this genre we built two corpora of the impact case reports submitted to the 2014 REF². Using the online database of case studies provided on the REF website, we were able to select the disciplinary submissions (Corpus 1) and types of impact each report aimed to address (corpus 2). We are therefore able to provide a broadly representative picture of hyping practice across a wide spectrum of academic endeavour and in key areas.

There are, of course, ongoing disagreements concerning how best to categorize disciplines and even questioning the relevance of the term itself. With increasingly blurred boundaries between them (e.g. Manathunga & Brew, 2012) and with the growth of interdisciplinary research work (Land, 2012), there are fewer clearer distinctions. However, we see important epistemological value in employing the traditional categories of hard vs soft sciences (Becher & Trowler, 2001), the former referring to fields which predominantly employ objective measurements of controlled variables, and the latter which study less tangible subjects. The distinction continues to have significant applicability as a window on changing sensibilities about what science is and recent changes in governance and production (Shapin, 2020).

We also see value in the distinction between pure (or basic) and applied fields Biglan (1973). This polar conception has also recently received some support (Shaw, 2022), and we have adopted it here. These social and epistemological structures seem to influence discursive practices in quite tangible ways (e.g. Hyland, 2005). Indeed, the distinctions continue to inform research in linguistics (Zou & Hyland, 2020), education (Söderlind & Geschwind, 2020), and technology (Fathema & Akanda, 2020). We selected cases to represent a range of disciplinary practice and exemplify both hard and soft sciences and pure and applied categories. Our cases are from chemistry, physics, computer science, allied health professions, history, English language and literature, education and social work. It is worth pointing out, however, that different types of institutions submit impact cases in different units of assessment, so that not all universities enter in all categories.

For the disciplinary corpus, we downloaded 100 case reports from each selected ‘Unit of Assessment’ and removed the two sections of references together with Section 2, which outlines the research that informs the impact and not the claims for the impact of the case itself. It includes details of the research, such as when it was conducted, by whom, and with what results, rather than its real-world outcomes. So, we included the two sections detailing the impact of the case itself: the Section 1 summary and Section 3 which is a ‘narrative, with supporting evidence, to explain how the research …made a distinct and material contribution to the impact and the nature and extent of the impact’³. This gave us a corpus of 800 texts of 764,687 words in total as shown in Table 1.

Table 1 Characteristics of the discipline corpus

For the corpus of impact areas, the 2014 REF provides a categorisation of each case study in terms of cultural, economic, environmental, health, legal, political, societal and technological types of impact. The type is selected by the submitting authors in the ‘summary of the impact’ section of the submission. To explore the hyping of impact types, we created a new corpus of 800 texts filtered by the type of impact indicated in each file. This allowed us to determine whether certain types of impact were hyped more than others. Table 2 profiles this corpus, ordered by total words.

Table 2 Characteristics of the Impact types corpus

To explore these corpora, we developed a list of hyperbolic items from a potentially open set. We followed Millar et al.’s (2019, 2020) conception of ‘hype’ to include instances of language promoting any aspect of research, while recognising a cline between modest and exaggerated promotion. We initially searched the literature on evaluative language referred to above and included hyperbolic terms (Millar et al., 2019), ‘drama’ words (Wheatley, 2014), value-laden vocabulary (Fraser & Martin, 2009), positive words (Vinkers, Tijdink & Otte, 2015) and ‘superlatives’ (McCarthy, 2015). These items were supplemented with boosters and positively marked attitude markers from Hyland’s (2005) stance framework. ‘Boosters’ express an author’s strong conviction and epistemic commitment to assert claims and shut down alternative voices (demonstrate, show, clearly). ‘Positive attitude markers’ indicate the writer’s affective perspectives and include personal evaluations (interestingly, fascinating). We also scanned sources such as the Oxford Thesaurus of English (Waite, 2009) and corpus-generated wordlists such as the Academic Vocabulary List (Gardner & Davies, 2013). This produced a list of about 400 hype items⁴.

We searched our corpora for these items using AntConc (Anthony, 2019) and then manually examined and counted each concordance to establish that the feature was performing a hyping function. For example, the word major was excluded in contexts such as ‘candida is a major fungus which resides in fifty percent of human oral cavities’ but seen as hyping when used to modify a claim such as ‘our magic bullet is a major achievement in this field’. Similarly, words such as important and definitive were ignored in the negative (not important/ no definitive conclusion) (see Fraser & Martin, 2009). Both authors worked independently and achieved a high inter-rater agreement of 97% before resolving disagreements. Then the results were normalised to 100 words to allow comparison across the corpora and determine statistical significances. The log-likelihood (LL) test was run by Rayson’s log-likelihood calculator⁵, and effect size (%DIFF) was also considered (Gabrielatos, 2018). We now report our findings.

 How much hyping is there in impact submissions?

The short answer to this question is quite a lot. In fact hyping seems to be significantly more common in these impact cases than in research articles. In the disciplinary corpus of impact cases we identified 16,165 occurrences of hyping expressions, or 2.11 per 100 words, whereas Hyland and Jiang (2021b), found 1.55 terms per 100 words in their corpus of articles (LL = 751.29, %DIFF = 26.68. p<0.0001) using the same inventory of items. The scale of hyping here thus acknowledges the highly competitive nature of the submissions and the serious consequences which hang on success. The numbers are, then, underpinned by the sense that writers, and the faculty teams that often support them in preparing these documents, must promote the extent, scale and significance of the impact.

Another striking difference between these two genres is that the writers of REF case reports, unlike article authors, greatly favoured positive attitude markers to hype their work rather than boosters (1.39 cases vs 0.72 per 100 words). These forms embroider a claim by expressing the writer’s strong affective judgement (all examples are from the corpus):

• (1) The success of the prototype depended on the superb electronics and mechanical engineering support in the Warwick Dept. of Physics. (Physics)

• (2) This is an extremely encouraging theoretical result and Siahaan has also improved the method… (Technology)

These positive personal evaluations not only carry heavily promotional claims, but also insinuate confirmation—thus both expressing a position and seeking to bring readers into agreement with it.

Boosters, on the other hand, work differently as hypes. They express epistemic conviction and invest statements with the confidence of factual reliability. They are persuasive because they present claims categorically as truth:

• (3) The new ion beam technology has been incorporated into a completely new type of Secondary Ion Mass Spectrometer that fully exploits the capability of the C60+ ion beam system. (Chemistry)

• (4) All the above clearly shows the global impact of the research undertaken given its widespread adoption. (Chemistry)

The most 10 frequent hyping items used in REF reports are new (0.41 cases per 100 words), first (0.20 cases), significant (0.14 cases), important (0.10 cases), very (0.10 cases), leading (0.09 cases), effective (0.06 cases), novel (0.06 cases), highly (0.05 cases) and creative (0.05 cases).

Which disciplines hype the most?

A longer answer to the question of ‘how much hyping is there?’ is less straightforward because the bottom line results discussed above are cross-cut by differences of discipline and impact types. Table 3 shows the distribution of hypes by discipline ordered by items normed to 100 words.

Table 3 Disciplinary use of hyping expressions in REF case reports

As we can see, submitting authors in chemistry and physics take significantly greater pains to promote the value of their work than the other fields. These are perhaps the two most abstract and rarefied fields of our selected disciplines, where research is more likely to be several steps removed from informing real-world applications. The spectroscopic observation of molecules in interstellar clouds, for example, or the use of light-scattering for analysis of macromolecular associations, is likely to require a greater investment in persuasion than disciplines closer to the immediate needs of everyday life.

(5) This marked a critically important step towards continuous strand sequencing. (Chemistry)

(6) To account for collisional quenching, they implemented a new technique using a laser-induced grating that provided unprecedented precision of temperature and air/fuel ratio measurements. (Physics)

Similarly, we find submissions from the ‘softest’ applied fields, Education and Social work and social policy, containing the fewest hyping items. Cases dealing with support for non-traditional learners in education and improving practice in treating substance abuse would appear to have more immediate and direct real-world outcomes.

(7) This addresses a significant area of concern raised at Government level…. (Education)

(8) The research findings uncovered an important area for practice development. (Social Work)

This picture gains further support when we consider the four groupings on the pure/applied and hard/soft axes. Table 4 underlines the considerable hyping found in the hard-pure impact submissions relative to the other groupings and particularly the soft-applied submissions (LL=231.64, p<0.0001).

Table 4 Variation between broad disciplinary categories of hyping items

Clearly, making a case for the significant impact of these often highly theoretical and abstract studies in pure science fields can involve considerably more rhetorical investment. We can also see differences in the ways that fields hype their submissions. Table 5 shows the most frequent hypes by discipline, with new, leading and novel dominating the science case studies and first, significant and important at the head of the soft discipline lists.

Table 5 The most frequent hyping markers by discipline (per 100 words)

Expressing a strong evaluation in an impact case involves both a statement of personal judgement and an appeal to shared values. Hype is therefore interpersonal; it requires writers to draw on their knowledge of what might be prized by the assessors and how they might best appeal to this. Based on the items in Table 5 and following Hyland and Jiang (2021a), we identified four broad categories of hyping which writers used to promote their work:

• i. Certainty (concerns the strength or importance of the statement—significant, important, strong, crucial, clear)

• ii. Contribution (refers to the immediate value or use of the work—necessary, essential, effective, useful, influential)

• iii. Novelty (stresses the originality of the claim—first, timely, novel, unique)

• iv. Potential (comments on the possible future value of something—promising, potential, apparent)

Overall, certainty items dominate the most frequent items, comprising almost half the forms and found across all the fields studied. These serve to boost the significance or consequence of the claims with a commitment that almost compels assent:

(9) SIMS is a very powerful tool for accurately analysing the chemical make-up of a solid material in order to build up a three-dimensional chemical map. (Chemistry)

(10) Our Progeria research has impact on the study and treatment of normal ageing since Progeria arguably is the best model for normal ageing. (Applied Health)

(11) The research has had a significant impact on testing bodies, organisations and test takers internationally. (English)

The second most frequent type of hype more specifically addresses the contribution made by the research reported in the submission, referring to its value, outcomes or take-up in the real world. This category comprised 30% of items and, interestingly, was most predominant in the soft fields of English, Education and Social Work:

(12) Since 2008 Bournemouth University's (BU) research-based continuing professional development (CPD) programmes have improved practice for 6170 health and social care professionals (Social Work)

(13) Our research focuses on the education of healthcare professionals and nurses to enhance the safe care of patients. (Education)

(14) The work has improved the tests themselves, but also allowed Cambridge Assessment to better communicate the qualities of their tests for accreditation and recognition.(English)

For this group of disciplines, making explicit claims about the value of the work they have submitted is a key part of seeking recognition for it. While we might assume that work produced by the social sciences often has a more obviously practical utility, especially in the applied fields, this nevertheless has to be made clear.

The hard sciences, on the other hand, employ novelty as a key aspect of their persuasive armoury. References to the originality and inventiveness of the work appeared four times in each of these disciplines and comprised 2/3rds of all mentions in the top 15 items.

(15) The group showed for the first time that an alpha-haemolysin pore could be engineered to contain more than one recognition site and thus identify groups of different nucleobases simultaneously in an immobilised DNA strand. This marked a critically important step towards continuous strand sequencing. (Chemistry)

(16) The novel bit-accurate precision of the new method enables, for the first time, proof of those software artefacts that could not be shown to be safe with existing technology. (Computer Science)

(17) …. A counting breakthrough conceived by the Medipix collaboration and is unique in its adaptability, high spatial resolution, high dynamic range and low noise. (Physics)

Clearly, being first, making breakthroughs and offering original solutions are the stock in trade of these fields and it comes to the fore in their impact cases.

Finally, while numbering only a handful of occurrences in the top 15, it is surprising to find references to the potential of submitted work rather than actual, proven benefits. The hard fields, however, occasionally brought the future into play as an additional benefit to those already demonstrated, promising even greater riches to come:.

(18) The University has placed 20 evaluation licences, and there are discussions for the potential collaborative projects across a range of market sectors. (Physics)

(19) Additionally, the results revealed that this sensor could be used as the basis for a novel imaging microscope system. (Computer Science)

(20) This enabled a broad range of projects to be undertaken spanning a wide range of possible future application areas which clearly demonstrate the disruptive potential of the technology. (Physics)

To summarise this section, there is a clear difference in both the frequency and, to a lesser extent, type of hypes used by the disciplines in our sample. Those working in the hard sciences tend to promote their work more than those in the softer sciences and pure research requires greater advocacy than applied. All disciplines boost their submissions with certainty hypes, while hard fields place greater stress on novelty and originality and softer fields on the contribution made. We now turn to look at hyping within ‘impact types’.

Hyping and impact types

We then looked at our corpus compiled using the metadata added to the REF database to associate case studies with the ‘type of impact’. While the compilers of the website admit that most case studies relate to more than one type of impact, the categorisation is a useful indicative guide to the content of submissions as seen by their authors. As shown in Table 2 above, there are eight Impact Types which broadly follow those widely used by Government policymakers. Table 6 shows the distribution of hypes across the impact types, ordered one again by normed frequency.

Table 6 Hyping expressions across impact types (800 papers per type)

The table indicates that hyping predominated in cases marked as having technological, economic and cultural impacts. There seems to be nothing which immediately links these types, although the first two are often related to submissions with connections to the business world:

(21) The project helped the company become more competitive through a recession period, leading them to the forefront of their business with greater knowledge support for clients and, increased market opportunities through provision of new analytical services. (Economic)

(22) The technology has thus not only secured financial benefits for X-Tec/Nikon but has significantly improved vital inspection processes available to aircraft engineering industries. (Technology)

It should be pointed out that the assessment panels for the REF impact studies were larger and more diverse than those of previous exercises, expanded from ‘peer’ to ‘expert’ assessors to include more end-users of research. The aim was ‘to involve a majority of research users (broadly defined) in the assessment of impact’ (HEFCE, 2009, para 71). The diversity of these panels may have led writers to assume that additional rhetorical assistance was needed to help readers recognise the value of the submission.

The hyping of projects deemed to have cultural benefits is perhaps easier to understand as this is an area where writers are not only appealing to a broad audience, but also about matters which may seem relatively insubstantial to outsiders. The establishment of digital cultural heritage communities and travelling art exhibitions may well require greater effort to sell to readers than areas such as health and environment. These two extracts from submissions designated as having ‘cultural’ impact, for example, illustrate how writers attempt to make the effect of their work on major areas of cultural importance as explicit as possible:

(23) The significance of DMVI lies in its unique capacity to furnish new audiences with cultural and social images from the past, without losing the information about their provenance and original connotation. (Cultural)

(24) The cultural significance of Cowie's impact is visible in her creative dialogues with a number of international filmmakers and multi-media artists. (Cultural)

Surprisingly for us, hyping in legal-oriented impacts is at the foot of the list, as one might expect the value of this issue to need the most promoting. We do find a substantial amount of hype in these texts, as here:

(25) Wells’ potent combination of detailed critique of the corporate offence in the draft Bill, together with recognition that the UK not seen as compliant by the OECD, prompted major substantive amendments to the Bill by means of a direct, linear change in the Bribery Act 2010. (Legal)

But more usually, there appears to be a preference for greater restraint among those submitting work in this area. More common is the measured recitation of the researcher’s relevant professional accomplishments (26) or the calm reporting of a major reform of the law as an impact (27):

(26) Burton’s impact on policy making in domestic violence cases has been partly as a result of direct involvement in advisory bodies considering her own and others’ research in this area. From 2001 to 2006 ….Burton was a member of the Domestic Violence Advisory Group (DVAG convened by the LCD/MOJ). She was invited to be a member of this group as a result of her expertise on the legal responses to domestic violence and was the only academic member. (Legal)

(27) As a direct result of the research the law was amended so that illegitimate children were given the same fixed rights of inheritance as legitimate ones, and the fixed rights of inheritance of a widow and widower are now unified. (Legal)

As with the disciplinary corpus, we also looked at the preferred hyping terms in each impact type. Table 7 shows the most frequently used items per 100 words in each type.

Table 7 Most common items used to hype impact (per 100 words)

Once again, new is among the most popular forms, sitting at the top of four lists, with first and novel in the top four of both health and technology types. Clearly these two areas place considerable value on novelty and the value of being the first to introduce innovative approaches to the market:

(28) The novel discoveries and testing methods developed in Cardiff made key advances that drove and enabled new biocompatible solution development. (Health)

(29) Our research on Active Shape Models (ASMs) and Active Appearance Models (AAMs) opened up a radically new approach to automated image interpretation, with applications in industrial inspection, medical image analysis and face tracking/recognition.(Technology)

(30) The novel bit-accurate precision of the new method enables, for the first time, proof of those software artefacts that could not be shown to be safe with existing technology (Technology)

Items appealing to impact novelty, however, seem to play a less central role in our impact-type texts than the disciplinary-focused corpus. Novelty comprises only 14% of the total compared with 20% in the disciplinary lists. In addition, novelty items do not appear in legal, political or societal texts of our selection at all.

Hypes which promote the contribution made by a project in these texts are three times more frequent than novelty and are particularly common in the legal, societal, technological, economic and environmental submissions. Here we find items which assert the direct and current benefits of the research to the particular area:

(31) .. to promote a global level greater focus on the strengthening of those FATF standards which relate to law enforcement, effective criminalisation of money laundering and effective confiscation. (Legal)

(32) Schmidt made a substantial contribution as to the thorny issues of liability, ethics and consent and his evidence, advice and recommendations were seminal. (Political)

(33) Research conducted in the School of Chemistry at the University of Bristol between 1992 and 2013 has played a leading role in global efforts to achieve reliable, long-term measurement of climatically important gases such as CO2, CH4 and N2O. (Environmental)

The most frequent category of hype in this impact-type corpus, comprising almost 50% of the total, is that of certainty, where writers express a clear assurance of the importance or benefit provided by their work. Such declarations of conviction comprise half of the most frequent items in this corpus, most obviously in the political impact texts, but also in the societal ones too:

(34) The research has had an extensive impact on the development of methodologies in HRIAs and EHRIAs and on policy debate at international, national and local levels, across a range of areas. (Political)

(35) Kerbcraft now plays a key role in the UK Government’s road safety strategy and has been cited as an example of best practice by the World Health Organisation and safety agencies across Europe, the USA, Australia. (Societal)

The frequency of these boosters in the corpus reveals something of the keenness felt by writers to ensure that their submission gets the recognition they feel it deserves.

Conclusions and implications

The requirement of institutions to submit impact case studies as a means of judging the social value of research was a controversial and much debated move in the UK. While having the benefit of encouraging academics to consider the impact of their research from the outset, the composition of heterogeneous panels and the view that impact is traceable back to a distinct research unit in a single institution remains contentious (Watermeyer, 2019). There has also been disquiet about the expense, with the pressures to prepare impact submissions costing universities around £55 million, although the biggest burden appears to have been the need to ‘evidence’ impact and develop an understanding of the concept (Manville et al., 2015). We should note here that in a simulated assessment exercise of narrative-based cases by 90 senior peer reviewers, Watermeyer and Hedgecoe (2016) observed that case studies which best sold impact were those rewarded with the highest evaluative scores. It is estimated that the most significant and far-reaching impacts gained something like £308,000 (Reed & Kerridge, 2017) so, with a lot riding on the outcome, it is little wonder that authors employed rhetorical means to promote the value of their projects.

Our research found substantial hyping in these case studies—with significantly more instances than in research articles. Chemistry and physics, the most abstract and theoretical disciplines of our selection, contained the most hyping items with a strong preference for boosting the novelty and certainty of the claims made. There is less rhetorical investment as we move along the hard/pure – soft/applied continuum with researchers in education and social work perhaps feeling that the real-world value of their work is more obviously apparent. We also show that hyping varies with the type of impact in a submission, with items targeting technological, economic and cultural areas the most prolific. We assume that writers feel that the value of these types will not be immediately apparent to assessors and so require some additional finessing to persuade them.

We recognise, of course, that there are limitations to our study. We have selected a restricted number of disciplines and focused on the textual submissions only. Further research might explore how other fields use these hyping items or take a different perspective by interviewing writers about their intentions and readers about the effect of these devices on their judgements. Additionally, comparisons might be made with submissions in later iterations of the REF—when texts become available. Another complicating factor is that, as we noted above, not all institutions submit in all units of assessment. We might speculate that more wealthy, globally focused institutions are more likely to be engaged in hard/pure work than primarily vocational and locally oriented ones (Manville et al., 2015) and may present submissions with the authority of substantial publishing and funding success behind them. This also raises the issue of who actually writes the submission. While we have assumed that academics are responsible for submitted texts, the larger, better resourced, universities may have teams, or whole units, of people devoted to polishing and embellishing cases for submission. Thus, hyping may not only be related to disciplinary practices but to the relative prestige and power of the submitting university, and further research might investigate the role that university size and wealth plays in the writing of impact cases.

Ultimately, however, our work offers empirical evidence, in the form of academic rhetorical practices, which supports what may be a decline in engagement with values of integrity and objectivity. Interviews, for example, show how REF assessments have had a negative effect on research culture as extrinsic motivations for impact ‘crowd out’ intrinsic motivations of academics, altering perceptions of self-determination and creating a barrier to engagement (Chubb & Reed, 2018). At the same time, the value placed by assessors on what Silversten and Meijer (2020) call ‘extraordinary’ impact while downplaying the significance of ‘normal’ everyday impacts which generally lead to useful new knowledge encourages the kind of exaggeration we have discussed here as hype. While we cannot comment on the effect of hyping on readers’ assessments of submissions, our study, together with these contributions, indicates the risks of narrative self-reports in undermining the impartial assessment of research impact.

Importantly, then, our study reinforces research pointing to potential shortcomings in the assessment of impact through the use of case studies. For some observers, the fact that impact is being valued highly in the REF is a positive step away from the Ivory Tower perception of research-for-research-sake. However, these financial incentives risk creating activities which game the system and submissions which exaggerate the value of projects, both of which have the potential to bring the academy into disrepute. Hype is part of researchers’ desire to claim the greatest significance for their work, to meet institutional demands for precedence and influence and to achieve individual professional goals. In other words, while these financial incentives may motivate academics to consider impact, they may also encourage them to chase it for career reasons. As currently implemented, then, the use of narrative cases to evaluate research impact tends to promote rhetorical practices which have the opposite effect of those intended.