REF impact and the discipline of politics and international studies

On 22–23 November 2016, the Department of Politics and International Studies (PAIS) at the University of Warwick hosted a lively conference on research ‘impact’, a key component of the Research Excellence Framework (REF)—the periodic peer review in UK Higher Education to evaluate the quality of academic work, in turn determining how nearly £2 billion of annual research funding is allocated among competing institutions. In contrast to many other conferences and workshops about impact, a feature of REF that has generated much fevered commentary across the university sector about the public value of academic research, our event was tightly focused on impact as it relates to the discipline of Politics and International Studies. Featuring presentations from a range of academic and non-academic constituencies—including journalists, government officials, and representatives from the Higher Education Funding Council for England (HEFCE),¹ the creator and (at the time of writing) controller of the REF—conversation was particularly timely, owing to the publication, a few months earlier, of the eagerly awaited Stern Review on the future of university research assessment. This special issue contains selected papers from the conference, with the goal of shedding light on how the discipline as a whole, but also its varied composite parts, is responding to the impact agenda in the lead up to REF2021, now looming close on the horizon. We are grateful to the contributors for their participation and for opening up many new facets to the ongoing dialogue about impact.

By way of an introduction, we would like to sketch out some of the most pertinent issues and challenges faced by political scientists amidst the new landscape of having to demonstrate the wider relevance of their research beyond academia. Several arguments will be advanced. One: Politics and International Studies are not inherently opposed to the basic premise of impact; indeed, we might even go as far as to say that an instinct for impact is in the disciplinary DNA. Created by leading figures in Progressive era thought, the discipline has long been moved by a utilitarian desire to produce knowledge that can enlighten and inform society for the greater public benefit. Notwithstanding a period in the second half of the twentieth century when disciplinary professionalisation led to a degree of detachment behind the walls of the Ivory Tower, political scientists do not readily conform to the popular stereotype of the ‘begowned dilettante cossetted from the concerns of the real world’ (Lawson and Sayers, 2016, p. 17). They are, and always have been, what Colin Hay describes as ‘public goods providers’ (Hay 2015, p. 51).

Two: owing to its disciplinary heritage, the primary debate for political science has not been whether impact should be done—a question that has delighted and divided many of its scholarly brothers and sisters—but whether impact, as presently defined, helps or hinders the discipline in its efforts to provide practical knowledge for the public good. Here, political scientists are concerned that HEFCE has set the bar too high. As conceived, excellence in impact requires tangible evidence of research leading to direct change, with a high premium put on influence in the policy realm. Believing that this is an unrealistic target, political scientists have called for more emphasis to be placed on tangible knowledge transfer.

Three: the discipline is blazing a trail in thinking about what Matthew Flinders has called the ‘Politics of Impact’. Increasingly, as they walk the corridors of power and attempt to juggle the world of public affairs, political scientists are realising that the impact agenda carries with it some delicate ethical issues and thorny professional dilemmas. HEFCE understands impact as research that has effected, changed, or benefitted society, but avoids specifying what qualifies as a positive or benign transformation. Notions of the ‘public good’ or the ‘public interest’ are, of course, contested and slippery—like trying to capture air. But, they are also political in a way that one suspects HEFCE did not foresee. Take, for example, a body of academic research that paves the way for better universal healthcare. On the majority of University campuses this would be held as an unqualified societal benefit. Elsewhere, however, say, in conservative circles that desire lower taxes or a greater proportion of public funds to be spent on defence and national security, this would not necessarily be welcome. What this example underscores, albeit somewhat crudely, is that society is not homogenous in the way that the current model of impact presupposes: rather, it is made up of subgroups, each with different histories, identities, cultures, and expectations of what should constitute the public good. Ethically, therefore, political scientists are entitled to ask whether all ‘impacts’ should be valued the same and whether the pursuit of certain stakeholder relationships, while expedient for the purposes of REF impact, constitute a Faustian bargain, not necessarily advancing the type of society that they themselves, as citizens, wish to see established.

The ‘special function’ of political science

Looking holistically at the political science community in the UK, we would suggest that there has not been the same level of ideological hostility to the notion of impact exhibited by other quarters of the academic profession. Typically, that opposition stems from the view that impact represents further evidence of the corrosion of academic freedom and agency by neoliberal forces determined to impose market-like discipline on universities and see them become Taylorist factories of knowledge mass production (Watermeyer 2016; Chubb et al. 2017). Already drowning in a cascade of processes, reviews, and achievement indicators, it is held that impact is yet another unnecessary and exhausting burden on academics, with the added danger that they will be placed on a dreaded performance improvement plan, or even fired, if they do not prove their worth. Among critics, a terrifying picture has been painted of jaded academics marching to the drum of bean-counting ‘research managers’, parcelling out vulgarised versions of their marketised products like door-to-door salesmen (Pettigrew 2011, p. 348), and urgently Tweeting about their research to a legion of online followers, in an exercise of self-promotion described sarcastically as the ‘Kardashian Index’ (Hall 2014).

With a few notable exceptions—Andrew Vincent, for example, has warned of academics hurrying from the lecture theatre into television studios, in what he fears will be the scholarly equivalent of ‘Celebrity Come Dancing’ (Vincent 2015, p. 481)—political science has not been an echo chamber for this kind of debate. Judging by conversations in leading conferences and periodicals, including this one, it is apparent that there is a broad-based acceptance within the discipline of the need for a more visible and engaged political science (Flinders 2013a; Campbell and Childs 2013; Mhurchú et al. 2017). To quote the editors of a recent volume on the subject: ‘a discipline that studied politics but had nothing to say to those involved in politics or who might be involved would be failing’ (Stoker et al. 2015, p. 220). On the question of whether impact carries forward a neoliberal or monetarist agenda, reducing research to a series of performative investments expected to produce a financial return (Chubb et al. 2017, p. 557.), political scientists have exhibited some interest and offered their opinion, but these conversations typically take place around the departmental coffee station or water cooler rather than in academic journals. In short—unlike, say, academics in education—it has not been one of their chief scholarly concerns. Operating from the position that academic knowledge should look to advance broader societal goals and political purposes, of greater interest has been the matter of operationalisation and whether the impact agenda, as currently conceived by HEFCE, allows for and rewards the type of positive contributions to public life that the discipline seeks to achieve.

It is not surprising, we would argue, that political science is more open-minded about the basic principle of socially useful scholarship than some other disciplines. Let us take this current section to consider and contextualise this. Current REF instrumentalities are but a moment in a long discussion within political science about its role and value beyond the Ivory Tower (Farr 1988). Taking many twists and turns, this discussion started at the beginning of the twentieth century with the birth of political science as an academic discipline, often linked to the creation in December 1903 of the American Political Science Association (APSA), the premier professional association in the field.

In the United States, the link between political science and the policy world was championed from the outset. At APSA’s inaugural conference in New Orleans, its founders argued for an engaged scholarship that sought to foster social and political reform along broadly Progressive lines, and they welcomed the prospect of practitioners treading the boards in lecture halls. Between the world wars, the belief in political science as a vital lubricant of democracy and democratic values was unshakeable, fortified by disciplinary giants like Abbott Lawrence Lowell, Frank Goodnow, and Woodrow Wilson, who were leading lights in the Progressive movement, as well as scholar-activists like Radcliffe College Professor Sarah Wambaugh who campaigned for universal suffrage. An expert on plebiscites, Wambaugh planned and supervised a range of plebiscites internationally, and was one of the few female technical experts to the League of Nations Secretariat (Sluga 2006, p. 107). At the 1943 APSA meeting, President William Anderson proudly referred to the ‘special function’ of political scientists to ‘give whatever we can of our knowledge and counsel when called upon by responsible public officials’ (Anderson 1943, p. 10). Political scientists, he stressed, had an obligation as scientists to understand and explain government, but also a ‘public responsibility’ as citizens to ‘see improvements made, to see rights and freedom enlarged’ (Anderson 1943, pp. 6, 7). In 1954, APSA President Ralph Bunche rhapsodised about the ‘assistance of the profession’ in responding to requests from government departments (Bunche 1954, p. 963). In 1959, the International Studies Association (ISA) was founded in part as a mechanism to facilitate conversations between scholars and practitioners, specifically officials working in the United Nations and in the US agencies of State, Defense, and intelligence (Teune 1982).

In the UK, while not institutionalised or pursued to the same degree of theological intensity as in the US, it is important to recognise that, here too, applied political science was an important feature of the nascent political science profession. This was evident, for example, in the immediate aftermath of the First World War with the creation of the Department of International Politics at the University College of Wales, Aberystwyth, tasked as it was with studying the problem of war and of trying to enhance understanding between nations as a means of preventing future conflicts. During the first half of the twentieth century, many of the discipline’s key figures were actively involved in London-based politics and public affairs, staffing senior committees, attending consultative bodies, and even contributing to party programmes and manifestos. As Dennis Kavanagh has written, they rejected the notion of value-free political science and described themselves as ‘social engineers’, providing practical knowledge and wisdom to the country’s leadership to help reform society (Kavanagh 2003, p. 610). As they saw it, a professorial chair was a privileged platform from which to voice their opinions with the goal of influencing political processes for the public benefit. In short, it was for standing not sitting on.

William George Stewart Adams, the first holder of the Gladstone Chair in Political Theory and Institutions at the University of Oxford (1912–1933), had been a member of David Lloyd George’s personal secretariat, editing the reports of the War Cabinet, and from December 1916–1918, served as one of the Prime Minister’s trusted private secretaries (Harrison 2004). Arthur Salter, his successor at the Gladstone Chair (1934–1944), was a Conservative MP for more than a decade and worked for 4 years in Winston Churchill’s War Cabinet (Rickett 2004). Harold Laski, Professor of Political Theory at the London School of Economics from 1926 to 1950, was one of the most influential thinkers and strategists in the Labour Party during the inter-war years, and was Party chairman from 1945 to 1946. W.J.M. “Bill” Mackenzie, builder of Manchester University’s famous Department of Government in the 1950s, and widely regarded as one of the discipline’s main architects in Britain during its post-war years of expansion, was a constitutional adviser to the new states of Tanganyika and Kenya and sat on various local and regional government committees during the premiership of Harold Wilson (Kavanagh 1996; Kenny 2004). Prior to this, he was part of the dons’ invasion into Whitehall during the war, even penning a Top Secret history of the Special Operations Executive (SOE). In addition to their direct role in government, figures like Laski and Mackenzie were public intellectuals who exerted more diffuse influence over the climate of political opinion by speaking on the radio, giving public lectures, as well as writing for newspapers, magazines, and the agenda-setting publications of the intelligentsia (Kavanagh 2003, p. 610). In modern parlance, we would call them ‘impact champions’.

In the 1960s and 1970s, however, on both sides of the Atlantic, the discipline’s passion for knowledge transfer beyond the academy noticeably waned amid concerns not too dissimilar to those expressed today in relation to REF impact. The driver for this was twofold. One, during this period, the discipline became methodologically fixated on the ‘Temple of Science’, which it regarded as key to achieving greater professionalisation and legitimacy within the modern university system. As Robert Putnam has argued, with the emphasis on scientific rigour, statisticians, and the joys of micro-specialisation, public relevance became synonymous with the ‘dumbing down’ of knowledge, intellectual brand-building, and the hawking of wares normally practiced by celebrities and sports stars. Accordingly, it took a back seat (Putnam 2003). Elsewhere, paraphrasing Russell Jacoby, Ernest L. Boyer has described the process of professionalisation that started in the 1960s as relocating intellectuals from the urban coffee shop to the campus cafeteria (Boyer 1996, p. 19).

The second factor that led the discipline to participate less vigorously in broader public discourse was the fear that political science and government had become too enmeshed, to the extent that the study of politics risked becoming politicised. During this period, there were widespread revelations in the press about the cosiness of certain intellectuals and student groups with national security actors like the Central Intelligence Agency (CIA), fuelling suspicion that government was calling the tune to which academia waltzes (Wilford 2008). Believing the integrity of the discipline to be at risk from the whims and arbitrary dogma of government, the reaction of many political scientists was to recoil altogether from practitioner relations. In short, they no longer walked on both sides of the street.

However, the withdrawal from the public sphere proved self-harming and did not last. By the 1990s, it had become apparent that despite setting out with good intentions, the vaunted imperative to professionalise had resulted in the discipline becoming too esoteric and jargon-laden, fluent in a dry private language only accessible by its peers; too aloof and withdrawn behind a curtain of rational choice and formal modelling; too self-referential; too enamoured with the sheer aesthetic pleasure of knowledge for knowledge’s sake; and, so the harshest critics contended, too consumed by the navel-gazing, narcissism and toe-to-toe conceptual combat of the sub-discipline of political theory (Shapiro 2002). As Matthew Flinders has noted, the shift toward micro-specialisation had encouraged a generation of thinkers who behaved less like ‘foxes’, eager to leave their particular intellectual domain to observe and comment on the world outside, and more like ‘hedgehogs’, anxious to tend their patch and, to quote Joseph Nye, know ‘more and more about less and less’ (Flinders 2013a, p. 164; Nye 2009). As a result, the discipline was accused of marching down the ‘road to irrelevance’ (Flinders 2013b, p. 2), disconnected from the world’s most pressing social, economic, and civic problems, perfectly summed up by the title of David Ricci’s 1984 book, The Tragedy of Political Science (Ricci 1984).

Concerned about the perception of irrelevance—the scholarly equivalent of the kiss of death—household names within the discipline used their status as a bully pulpit to advocate for political science to rediscover its ‘special function’. Evoking the spirit of the discipline’s founding fathers, Samuel Huntington used his APSA presidential address in 1988 to argue that ‘works in the social sciences should be judged not only on their intellectual merit but also by the contributions they make to achieving moral purposes’, adding that it was the moral responsibility of political science to promote what Roland Pennock (1966) called ‘political goods’ such as social justice, well-being, order, liberty, peace among nations, and responsible government (Huntington 1988). Drawing on his own work on social capital and political participation, Robert Putnam used his APSA address in 2002 to argue that scientific rigour and public relevance, far from being opposing virtues or antithetical, were mutually supportive. Pouring cold water on the sniffy suggestion that relevance undermined academic complexity and standards, he explained that political scientists who leave ‘the congenial ivory-covered tower’ for the ‘hurly-burly of the public square’ actually up-skill rather than dumb down, since communicating difficult subject matter in terms that different audiences can understand was an art in itself (Putnam 2003, p. 252). Like Huntington, he also referred to the genes of the discipline’s parentage, insisting that ‘an important and underappreciated part of our professional responsibility is to engage with our fellow citizens in deliberation about their political concerns, broadly defined’ (Putnam 2003, p. 249). Contributing to the public weal, he emphasised, should not be an ‘afterword’ or ‘optional add-on’: it was a core function of the discipline.

In the US, this message was given further impetus by the so-called ‘Perestroika’ movement. Established in 2000 and emanating from the grassroots of the political science community, the movement criticised a perceived stranglehold on the discipline by quantitative and rational choice approaches (Monroe 2005). Such ‘methodological totalitarianism’, its followers claimed, meant that the best tenured jobs and top peer-reviewed journals, like APSA’s flagship, the American Political Science Review, were out of reach for anyone who dared to be different. Worse, it had insulated the discipline and robbed it of passion and purpose. According to the Perestroikans, as they came to be known, conspicuous by wearing red buttons at conferences as a show of solidarity, greater methodological pluralism was essential if the discipline wanted to reconnect with wider audiences, since those engaged in politics either as policymakers or citizens tended to favour less technical and more accessible modes of scholarship (Stoker et al. 2015, p. 10).

In the UK, the campaign to promote a more relevant and purposive discipline was led to a large extent by the Political Studies Association (PSA). In 2000, to celebrate its fiftieth anniversary, it handed out one-off Lifetime Achievements Awards to six individuals who warranted the distinction of being one of the most influential political scientists since the Second World War. In a symbolic move, among the select group was Bernard Crick (Jeffery 2009, p. 464), a scholar who stood in the honourable tradition of the publicly concerned intellectual. Published in the first decade of his career, his classic works—The American Science of Politics (1959); In Defence of Politics (1962); and The Reform of Parliament (1964)—had warned about modelling the study of politics methodologically upon the natural sciences, describing abstract theories and neutral scientism as a ‘bleak and barren world’ (Crick 1959, p. 208), and championed the integration of political theory and political practice (Hayward 1991, p. 310). In 1964, with Michael Rye (Senior Clerk in the House of Commons), he had co-founded the ‘Study of Parliament Group’, bringing together civil servants and researchers to make proposals on parliamentary reform. By acknowledging Crick’s contribution, the PSA was signalling to the profession that applied research had its place.

While not without its opponents, efforts in the 1990s and 2000s to promote a more outward looking political science ensured that when impact was thrust upon the discipline for the purpose of REF, to some extent it was knocking on an open door. Evidence of this could be found in the creation of journals dedicated to communicating research to wider stakeholders including students, teachers, and policymakers. In 2003, for example, in the service of fostering what it called a ‘political science public sphere’, APSA chartered a new journal, Perspectives on Politics, as a space for broad and synthetic discussion between the profession and wider reading publics on topics of real public relevance. In 2010, the PSA followed suit with the launch of the magazine Political Insight. It also gives prizes in the name of pioneering citizen-scholars Mackenzie and Crick. Because of the struggles waged by the coalition of Perestroikans, senior figures like Putnam, and bodies such as PSA, arguing forcefully that rigour and relevance can coexist, REF impact did not suddenly plunge the discipline into some sort of existential crisis or showdown with itself. Although it would be wrong to suggest that political science was unique among the disciplines in this respect, it is tempting to conclude that political scientists needed less convincing about why impact was important—for this battle had already been fought and hard-won—meaning that they could focus more quickly on what impact could be achieved, and how, and to what extent HEFCE parameters prove cooperative or constraining.

REFlections

HEFCE understands impact as research that has had ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ (REF 2012). For the purpose of REF, academics are asked to encapsulate and evidence their impact on private, public, or third sector domains in structured narrative case studies, which are assessed by expert panels that grade them on a scale of one to four star against the twin criteria of ‘reach’ and ‘significance’. For REF2021, departments will be required to submit a minimum of 2 case studies for up to 15 Full-Time Equivalent (FTE) staff, with one additional case study for each further 15 FTE returned (REF 2017/04). In the last REF, it has been calculated that a top-rated 4* case study was worth as much as £324,000 over a five-year period (Reed and Kerridge, 2017).

One of the greatest challenges presented by HEFCE’s conceptualisation of impact is the requirement that case studies need to provide obvious and ‘auditable’ evidence of published research directly changing the behaviour or actions of external stakeholders. Across the discipline of political science, this has been met with concern, with Patrick Dunleavy claiming that HEFCE is ‘asking for the moon’ in terms of what academics can achieve (Dunleavy 2012), unrealistically equating them with philosopher-kings. Certainly, this is not what Putnam envisioned when he called for a greater public presence for political science. More modest in ambition, his vision was of a discipline that brought fresh perspectives into the policy arena; pricked bubbles; reduced groupthink by playing devil’s advocate; and challenged conventional wisdom by adducing facts and elucidating values. ‘The most important contribution that political science might make to public life consists not in answering questions currently being asked’, he outlined, ‘but in framing new questions. Our role is to highlight ignored values, and to explicate the underlying logic that links facts and values’ (Putnam 2003, p. 251). In other words, ‘impactful’ political science was more about the conversation pursued than the consequences achieved.

Reservations about HEFCE’s definition of what constitutes impact are derived from a deep understanding of the policymaking process and the extent to which academia (and different forms of research) can realistically penetrate this world. Underpinning HEFCE’s model is an assumption of linearity: it is based on the belief that academics will diagnose a problem; design and evaluate a solution; publish that solution; and then effortlessly communicate it to grateful, rational, policymakers who will make discernible adjustments for the public good based on the intervention (Flinders 2013a, p. 153; Mhurchú et al. 2017). However, as political scientists know only too well, the real world does not always work this way.

Firstly, HEFCE presupposes that direct change on policy can be achieved by a conscious and planned set of actions, when experience shows that a temporal dynamic exists, meaning that serendipitous external factors over which the academic has little control is often the key determinant of what allows research findings to resonate with policy agendas (Stoker 2015, p. 20; Pettigrew 2011, p. 350). For example, the scholar of international relations who studies the North Korean nuclear programme would presumably be of greater interest to American defence officials today than, say, in 2003 when Iraqi WMDs topped the Pentagon’s list of security threats. Similarly, one would expect scholars working on the politics of UK social housing to have greater traction now than in the past owing to the tragedy of the Grenfell Tower fire, which killed at least 71 people in June 2017.

Secondly, as Peter John has written, there are no guarantees that policymakers will act upon or even listen to the advice given to them, regardless of how convincing or strenuously argued it is, while much research does not easily lend itself to straightforward policy prescription in any case. Even when it does, though, academics are but one of many voices competing for their attention (John 2013, p. 169). Policy advocacy is a dog-eat-dog environment, dominated by think tanks, media organisations, and professional associations who are inherently closer to policymakers than academics, and who have been perfecting the art of lobbying government for decades, often backed by considerable corporate investment (Stoker 2015, p. 24). As Gerry Stoker has explained—unlike political scientists who want to make their work as robust as possible, dotting every ‘i’ and crossing every ‘t’, and therefore operate with an elastic view of time—these rival groups are set up to produce results as quickly as possible, which is often favoured by policymakers who do not have the luxury of waiting, or perhaps even the inclination (Stoker 2015, p. 24).

Policymakers and practitioners have their own vested and personal interests, evidenced, for example, by the way many politicians transition onto the boards of various companies, perhaps most notably in the arms industry and the financial sector. It is often said that they have a propensity to seek out expert knowledge only when it fits their agenda or existing worldview, and have little time for those who look to speak truth to power. ‘Politicians’, Ronald Rogowski has written, ‘prefer policies (often put forward by charlatans) that better suit their interests, and seek to suppress or ignore evidence-based research that contradicts their own, or their ‘base voters’, ideologies. When these same politicians assert piously that political science offers no policy-relevant research, what they really mean is that it offers no research that supports their own biases’ (Rogowski 2013, p. 216). Assuming—for the sake of argument—that an academic has the stroke of good fortune to have the right research at the right time; defeats the competition from policy wonks; and is a messenger that public servants take heed; as Dunleavy has argued, in such a complicated and inherently multi-causal political world, can he or she really claim that their particular piece of research was unambiguously ‘the difference that made the difference’ in leading policymakers to chart a new course (Dunleavy 2012)? The honest answer is ‘no’.

Thirdly, there is also the problem that any model of impact that measures the outcome rather than the process sets academics up to fail, since it ignores the fact that politics is constantly in flux. A week is famously a long time in politics. As the urbane Sir Humphrey in Yes Minister was fond of pointing out, over-excited ministers and their policies come and go. It is not inconceivable that, in the period between the REF census date and the impact case study actually being read by the subpanel, policy will have moved on or been abandoned, meaning that any claim of direct change will be false and look stupid. Imagine, for example, a case study, submitted in 2020, showing that academic research had been instrumental in convincing the government of the day to change its position on Brexit, to pursue a ‘soft’ rather than ‘hard’ option. Perhaps it even features a letter from an appreciative Theresa May, confirming her volte-face. What, then, if during the case study being assessed, the terms of Britain’s exit from the European Union are suddenly renegotiated? What, as some commentators still predict, Brexit collapses? Tough luck presumably—ungraded—unless subpanels proceed with fingers in their ears, not listening to the real world of politics, until the grading has been completed.

By setting the bar at the level of auditable change, which is difficult to determine and evidence, rather than auditable knowledge transfer, HEFCE actually risks undermining its core objective of promoting research with positive real-world application. To meet expectations of demonstrating direct public utility, which is inherently intangible, there is growing tendency for academics to hold fellowships in government, as ‘researchers-in-residence’, in order to generate impact testimonials. Problematically, it could be argued that such positions confine academics to the everyday grindstone of public administration, when their true value lies in shaping the wider climate of ideas that frames policy. Put another way, it is like asking Mary Berry to make sausage rolls in Greggs. As Dunleavy has claimed, there is also the danger that academics will construct ‘fairytales of influence’ (Dunleavy 2012), loading their case studies with obfuscating proxy metrics like citation figures and ‘impact factors’ which, by nature, are baffling and alienating to outsiders. Who in the outside world would be impressed by a Thomson Reuters Journal Impact Factor of 9.216 (Barcan 2013, p. 96)? Sadly, all political scientists would! Anecdotally, it has been reported that for REF2014 some departments employed professional writers and journalists to ‘sex up’ case studies that were light of decisive evidence. Since then, there has been a proliferation of workshops offered by university impact officers, as well as corporate bodies charging hefty fees, to help academics better craft their impact narratives. The fear here is that unless subpanels spot the window-dressing, the reward will go to the best storytellers, not necessarily the most impactful cases.

Perhaps more worryingly, as Diane Stone and Tim Legrand discuss in this special issue, there is the danger of instrumentality that would see academics turn their back on blue skies research, where the social value might be nebulous at first but significant in the longer term, in favour of more mundane and less risky projects that have a greater likelihood of yielding a measurable end product within the current REF reporting period (typically a 5–6-year timeframe). Research Councils UK (RCUK) already imposes on academics a degree of instrumentality with the inclusion of ‘pathways to impact’ sections in funding applications. The danger of this is that impact frames the very way research projects are constructed, with ‘hoped for’ outcomes shaping approaches, methodologies, and strategies. Projects, in effect, become loaded from the start, serving to confirm anticipated results, rather than fostering a more open-ended approach to research outcomes. The idea of having to chase ‘easy wins’ (Lawson and Sayers 2016, p. 18), based on short-term cost–benefit analysis, is understandably something about which political scientists are uncomfortable. The discipline can boast many examples of important contributions beyond the university context—think James C. Scott’s work on subaltern politics and resistance or Aaron Wildavsky’s analysis of public policy and budgeting—but these transformative impacts were not predetermined and ‘trickled down’ (Stoker 2015, p. 25) into society over time. Research, let alone impact, is a probabilistic business: results are not guaranteed.

Concerns about the conceptualisation of impact have not fallen on deaf ears. Reporting in July 2016, the Independent Review of the REF, chaired by Lord Nicholas Stern, the President of the British Academy, argued in favour of moving away from a narrow and ‘hard’ definition of impact, typically associated with demonstrable change in the policy arena, towards a broader and ‘softer’ model that recognises the wider benefits of research on society, including influence on civic debate, cultural life, plus public engagement and understanding. There is little doubting that this would be welcomed by the discipline; political scientists have long championed that impact can mean different things for different researchers and sub-disciplines and that the ‘one size fits all’ model used in REF2014 penalised those whose research was less adaptable to applied operationalisation to policy and whose ‘impact’ is achieved through a web of influence as opposed to a linear progression. One of the interesting discussions at the PAIS Impact Conference was whether a broader understanding of impact would allow for case studies of ‘heroic failure’. Numerous examples were relayed of researchers investing enormous amounts of time and energy in cultivating stakeholder relations, only to reach a dead end, thwarted by forces beyond their control. Under REF2014 parameters, which hinge on a ‘but for/because of’ model of causation, these efforts go to waste. Would it not be fairer, therefore, as Stern seems to be recommending, to reward the dialogue with end users as much as the deliverable?

At the time of this special issue entering production, it seems almost certain that impact will have a broader benchmark in the next REF. Departments, however, will need to look closely at the detail. Suspicions are likely to exist as to whether broad level impacts leading to a more informed citizenry will rate as highly as narrow impact on government policy, especially as impacts on public understanding and culture are notoriously difficult to capture and evidence. Believing that there is a gold, silver, and bronze standard, and that the notion of broadening is a bit of false promise, some departments may decide to play it safe and continue to submit instrumentalised case studies that are geared towards policy work. Indeed, our experience is that while almost all REF information and training sessions start off emphasising a broader approach to impact, they invariably end up concentrating on gold standard policy change. Admittedly, this is not so much a problem with HEFCE impact rules, as with the internalised perceptions of impact by academics and university administrators. Notwithstanding this, the harsh reality is that policy change is tangible in a way that wider notions of impact sadly are not. By virtue of their location, plus historical ties to Westminster and Whitehall, it could be argued that this gives the Golden Triangle of Oxford, Cambridge, and London an advantage, or at least a head start, relative to institutions in other parts of the country. To our knowledge, the uneven geography of impact networks is an issue that has not entered any of the discussions of the REF Review Group, which is disappointing.

In scoring the impact element of the REF submissions, the Subpanel for Politics and International Studies will need to make delicate judgments about whether HEFCE’s guidelines should be followed to the letter. In REF2014, it is interesting to note that the impact component was scored systematically higher for most submissions, thus allowing departments to boast an overall Grade Point Average (GPA) profile that was markedly better than their performance on outputs. 45 of the 56 institutions that submitted a return to the Politics and International Studies Unit of Assessment achieved scores for impact that were higher than their overall GPA score. The data for the ‘Big Five’ departments as named by the PSA in the immediate aftermath of the REF2014 results are particularly revealing. Within this sample, there is a significant gap between the scores that were awarded for impact and the overall profile. In ascending order, these gaps were 4.52% for Essex; 4.75% for the LSE; 6.17% for UCL; 9.72% for Warwick; and 16.83% for Oxford. Reworking these figures as a global average for the ‘Big Five’ departments combined, the gap between the scores was 8.41%. Even more striking is the gap between the scores that were awarded for impact and the scores that were awarded for outputs. Again in ascending order, they were 8.28% for Essex; 9.45% for the LSE; 11.19% for UCL; 20.32% for Warwick; and 30.73% for Oxford. Averaged, the gap between the scores for impact and the scores for outputs was 15.66%. Tentatively, one can draw from this that the Subpanel did a good job in defending the profession when it came to grading impact. Indeed, there seems to be prima facie evidence that the crossbars used by the Subpanel were not those implied by a literal reading of the excessively demanding criteria set by HEFCE. One hopes that Subpanel representatives will adopt a similarly supportive stance in REF2021.

The politics of impact

Underpinning a number of the points made in the previous section is a concern that the REF impact agenda is politically and ethically naïve. This naiveté, we would contend, is a function of the ‘one size fits all’ approach, of trying to establish a common metric to evaluate research across a broad range of academic fields. Thus, while it seems relatively straightforward to determine (and evaluate) the impact of advances in the field of medical science, where notions and measures of progress and positive change are relatively uncontentious, in the realms of (international) politics and social policy that are the primary concern of political scientists, things are less clear and potentially much more problematic, politically and ethically. Accordingly, the HEFCE definition of impact as research that has had ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’ is worth reflecting on.

A key problem is that notions of ‘effect on, change or benefit to’ may not always be in alignment, since it is certainly possible to effect and change without necessarily benefitting society. In this respect, HEFCE guidelines also steer clear of engaging with the politics of the relevant constituencies/stakeholders of different forms of academic research. Society appears in HEFCE documents as largely undifferentiated and possessing unified purposes, whereas as every first year undergraduate of political science knows, politics is about the often intense debates over the distribution of resources between groups: the who gets what, when, and how. Underpinning HEFCE rhetoric of positive change, therefore, is what can appear to be a wishful and almost Socratic notion of academics taking us closer towards an unproblematic social ideal. As it relates to political (and social) scientists, the impact agenda is essentially silent and discounting of the very nature of politics as politics—a social environment of competing interests, values, and prioritisations.

Ethically, therefore, political scientists are likely to ask whether all forms of impact should be valued. Under current guidelines, it seems perfectly possible to make a 4* impact narrative based on research that has demonstrably had a positive effect on the ability of populist politicians to mobilise support and spread their message. Whether this should be viewed as a progressive or problematic effect of academic research obviously depends on one’s own political standpoint and view of a preferred social order. Anecdotally, having attended various impact training sessions, we are aware of highly regarded impact cases from the last REF that were in many respects diametrically opposed. For instance, a 4* case study based on research that was designed to assist large companies with public sector contracts enhances shareholder value and dividends by cutting back the services they had been initially contracted to provide, in contrast to a 4* case study that called for more public spending in areas of competence which some local authorities have now contracted out. Here, bizarrely, when read side-by-side, the impact of one case actually has limits on the success of the other. In REF2021, case studies relating to Brexit—a decision that has split parties and divided communities and families—are likely to be a touchstone for what exactly is the public good that HEFCE identifies. Would a case study demonstrating that academic research had successfully helped to engineer British withdrawal from the European Single Market, to the delight of hard Brexiteers and presumably a healthy chunk of the 52% who voted for Brexit, be treated as a ‘benefit’? What about one that aided tougher border restrictions and immigration rules? The answer depends to a large extent on which side of the Brexit fence one sits. Clearly, there is a high-wire act here that subpanels will need to negotiate, but the bottom line is that fairness dictates that they must leave their own politics at the door and assess impact through the self-referential claims of case studies and evaluations provided by specific stakeholders. Imagine the uproar if the Daily Mail believed that case studies with a pro-Brexit position had been unfairly penalised by ‘liberal’ subpanels on political grounds.

In instances such as this it is clear that HEFCE’s nebulous notions of research beneficial to society do not take us very far since at stake is precisely the nature of society being constructed and the interests of which stakeholders are to be prioritised. While it may be understandable why HEFCE steers clear of elaborating on what exactly it understands a progressive agenda or the public good to be—largely delegating this to the moral compass of impact leads and self-referential terms of case study narratives themselves—the very nature of the impact agenda may itself create incentives for academics to undertake research potentially leading to ‘easy wins’. In particular, research on circumspect short-term projects, framed around positivistic types of research questions, hypotheses, and methodologies with a potential for immediate impact might end up being preferred—at least as REF impact cases—to research on more politically contentious issues where evaluations of effect, change, and benefit are inherently more subjective and therefore risky.

Although political scientists might reasonably question HEFCE’s unreflective and uncritical failure to engage with the uncomfortable political questions that the impact agenda raises about notions of society/stakeholders and progress/change, political scientists themselves might also only just be waking up to how the impact agenda is creating delicate political choices for themselves and beginning to politicise departmental politics. While many political scientists still see themselves as relatively neutral and objective observers of social processes, driven primarily by evidenced based research in line with their ethical commitments, the impact agenda has begun to create tensions with this view, and so far there has been surprisingly little discussion of the compromises they are increasingly making. In part, this may be because of the way impact has tended to be viewed (even among political scientists) as a largely technical and procedural issue calling for researchers to become more active in terms of communicating and disseminating their research. However, as departments and universities have sought to maximise impact it has also begun to take on more institutionalised and official dimensions.

Collaboration and co-production of knowledge with end users are an essential part of building an impact case study. Gold-standard impact hinges to a large extent on excellent stakeholder relations. However, as several essays in this special issue show, proximity to power for the purpose of impact brings risks and challenges that hitherto have largely eluded discussion. As discussed in the previous section, policymakers have a tendency to seek out views that confirm or complement their existing assumptions, which raises the spectre that academic expert knowledge is used not to improve policy but to demonstrate rationality in the decision-making process and underwrite and give credibility to entrenched partisan beliefs (Stoker 2015, p. 21). In his contribution to this special issue, Richard Hayton warns of academics being co-opted into the ‘Westminster bubble’, drawn in, consciously or not, by the seduction of privileged insider access, such as invitations to participate on government-commissioned panels, and the promise of glowing impact testimonies from officials. By means of a close textual reading of the case studies submitted by the top 15 Politics departments in REF2014, Jan Selby’s analysis suggests that there is perhaps some veracity to this observation. Not one of the 43 case studies, he argues, corresponds to the criteria of ‘activist scholarship’; not one is undergirded by Marxist, radical feminist, or post-colonial thinking; none have titles that speak of ‘confronting’, ‘resisting’, or ‘condemning’. Rather than slaying dragons, as he puts it, or opposing policy orthodoxies and established power structures, the majority of case studies are essentially mildly reformist narratives of political scientists working with end users, in a benign and non-conflictual relationship, to fine-tune the techniques of liberal governance. Seen in this way, the impact landscape is one of academics who perhaps started off with the intention of sweeping away the cobwebs of state corruption, malfeasance or failure, but who often end up notably conformist, co-opted and compromised to varying degrees.

An interesting example of the hitherto underexplored ethical challenges raised by the impact agenda concerns departments increasing use of boards of ‘impact advisors’—typically non-academic representatives from political circles, business, and NGOs—who it is hoped might help them cultivate links with diverse groups of external stakeholders, providing easier access for the dissemination of departmental research. The University of Sussex, for example, has a Global Studies Impact Advisory Board and in 2016 appointed Yvonne Roberts, the prize-winning feminist journalist with the Observer, as the University’s first ‘Political Writer in Residence’. At Warwick, PAIS has an impact board chaired by former Secretary of State for Defence Rt. Hon Bob Ainsworth MP. Several matters can arise when departments seek to establish such boards and positions. Not least among these is the fact that people approached are, by definition, believed to have access to institutions and people that are influential, and may also be viewed as influential themselves as a result of their previous or ongoing roles. Given the nature of our subject, this can mean establishing links with individuals and networks whose track record (depending on one’s perspective) may be problematic. Links with people close to the military, arms industry, or closed practitioner communities like intelligence may be an obvious case; but we might also think about people with ties to major corporations with track records of worker and environmental exploitation—or indeed, political parties. Advocates of such boards would suggest that these links provide an opportunity to close the gap between the policy world and academic researchers and, crucially, allow the latter to speak truth to power. Critics might counter that any such notion is naïve and that these links simply end up lending academic credibility to particular individuals and organisations.

Relatedly, however, issues may also arise in regard to the personal politics of people on impact advisory boards and the extent to which their politics are deemed to be relevant to their appointment or not. In a context of rising populism, in which it has become clear just how out of touch the academy (including Political Science as a discipline) has become with the politics of a very large proportion of the population, there might be a good argument for opening up such advisory boards to a broader range of political viewpoints—even if those views are deemed deeply problematic to large numbers of faculty and students. The trade-off is that appointments like this may provide such people with an enhanced platform and legitimacy that comes from a University affiliation (even if that has become increasingly questionable). It may also be that the objectionable views of impact advisors only come to light down the line, with their having been appointed primarily for their technical expertise and networks, not for their personal politics. Of course, insofar as the personal politics of impact advisors do not infringe any laws or University ordinances then these should be irrelevant. Moving forward, though, the ethical complexities of the impact agenda might make it incumbent on departments and universities to draw up terms of reference for engagement with external stakeholders. This will not be an easy task. Any policy would need to be consistent with extant legal parameters relating to freedom of speech and scholarly freedom. It would also need to be sensitive to the values of departments, however they themselves are defined, plus understandings of reputational risk. When appointing advisors, departments might be advised to ask whether the person’s politics or political associations are likely to compromise their ability to give objective guidance relating to impact as required by the advisory board role. Establishing ground rules on the appropriateness of impact bedfellows is clearly a grey and contentious space; but it is our view that the need to do so will become ever pressing not only as the demands for impact increase, but as universities manage their commitment to free speech in the face of a growing trend among some students for the ‘no-platforming’ of certain individuals on campus.

Conclusion

In conclusion, this introductory essay has sought to situate REF impact within a longer disciplinary history and framework of political science. To date, the discipline has been relatively open-minded about the impact agenda—not least understanding public engagement to be an essential part of its remit from its earliest days. This position, we have suggested, has its roots in the discipline’s genesis and formative years during the first half of the twentieth century when key figures within the profession preached the importance of harnessing expert knowledge to solve real-world problems. Indeed, for political scientists, intervening in the marketplace of ideas was believed to be a civic duty as much as a scholarly pleasure or pursuit. Although attempts to professionalise the discipline in the 1960s led to a period of pulling up the drawbridge to the world outside, by the turn of the twenty-first century a commitment to applied scholarship was again high on the agenda. As a species, we would argue that political scientists conform less to the stereotype of the traditional public intellectual, a critic who explains why a policy is ineffective or immoral, and more to the model of what David Dresner calls the ‘thought leader’, a creator who has positive ideas for change and is determined to make a difference (Dresner 2017).

Nevertheless, important questions about impact remain. Some of these, we have argued, concern how it is presently conceptualised and measured within REF and where the notion of a ‘gold standard’ connected to auditable change places an emphasis on the policy realm. A positive reading of the ‘but for/because of’ model of impact is that it might discourage political scientists from becoming world-weary residents in a metaphorical ‘Grand Hotel Abyss’—the phrase coined by Hungarian Marxist György Lukács to describe the thinkers of the Frankfurt School who, he claimed, engaged in comfortable armchair philosophising about the unfolding catastrophe of monopoly capitalism and its destruction of the human spirit, all the while cocooned, doing nothing practical to stop it (Jeffries 2016). A less charitable reading, however, is that auditable policy change is not only unrealistic, but also potentially retrograde in its failure to understand the nature of the decision-making process, let alone the nature of much disciplinary research that simply does not translate well into clear policy recommendations. As discussed, policymaking is the result of a complex and at times chaotic interplay between different actors, networks, and institutions, each with their own politics and path dependencies, meaning that it is hard for anyone from any field to have direct influence on policy. Moreover, at a time when the value of the ‘expert’ has seemingly declined and the feedstock of social and political life is a mishmash of fake news, alternative facts, and post-truth politics, it might be argued that it has never been harder for academics to influence policy in the manner gold-standard impact demands. Furthermore, ironically, the best evidence to demonstrate ‘but for/because of’ policy change comes from statistical correlations and the number crunching of datasets—precisely the kind of alienating esoteric scientism that political science has rebelled against as a barrier to public relevance and which, one suspects, flies in the face of everything the impact agenda seeks to achieve.

Alarmingly, there is at least anecdotal evidence of the impact agenda impacting on the nature of the research process, which should be sacrosanct. This is no less evident than in how ‘pathways to impact’ need to be written into RCUK funding applications. Inevitably, this plays a disciplining role in terms of shaping the questions, approaches, methodologies, and strategies researchers are liable to adopt. In this respect, any move towards a broader and looser conception of impact more in line with Putnam’s emphasis on political science’s ability to raise new and challenging questions and highlight underpinning logics and ignored values—as opposed to a narrower focus on providing answers to discreet questions—can only be welcomed. It is essential that political scientists remain resolute in defending the critical ethos that underpins much of the discipline. One way of doing so will be to have the courage to avoid the temptation of ‘playing safe’ and, in the forthcoming REF, be open to submitting a broader range of case studies (including heroic failures) that better reflects the full breadth of the discipline. In writing our case studies, we should also not capitulate to the tactics of advertising gurus who hold that evidence and analysis are insufficient, and that facts have to be sold as well as told.

Finally, there are important questions relating to the politics and ethics of impact. Some of these questions go to the very heart of the impact agenda where issues of positive change, the public good, and the public interest exist largely as empty signifiers whose use seems to preclude any real engaged debate as to what they might mean in practice to different constituencies. The questions, however, are not just for the crafters of the impact architecture, but also for the everyday practices of political science departments and political scientists themselves. Instead of viewing impact as a largely technical add-on, an issue primarily of communication strategies and public engagement, there needs to be greater recognition of the compromises and tensions that the impact agenda is bringing into our everyday practices. In the absence of definitive answers, it is beholden on us to remain critically reflexive as a community about the political and ethical choices we face.

The articles in this special issue speak to many of the points discussed above, but also raise further important elements for consideration and future debate. The first five articles address more general concerns about how REF impact has been understood and received by the discipline. The first, by Claire Dunlop, provides a political economy-based analysis of politics and international studies REF2014 case studies, assessing case study submissions in regard to the themes of: who has what impact and when; impact’s beneficiaries; impact’s evidence base; and the challenges of generating and validating impact. In the second article, Jennifer Chubb and Mark Reed pursue a similar line of enquiry. Drawing on qualitative interviews with UK and Australian academics, they note that the desire to demonstrate the societal and economic value of research has raised questions about the unintended implications of impact agendas on the nature and quality of research.

In the third article, by means of a case analysis of electoral integrity in Britain, Toby James examines the types of policy impact that are possible within political science, as well as raising questions about the difficulties and dangers inherent to evaluating impact within our field. Overall, he argues that while impact is possible and desirable, it requires us to reconceptualise the nature of knowledge that political scientists are ultimately able to generate, one that moves away from realist testable scientific understandings to embrace how political science knowledge is socially produced and therefore liable to change and be adapted once it becomes known by social agents. In this respect, article four, by Jan Selby, provides an interesting (and sobering) reflection on the underpinning research that dominated REF2014 case study submissions to the politics and international studies subpanel. As he notes, the vast majority of case studies lacked a distinctly ‘critical’ or oppositional component with respect to challenging established orthodoxies. In response, he argues for greater willingness on the part of critical scholars (broadly defined) to undertake impact, but also for research councils and departments to broaden their own conception of what impact is and how it is achieved. His rallying call is that ‘additive’ approaches to impact are all very well, but our ‘impact heroes’ also need to be seen to be slaying some dragons. In the fifth article, Sioned Pearce and Dan Evans then provide a historical contextualising somewhat analogous with the arguments outlined in this opening Introduction. Impact, they argue, is not as new as frequently depicted in debates about REF, but has a heritage extending back decades. What is new, however, is how impact has become embedded into core funding allocations to universities, with this raising important questions for the nature of research undertaken.

Articles six, seven, and eight then shift the focus to more sub-disciplinary issues and concerns. In the sixth article, Richard Hayton considers what the impact agenda may mean for the study of British politics and of British politics as a sub-discipline. From a period of having been on the retreat, he argues that impact actually seems to be reinvigorating the sub-field, with a striking number—far exceeding what might be expected given the recent general trend towards internationalisation of political science—of REF2014 case studies having a British politics focus. However, he also suggests that we need to be mindful that the reinvigoration of research in this area does not become focused on overly narrow issues. In the seventh article, Robert Dover and Michael Goodman discuss the implications of the impact agenda for Intelligence Studies. Although the clandestine world is secretive and typically closed to outsiders, they argue that achieving impact in this realm is possible, albeit the issue of how to record impact can be tricky when working with security and intelligence practitioners. However, they also note that intelligence studies are not unique in this respect and that there may be lessons to be shared with scholars working on other topics where confidentiality is also a key concern.

In the eighth article, Diane Stone and Tim Legrand shift the sub-disciplinary focus to the area of public policy, but in doing so, also highlight the extent to which impact agendas are becoming increasingly transnational. In this process, they indicate that general conceptions of REF impact in terms of how academic research can effect positive change represent only part of the picture. What we also need to be mindful of is how (social) science is fast becoming a site of interstate competition in a global knowledge economy. Thus, it is not simply that research is seen to inform public policy and that we increasingly see science being utilised in diplomacy, but that diplomacy is increasingly being used to enhance science (in the form of international cooperation and funding mechanisms). Moreover, such cooperation is also seen as a means of improving relations between countries. We can therefore expect that universities will themselves place greater emphasis on ‘science diplomacy’ as a modality when thinking about research impact in the future.

In the final article, Andrew Hammond, a former Andrew W. Mellon Postdoctoral Fellow at the 9/11 Museum and Memorial in New York, draws upon his personal experiences in the creative and knowledge economy to investigate the value of museums for political scientists, as objects of study, as collaborators in the production of knowledge, and as partners for impact. To date, he argues, the discipline of politics and international studies has failed to properly study or engage with museums, which is surprising given that they are inescapably political. Museums feature exhibitions that address, both directly and indirectly, political subject matter. Moreover, they have their own ‘politics’ and are often at the mercy of political pressures. Even in Western liberal democracies, processes of remembrance and representing the past are heavily politicised, involving a complex array of stakeholders, including governments, curators, museum staff, funders, and members of the public. Importantly, for impact purposes, suggests Hammond, museums are sites of memory that political scientists can look to embed their research, with the goal of influencing popular attitudes and shaping public understanding.