The building of weak expertise: the work of global university rankers

Expertise is among the most important and valued ‘commodities’ in society. Several cultures of expertise have emerged around the figure of the expert (Holmes and Marcus 2006; Shore et al. 2011; Strathern 2006). Although, as recent political events have shown, their authority is increasingly challenged, those who are recognized as experts usually have their voices heard and wield an array of social, political, and economic power. In this paper I introduce the concept of weak expertise.

Drawing insights from a two-year field study in the sector of university ranking organizations, observations from nine important conferences and fairs on the rankings circuit, eighty-four interviews, and an analysis of hundreds of related documents and news articles, I argue that this kind of deeply contested yet influential expertise can be built in a variety of ways. The constantly negotiated balance between the relevance, reliability, and robustness of university rankers’ data and their relationships with their key readers leads to the development and exercise of this kind of expertise.

University rankers are having a real impact in how education is being delivered and how research is being done (Hazelkorn 2007). This is in spite of the critique that many of the organizations and individuals that produce rankings are not recognized as traditionally having formal training or qualifications in education studies. If anything, the trend is also a critique of those who work in higher education. Leon Bolstein, president of Bard College, observes that ‘(i)t’s one of the real black marks on the history of higher education that an entire industry that’s supposedly populated by the best minds in the country… is bamboozled by a third rate news magazine¹’ (Bolstein, in Espeland and Sauder 2016).

The influence of rankings in the face of the world’s ‘best minds’ makes their influence even more intriguing. University rankers wield an extraordinary power that some have situated in the wider audit (Power 1997) and neoliberal (Ball and Olmedo 2013) society. Here I show, from the perspective of the rankers, how ranking organizations craft active responses, strategies, and new products, to respond to the needs of universities, as sense-making organizations (Di Maggio and Powell 1983) that are in pursuit of their community’s commonly held notions of ‘excellence’ (Readings 1996).

There is an important gap in this field. The perspective of what rankers do, from the ‘inside’, is absent. By observing parts of the production of the 2014–2015 THE World University Rankings and events on the international rankings circuit,² I suggest that rankers mobilize two important resources to shore up their credibility: (1) data and (2) their relationships with their audiences. These are brought together by the ranker to construct rankings that are robust, reliable, and relevant. These key elements that underpin rankers’ influence in their sector are also the qualities that must be studied in the work of new and emerging experts in other fields.

This article aims to contribute to our understanding of how evaluation instruments affect organizations in the higher education sector, but could be usefully applied to many other organizations which grapple with organizational anxieties (Sauder and Espeland 2009; Fligstein and McAdam 2012). It offers insights into what constitutes expertise. In particular it aims to unpack the strategies, methods, and particular practices that university rankers, as exemplars of weak expertise, undertake to legitimate their knowledge. This explanation requires an approach that includes an insight into the internal life of the university ranker as well as from the ranking events and conferences at which rankers are actually at work.

This paper relates and explains the processes that constitute rankings production as well as the actual strategies by which ranking organizations publicize their products. It builds upon the knowledge of the reactivity of universities to rankings (Espeland and Sauder 2007) and adds an understanding of a crucially important element: the processes which rankers use to harden the facticity of their ranking knowledge and, as a result, give them a form of expertise in questions around quality in higher education.

Concretely, I show how one ranking organization, the Times Higher Education (THE), sought to maintain its legitimacy in the face of opposition and critique from important stakeholders whom it considered as part of its core readership and target audience. I discuss the THE’s decision to develop a standalone ranking separate from its former partner Quacquarelli Symonds (QS). I also present the introduction and development of a new instrument—the ranking of universities by their innovativeness—in which the THE needed to coordinate and solicit the help of its previous data provider, Thomson Reuters,³ and its current partner, Elsevier. The THE also asked for feedback from academic advisers as well as the larger public about how to measure the university sector’s ‘third mission’ performance, i.e. making a contribution beyond teaching and research.

Rankers produce their ranking instruments through a process of gathering feedback, engaging with their audiences, and carrying out strategic work to legitimate their instruments. I suggest that rankers produce contested data and need to frequently respond to challenges to their legitimacy. The reason for all this is the need to build trust and credibility, which are the bases of the rankers’ claim to expertise and also the root of revenue. Building on this, the concept of weak expertise opens up new questions about the nature and exercise of expertise in higher education and, more widely, in many other spheres where experts actively build and promote their influence.

When rankings are discussed, there is often great concern over impact (Marginson and Van der Wende 2007). The changes in behavior that rankings bring about—impact—have been studied on the part of many actors. Hazelkorn’s (2007) large survey of university managers concludes that rankings are ‘here to stay’ and that, despite the criticism that rankings receive, people will just have to ‘live with them’ (Hazelkorn 2008). More than learning to live with rankings, it is becoming more and more important for university managers to play the rankings game well. Governments are also concerned (Lim and Williams Øerberg 2017). In 2008, the European Commission supported an alternative rating system, eventually U-Multirank, in response to the dominance of American and Anglo-Saxon universities in the then THE-QS rankings.

The prominence of rankings in higher education is part of a wider, and longstanding, rise in the importance of indicators and measurements in general society and in the larger field of education. Lawn (2013) explores the ‘voyages of measurement’ which shows the inexorable rise of metrics, data, and standardization in education throughout the twentieth century. Normand (2010) shows the networked nature of organizations such as the Organisation for Economic Co-operation and Development (OECD) to explain how ideas flow between policymakers and expert communities. Schuller (2005) shows how the prevalent ideas among the OECD epistemic community ‘identify and prioritize’ issues (Schuller 2005) as well as shape the techniques by which these issues are to be studied and potentially resolved. Politics, ‘political work’ (Grek 2013), and networking all play a large role in the agenda setting around ‘technical work’ in education. Networks of academics, policymakers, and experts were all necessary to provide the conditions for the rise of metrics in education.

Normand (2010) suggests that policymakers and leaders have a choice when picking which experts to listen to, but that they often choose those whose opinions closely match their own. In education policymaking, the understanding of the relationships between policymakers and experts exposes some of those choices. I argue that university stakeholders are not powerless in the face of rankings and have the ability to choose the importance of some rankings, or some experts, over others.

The networks of experts and the spaces in which experts meet and socialize are important sites to understand how weak expertise is generated, hardened, challenged, and defended. This account of the THE and other rankers’ experimentations and hosting of conferences addresses the need for thicker descriptions of the processes around the production of university rankings.

To begin understanding rankers is to reflect on their operational context. The THE styles itself as the ‘global authority on higher education’ (Baty 2015). The THE rankings are produced in the context of a newsroom. This sets it apart from other ranking producers. This newsroom is itself part of a larger organization that can be described as an education community platform, TES Global, which is run as a business. However, this context is not static: during the research period it was and continues to be in the process of reorganization which is changing the way the THE goes about its business. This is crucial to understanding both the networks around the THE, its target audiences, its strategic actions, and the potential directions it will take in the future.

The World Academic Summit, organized by the THE and hosted by the University of Melbourne in October 2015, had the air of a diplomatic summit rather than an academic meeting. Present were a number of vice chancellors and other university leaders. The venue was the newly built Melbourne School of Design, a visually stunning building at the heart of the university’s campus. Organizers recognized it as an ideal place to provide a platform for dialogue among global ‘thought leaders’ in education. Central to the summit was the launch of the THE 2015 World University Rankings and a rankings ‘masterclass’ held by THE editor-at-large Phil Baty. The attendance of high level university leaders and policymakers is considered one measure of success. At Melbourne, Dirk Van Damme, head of the OECD’s Innovation and Measuring Progress (IMEP) division within the Directorate for Education and Skills, spoke about the importance of metrics about teaching in higher education (Van Damme 2015). At this, and its other events, the THE aims to foster conversations about standards-setting at the highest policy levels in education. The desired tone is one of relevance and broad policy impact.

At Melbourne, Baty and a large number of THE staff invited their audience to contribute to what was presented as a work in progress: the evolving THE university rankings. The THE aimed to use the Summit to offer the space in which to reflect upon, gather data for, and ultimately answer some of the issues raised around its own ranking instruments. Baty openly posed questions about the THE’s own practices and asked the audience to help the THE make progress. Many of those in Melbourne actively engaged in this reflective practice and appeared to look forward to the next Summit, held at the University of California-Berkeley in 2016.

Observing ranking summits brings up two important features of university rankings that are relatively understudied: how rankers become experts and what work they carry out in order to produce new, better, and more relevant products. At ranking events, both ‘who’ and ‘how’ questions about university rankers can be addressed. By observing the conference circuit it becomes clear which ranking organizations operate in which countries and policy spaces, as well as which individuals work for, manage, and advise them. The circuit also shows how rankers develop material (e.g. brochures and magazines) as well as virtual objects that serve as ranking ‘instruments’. These instruments include the many related events that are now a part of the university ranker’s growing list of operations.

The growing community of ranking practice that gathers around the annual events organized by university rankers reinforces the need to answer the ‘who’ question: it appears that a relatively small group of people are having an outsized impact in higher education policy. Although Hazelkorn (2008) and Espeland and Sauder (2007) show the extent of the effects of university rankings on university managers, academics, teachers, and education policymakers, there is not much by way of what the rankers actually do in terms of their community-building efforts. Rankers are not only affecting university leaders ‘at a distance’, to use a term from governmentality scholars (Foucault 2000); they are actively organizing events and other platforms that serve as spaces for direct dialogue, debate, and sharing of ‘best practices’ between themselves and university leaders.

University rankers begin their work by trying to conceptualize ‘quality’ in higher education. In this task, rankers have to choose both the likely indicators as well as the ways to obtain or generate the relevant data. Choices need to be made regarding the use of certain kinds of inputs: subjective and/or objective. If they use certain kinds of data or propose a certain indicator, they need to have a reasonable and articulated justification for it. In the case of the THE, the use of thirteen indicators was meant to provide a ‘balanced and comprehensive’ view of excellent performance. This wide range of indicators is not only meant to provide ‘balance’ but also relevance to political and social concerns.

Rankers are sensitive to national and international policy discussions (Lim and Williams Øerberg 2017). For instance, some policy concerns have recently shifted towards the need for better measurements of teaching quality. In the UK, these include proposals around a teaching excellence framework (HMG 2016) and of other ways to improve the social value and impact of higher education. Rankers have taken up these ideas and set to work on experimenting on the tools and devices that could potentially address these concerns. Ranking organizations such as the THE have begun to expand from the measurement of research towards the measurement of other factors, e.g. innovation. Rankers provide a link between these policy concerns and calculative devices. They can attempt to operationalize ideas that are present in the discussions of the higher education community’s ‘thought leaders’.

Ranking experts seek to ‘translate political concerns into expert languages’ (Tan 2014: 306). These concerns could include prevailing government discourses around improving ‘value for students’ or gaining ‘competitiveness in a knowledge economy’. Rankers build new languages around these directives that contribute to the new ways of seeing university quality and behavior. To policymakers and users, experts provide “the techniques needed to fulfill their aspirations” (ibid) as well as the ability to teach them these techniques. The new experts then begin to gain the force of legitimacy. They begin to produce facts and their viewpoints are taken up in the outside, real, world.

As ranking devices become more complex, they require increasingly sophisticated data and the corresponding data expertise. Some university ranking outlets began, quite literally, as one-man affairs. Today, however, they involve many more professionals and workers with various tasks and the corresponding skill sets. Rankers are still both surprisingly small—in the sense that a very small number of people make decisions—and surprisingly widespread, when one considers the many auxiliary professionals, associated experts, and targeted audiences that work for and engage with them. Thus the ranking community is now wider and deeper than it was previously.

The network of experts around rankings suggests that ‘it takes a village’ of regulators, operators, audiences to raise a ranking instrument. Some professionals are entrusted with the verification and dissemination of the data. The production of rankings then expands from the world of the rankings editor and extends to the personnel at the data provider and then on to the various data professionals, visual artists, bibliometricians, those who carry out the reputation survey, the staff at the events team that organizes conferences around rankings, those who manage the rankings website and social media, the academic advisers of the ranking organizations, the media contacts who use rankings as part of their news feeds, and many others.

The ranking space then extends far beyond the physical office of the rankings maker. I argue that it encompasses the various conference sites, even spaces in universities themselves where rankings are studied and incorporated into strategy, insofar as the professionals working in these offices return feedback to the ranking makers. This wide set of actors with their different kinds of expertise comes together to produce the underlying ‘science’ of assessment. All these actors are linked to the production of the ranking instruments and building of the ranker’s expertise.

Data is of primary importance to the expert. It is the basis on which expertise is built (Grek 2013). Rankers conduct experiments on their data and ask academics to evaluate their work. In this way, rankers develop both new kinds of data and new perspectives on their data. This gives them the potential to extend their expertise. While they take into account the feedback they receive, in the end, rankers decide on their own what data to use, which indicators to include, and what material to publish.

However, for the university ranker, it is not enough to have data. The datasets needs to be processed and they need to be re-presented and made relevant to the THE’s various audiences. The process by which the THE treats and gathers its data is the subject of an infographic or special section in every THE rankings issue. There is always some commentary on the THE’s methodology as well as commissioned thought-pieces on the rankings’ data from technical experts. Conversations about the data are present at every THE-organized summit. Here, the THE aims to get its primary audience in the room and have the most direct conversation as possible with them. Baty explains that their events are ‘absolutely for university leaders… mainly presidents and vice presidents, Vice Chancellors of universities, strategic leadership…’ (Baty, personal communication, 12 September 2014).

The development of the THE ranking methodology has always aimed to address concerns among its core audience: academics and university managers. Prior to 2010, the THE and QS produced a single set of university rankings. Critiques of this joint ranking ranged widely, from technical shortcomings (Bougnol and Dulá 2014), to anchoring effects that favored long-established universities (Bowman and Bastedo 2011), to audience mis-targeting (Cremonini et al. 2008), among others. Baty and Anne Mroz, then editor of the THE, felt that if academics and high-level administrators were unhappy with the rankings, this could potentially result in difficulties with the circulation and thus the revenues of the magazine.

The changes that the THE introduced were not only defensive changes. They did not only change the indicators that received the most criticism, such as the perceived overweighting of the reputation survey or the skewness of the citation metrics towards the hard sciences. The go-it-alone THE rankings also introduced new indicators, such as an innovation indicator, to reflect the concerns of its audience. The goal was not only to correct flaws but also to increase its own relevance. The experiments around these methodological changes needed to yield results that corresponded to commonsensical outcomes that were worthy of the higher education community’s trust. For instance, there was a common ‘feeling’ that Ivy League universities should end up somewhere close to the top of the table, as this was held to be ‘common sense’.

I suggest that it is vitally important for experts to have robust data. However, this data needs to be made relevant and built into the discussions in communities of practice. Data and relationships are the two resources that rankers jointly use: they involve their audiences in gathering and strengthening the datasets that they gather and produce in order to arrive at products that are seen as robust, reliable, and relevant. One of the ways that rankers develop their data and their relationships is through experimenting with different kinds of indicators as well as representations and tables that their audiences accept and value. University rankers carefully consider what kinds of data and indicators are relevant to higher education stakeholders and enter into a dialogue as to what kinds of data are robust (Baty, personal communication, 12 September 2014).

This kind of gradual ‘assemblage’ of calculative ideas and devices is the foundation on which expertise is built (Tan 2014). University rankers gathered their own datasets and developed different instruments suited to different audiences. The differences among the ranking instruments, such as the different methodologies, aimed to capture different qualities of higher education excellence.

Different rankings are oriented to different audiences over which these rankers want to exercise a certain expertise. The QS rankings, for instance, can generally be described as more (international) student-oriented. The Leiden university rankings are oriented towards readers who are more concerned with research performance measured by bibliometric output. The differences between rankers, their audiences, and the various professionals that support them point to the networked nature of the industry of rankings. In the world of rankings, there are multiple actors, multiple sites, and multiple experts.

However, the general trend that binds the rankers together is that all universities have stakeholders who are interested in the excellent performance of their institutions. Rankings have become the mechanism by which the organizational ritual (Powell and DiMaggio 2012) of excellence is more commonly understood and measured. This, in turn, has led to the establishment of ‘research and/or reputation management’ units within the universities—new institutional forms (Meyer and Rowan 1977; Fligstein 2001) that have spread throughout the sector as legitimate agents of excellence and accountability—that gather data for and deal directly with the ranking agencies.

The point on data ownership eventually proved to be a critical one. Although Reuters’ data was the foundation of the industry-income-innovation indicator used in the THE World University Rankings until 2014, in the end, Thomson Reuters produced its own rankings. In 2015, Thomson Reuters released its Innovative Universities ranking, in which it “set out to find and rank the world’s top 100 innovative universities empirically, building a methodology that employs 10 different metrics” (Thomson Reuters 2015b). Their emphasis on the reach of their data led them to boast about reaching the ‘essence’ of university innovation and identifying ‘reliably’ impactful institutions (see Fig. 1).

Open image in new window

The THE could not produce an innovation ranking comparable to Thomson Reuters’ product, because it did not own the data. It decided to shift to another data provider, Elsevier, in 2015, and produced its own innovative university rankings based on the information it received from its new data partner. These rankings were based on four ‘innovation indicators’ that the THE used in its World University Rankings. These were: (1) resources from industry, (2) industry collaboration based on papers published with an industry partner, (3) patent citations and (4) proportion of industry contribution to the university’s overall income. The Elsevier database seems less comprehensive with regard to industry and patents than Thomson Reuters; nevertheless, the THE presented some of their new datasets in ways that would be interesting and relevant to university leaders. The THE showed, for example, a comparison of industry income generated on average by each academic in a range of countries. They then extended this analysis to individual universities in order to rank them (Fig. 2).

Open image in new window

Control over the data to produce these different rankings was very important. As I have shown, although Baty wondered whether innovation rankings would themselves be a commercial success, it was important to have the ability to produce new kinds of rankings that could then become potential revenue streams. These new rankings would depend on new datasets. In the case described above, the THE was inventive in generating its own innovation ranking despite changing data providers and in the face of its old partner Thomson Reuters developing their own similar ranking.

Up until 2014, there continued to be limited internal capacity on the THE’s part to understand the data behind the rankings they published. Within the THE, there were no ranking data analysts, professional bibliometricians, or statisticians hired directly to work on the rankings. These were left to data analysts working with the THE’s partners, such as Thomson Reuters. Because the integrity of data and its correct interpretation and presentation were crucial to cementing the THE’s claims of expertise in providing a useful aid to thought leaders in the sector, the THE decided to invest in building its data collection and analysis team. This process continues and Baty described this ongoing change as ‘the largest investment in [THE’s] history’ (Baty, personal communication, 15 April 2015).

The THE investment in building its data-building capacity was meant to secure the ‘sustainability’ of its rankings and the business model around it (ibid). Building its own data team meant that the THE could collect new forms of data, allowing it to produce other rankings based on a different mix of indicators as well as undertake in-house comparative data analyses of universities. This model underlies the THE’s claims to expertise. As the THE transforms into a data company itself, it is seeking to guarantee one of the key sources of its expertise even as it maintains and strengthens its ability to make data relevant to its community.

Credibility and relevance are both crucially important to the THE. It is not only a matter of its standing in the higher education community. To the extent that it is successful, the THE is able to convert credibility and relevance into material resources and a revenue stream. The magazine has a core audience; it claims it is meant specifically for higher education professionals and researchers. Yet it also has a wider audience. Many other kinds of readers, such as students, parents, and the wider public, are interested in the THE World University Rankings as well as its other regional and reputation-only-based rankings. The magazine needs its expertise to be recognized by these various audiences.

In order to sell its magazine, the THE needs to be trusted. The THE’s subscription and advertising space model works on the basis of its relationship with its readership. Its audiences need to be convinced of the value of the magazine to buy the magazine issues and to advertise in its pages. For the THE to profitably organize its conferences, it needs its expertise to be recognized. At its conferences, commercial exhibitors need to be convinced that the THE’s expertise will draw the right kind of thought leaders to attend.

Baty stresses that ‘for a newspaper trust is everything’ (Baty, personal communication, 9 December 2014). Trust was ‘everything’ for the THE because it maintained the value of the newspaper and its judgment in the eyes of its targeted readership and community. Different mechanisms were and continue to be used to build that trust. These include the summits, Baty’s many field trips, and the THE’s many online media platforms in which Baty and the organization dialogues with their different audiences. Through these various platforms, Baty and the THE have the chance to explain ranking developments including potential changes in methods for gathering data, the reason for various methodological choices, and why they gave certain weights to certain characteristics. These discussions are all meant to be transparent and open and to give audiences a potential stake in building the ranking instrument. The list of trust-building strategies is a long one because, despite the many efforts on the part of rankers, some among its audience remain deeply skeptical.

In this paper I argue that ranking organizations build their expertise using a variety of strategies. These include the use of ‘hard’ data such as bibliometrics and interaction with various stakeholders—university leaders, other data experts, and even students. The first involves the gathering, compilation, and delivery of ‘robust’ data on higher education. The second entails various processes of building a community of users, of an audience, that places its trust in the ranker and who considers its expertise relevant to real problems. These are at least two of the components that form the basis of the expertise that global rankers seek to build. This paper contributes important insight into this process, because there is a gap in the understanding of how the influence of rankers as ‘new experts’ takes place from the perspective of the rankers’ own activities.

Throughout the paper, using sources both within and outside the ranking organizations, I have outlined the ways in which the THE has expanded its network of both data professionals and workers in order to build the capacity to gather data, to test data, and to develop changes. Other workers have taken up other roles—e.g. acting as interlocutors with their audiences, organizing platforms for dialogue among higher education thought leaders, and gathering feedback about their own instruments.

The overall structure upon which rankers carry out their expertise-building strategy is widespread, but can be experienced more readily in a ranking ‘moment’ such as the World Academic Summit at which many of those who are members of the ranking community are present. These conferences matter because rankers do not become experts in the field on their own. Rather, they engage in many activities that are brought about by a need to win over a certain complicit consent from their audiences.

There are many reasons why organizations seek external validation, but one of the most important is the need to be seen as an excellent university or at least on the road to excellence. Universities are in the grip of a powerful reputational ‘iron cage’ (Di Maggio and Powell 1983) in a sector increasingly subject to comparison. However, by presenting the strategies used by university rankers, I also want to suggest that higher education leaders have the capacity to reflect, resist, and, importantly, shape the metrics by which they accept to be ‘judged’. Rankers are only weak experts in that they constantly need to strengthen their expertise through a sustained engagement with their principal audiences.

Rather than assuming that all of the trust-building mechanisms have solidified the hold of the THE over its audience, they can be seen as signs of a constant struggle for trust and credibility among a skeptical audience. Nevertheless, the THE and other ranking organizations do indeed have policy effects. The THE’s claim to be the global authority in higher education seems a vastly ambitious one but it does not detract from its actual influence (Hazelkorn 2008).

However influential they may be, rankers are only weak experts. The weakness of the ranker’s expertise is one of the reasons behind the investments in data-gathering at the THE and the growing refinements in data treatment among the other rankers. Although they are still subject to much criticism, rankers have had to change their methods in order to make them, in the eyes of their audiences, more robust. The various feedback mechanisms—including social media and conferences—are ways that rankers such as the THE maintain a conversation with users but also critics. They are signs not necessarily of strength but of a constant need to keep the academic community on board.

Rankers never achieve a full and unquestioned grasp of their data; their audiences are continually bringing up various critiques. The spaces (e.g. conferences) and processes (e.g. refining methodologies) of trying to harden data and getting audiences to accept their methods shows, on the one hand, the weakness of rankers’ expertise, but on the other, a real engagement and influence in the higher education community. Here I have shown how the THE aimed to shore up its own datasets as well as obtain an active, trust-building engagement with its audience to make good on its ambitious claim of being the ‘global authority on higher education’ (Baty 2015). The need to make this claim is a sign of both weakness and expertise.