Data is of primary importance to the expert. It is the basis on which expertise is built (Grek 2013). Rankers conduct experiments on their data and ask academics to evaluate their work. In this way, rankers develop both new kinds of data and new perspectives on their data. This gives them the potential to extend their expertise. While they take into account the feedback they receive, in the end, rankers decide on their own what data to use, which indicators to include, and what material to publish.
However, for the university ranker, it is not enough to have data. The datasets needs to be processed and they need to be re-presented and made relevant to the THE’s various audiences. The process by which the THE treats and gathers its data is the subject of an infographic or special section in every THE rankings issue. There is always some commentary on the THE’s methodology as well as commissioned thought-pieces on the rankings’ data from technical experts. Conversations about the data are present at every THE-organized summit. Here, the THE aims to get its primary audience in the room and have the most direct conversation as possible with them. Baty explains that their events are ‘absolutely for university leaders… mainly presidents and vice presidents, Vice Chancellors of universities, strategic leadership…’ (Baty, personal communication, 12 September 2014).
The development of the THE ranking methodology has always aimed to address concerns among its core audience: academics and university managers. Prior to 2010, the THE and QS produced a single set of university rankings. Critiques of this joint ranking ranged widely, from technical shortcomings (Bougnol and Dulá 2014), to anchoring effects that favored long-established universities (Bowman and Bastedo 2011), to audience mis-targeting (Cremonini et al. 2008), among others. Baty and Anne Mroz, then editor of the THE, felt that if academics and high-level administrators were unhappy with the rankings, this could potentially result in difficulties with the circulation and thus the revenues of the magazine.
In 2009 the THE decided to split from the QS due to the growing critical reaction from its readership. The THE says it held a wide consultation over the production of its new go-it-alone rankings. In creating its own distinct rankings separate from QS, the THE claims it undertook ‘the largest consultation exercise ever undertaken to produce world university rankings’ (Baty 2010a). This consultation involved ‘more than 50 senior figures across every continent’, ‘more than 250 pages of commentary’, and ‘more than 300 postings on our website’ (ibid) from the wider university community. The idea was to present an open and transparent process in order to produce better rankings relative to the earlier joint version. The sense of being open to changes, and to experimenting with new and better ways to measure university excellence, was important. Trying to have a consultation with its core audience was something that the THE felt to be a strategic activity. There were important commercial reasons to develop new datasets and new models distinct from the previous ‘mistrusted’ methodology.
“There’s no question that at the very beginning, the first task was to restore credibility with our core audience who are academics and senior administrators. We had to give them something that they felt more comfortable with. If the Vice Chancellor or President rings Times Higher Education telling you that your ranking’s silly, we have a credibility problem because we need them to trust us and see what we do is responsible. So I think our first audience is definitely senior administration, senior academics.” (Baty, personal communication, 9 December 2014)
In order to distance the new standalone THE ranking from the previous one and show how the new rankings improved upon the perceived flaws of the old, Baty ‘confessed’ to the sins of the past instrument. ‘The rankings of the world’s top universities that my magazine has been publishing for the past six years, and which have attracted enormous global attention, are not good enough.’ (Baty 2010b) Among the deficiencies that the THE highlighted was the large weight of the subjective reputation survey and that the citation metrics as well as other research-oriented metrics were better suited to capture research production in the sciences over the arts and humanities (Mroz 2009).
“Two main flaws in the current rankings were identified: the survey of academic opinion that makes up 40 per cent of the overall score in our rankings was deemed too small - this year it was based on fewer than 4,000 responses from around the world, which when aggregated with previous years’ results produces a total of 9,386; the other concern was about our use of paper citations to measure research quality.” (Mroz 2009)
But what is more important than the technical methodological change is the way that the THE described the process by which they strengthened their rankings. The process of identifying flaws, trying out better metrics by iteration, and appealing to core audiences for legitimacy were actions seeking to refine expert claims. The THE editors highlighted the involvement of other experts in the process. At the point when they separated from QS, they took on Thomson Reuters as their new data partner. Reuters’ data analysts would be brought in to strengthen the instrument. The THE showed that they together with these new experts would be involved in building the THE’s own knowledge base.
“…we will work with our new data partner, Thomson Reuters, to produce a more rigorous and transparent ranking for 2010 and beyond. We will seek further advice from our editorial board and solicit the views of international rankings experts and our readers to develop a suitable methodology for the sector. And this week, Phil Baty, deputy editor of Times Higher Education and editor of the rankings, met experts in Shanghai at the Third International Conference on World-Class Universities to discuss the way forward.” (Mroz 2009)
The changes that the THE introduced were not only defensive changes. They did not only change the indicators that received the most criticism, such as the perceived overweighting of the reputation survey or the skewness of the citation metrics towards the hard sciences. The go-it-alone THE rankings also introduced new indicators, such as an innovation indicator, to reflect the concerns of its audience. The goal was not only to correct flaws but also to increase its own relevance. The experiments around these methodological changes needed to yield results that corresponded to commonsensical outcomes that were worthy of the higher education community’s trust. For instance, there was a common ‘feeling’ that Ivy League universities should end up somewhere close to the top of the table, as this was held to be ‘common sense’.
I suggest that it is vitally important for experts to have robust data. However, this data needs to be made relevant and built into the discussions in communities of practice. Data and relationships are the two resources that rankers jointly use: they involve their audiences in gathering and strengthening the datasets that they gather and produce in order to arrive at products that are seen as robust, reliable, and relevant. One of the ways that rankers develop their data and their relationships is through experimenting with different kinds of indicators as well as representations and tables that their audiences accept and value. University rankers carefully consider what kinds of data and indicators are relevant to higher education stakeholders and enter into a dialogue as to what kinds of data are robust (Baty, personal communication, 12 September 2014).
This kind of gradual ‘assemblage’ of calculative ideas and devices is the foundation on which expertise is built (Tan 2014). University rankers gathered their own datasets and developed different instruments suited to different audiences. The differences among the ranking instruments, such as the different methodologies, aimed to capture different qualities of higher education excellence.
Different rankings are oriented to different audiences over which these rankers want to exercise a certain expertise. The QS rankings, for instance, can generally be described as more (international) student-oriented. The Leiden university rankings are oriented towards readers who are more concerned with research performance measured by bibliometric output. The differences between rankers, their audiences, and the various professionals that support them point to the networked nature of the industry of rankings. In the world of rankings, there are multiple actors, multiple sites, and multiple experts.
However, the general trend that binds the rankers together is that all universities have stakeholders who are interested in the excellent performance of their institutions. Rankings have become the mechanism by which the organizational ritual (Powell and DiMaggio 2012) of excellence is more commonly understood and measured. This, in turn, has led to the establishment of ‘research and/or reputation management’ units within the universities—new institutional forms (Meyer and Rowan 1977; Fligstein 2001) that have spread throughout the sector as legitimate agents of excellence and accountability—that gather data for and deal directly with the ranking agencies.