An analysis of the titles of papers submitted to the UK REF in 2014: authors, disciplines, and stylistic details

Authors of an academic journal paper tend to choose a title with a view to maximising the impact of that paper, particularly perhaps in terms of citations. Hartley (2008) lists thirteen different types of title, including those that state findings and those that ‘bid for attention’. There are two factors to consider here, the information content of the title, and the extent to which it attracts academic attention. For example, longer titles may increase information content up to a certain point but reduce the interest generated. Use of colons may facilitate the latter, particularly if part of the split title is focused on generating interest. It may also facilitate a title which is more informative, or enables the authors to strike a better balance between the twin goals of the title. But in the process, the title tends to become both longer and more complex (van Wesel et al. 2014) and in some, but not all, circumstances this may lead authors to shorten the title and in the process reduce its readability and attractiveness.¹ Ball (2009) argues that question marks can be used to both inform and stimulate interest. However they set a certain style to the title and in this way may also constrain it. For example we later provide evidence that question mark based titles are significantly longer than other titles. However, a further factor complicates this decision and that relates to when there is more than one author to the paper. In that case the title may be a compromise between what the various authors would wish, and may be sub-optimal to all of their preferences.

In this paper we examine the impact of multiple authorship on various aspects of the title and thence their impact on citations. There has been some work on different aspects of the title including the impact of multiple authorship, but this has tended to be on particular academic disciplines or groups of disciplines, rather than across all disciplines as a whole. Our paper looks at journal papers across disciplines making use of a unique data base from the UK. Being as the UK has academics from all over the world who work with other academics from across the world, it does not reflect the practices of UK academics as such, but relates to papers most of whose authors work in the UK. Apart from exploring aspects such as title length and the use of colons and question marks, we will also be implicitly examining one further aspect of academic work, i.e. the amount of time it takes to complete a paper. We cannot do this directly in this research with the data base we have, although in future research that may be possible. Instead we will be analysing the issue indirectly via an observable variable which, given certain assumptions, is linked to research time.

These papers allow us to evaluate the working habits of academics in different disciplines, with the analysis focusing on the best work in these disciplines because this work was chosen to submit to the REF. In doing this we shall be analysing the average number of authors, the length of titles, the use of colons and question marks, and the number of citations. We will also examine the impact of multiple authorship on the working style of academics and the number of citations each paper gets. Being as our analysis is across disciplines, it helps explain some of the contradictory conclusions previously reported in the literature which tended to be focused on a limited number of disciplines. Finally, the analysis will shed light on the weaknesses of the REF in evaluating academic quality within and between disciplines.

The paper proceeds as follows. In the next section we shall review the relevant literature. We then discuss the data and how they were collected and analysed. Surprisingly perhaps, this was not totally straightforward. We then present the results, both summary data and regression results. Much of the work in this area tends to simply report, for example, average citations or incidence of papers with and without colons. But this might lead to a spurious correlation if title length, or the number of authors, impacts on citations, and use of colons is correlated with either of those two variables. In order to unwind multiple impacts, one needs multiple regression analysis. This we use both to analyse (i) the impact of author numbers on title length, use of colons and question marks and (ii) the impact of these title characteristics on subsequent citations. We will also analyse the impact of multiple authorship on when the paper was published within the context of the REF cycle. Finally, we conclude the paper.

There has been substantial work done on the characteristics of journal titles in different disciplines. Titles serve at least two purposes (Lewison and Hartley 2005). Firstly they need to attract readers. Secondly they need to inform the reader about the paper’s contents. The structure of the title can do this in several ways. Lewison and Hartley argued that a two part structure using a colon both increases the number of words and increases the information content. There are substantial disciplinary differences in the use of colons. For example, usage tends to be more common in psychology (50 %) than computer science (7 %) (Lewison and Hartley 2005). In contrast, they concluded that authors rarely used a question mark in their titles. However, Ball (2009) found the use of a question mark in a title to have increased over time and to be higher in medicine (approx. 5 %) than life sciences (approx. 2.3 %) which in turn was higher than physics (approx. 0.57 %).³ Ball goes on to argue that the use of question marks can be explained as they both help to structure the title in a way as to provide informational content, but they also may stimulate interest in the paper, by provoking a potential reader. He also argues that they may also facilitate the rapid publication of results about which the authors are not fully confident (Ball 2009, p. 677). In addition there is a trend for longer titles (White and Hernandez 1991). There is also evidence to link the structure and style of the title to the number of authors, which in itself has tended to increase over time (Hudson 1996). Nagano (2015) argues that title lengths differ between disciplines, due in part to custom. She also finds titles tend to be longer in what she terms the ‘hard sciences’ such as medicine than the ‘soft sciences’ such as sociology. Finally, there is evidence of a positive relationship between the number of authors and the length of a paper’s title (White 1991), with this being more common among science, than social science or humanities, journals (Yitzhaki 1994). Lewison and Hartley (2005), as well as Hartley (2007), also find that in most disciplines single authors more commonly use colons in the title than is the case with multiple authors. However, when the number of co-authors is high this result tends to be reversed. This is in part consistent with the need for multiple authors to negotiate differences of opinion to a consensus (Johnson 1998). Johnson goes on to comment that co-authorship involves disinvesting in preferred ways of writing and this relates to language and not just content.

In persuading academics to read a paper, devices such as informative titles, question marks and colons may also stimulate impact as, for example, measured by the number of citations (Haslam et al. 2008). However, the relevant empirical work is not conclusive. Haslam et al., themselves found no link between ‘the catchiness’ of a title and citations in social and personality psychology, a similar finding to Hartley (2007) more generally. Indeed Hartley went on to comment that it would be startling if something so simple as a colon in a title led to repeated citations. Nonetheless, in a regression analysis Haslam et al. (2008) did find that a colon had a positive effect on citations, and title length had a small negative effect. Jacques and Sebire (2010) found a positive correlation equal to 0.62 which was significant at the 1 % level, between the number of citations and the length of the title. The presence of a colon also increased citations. However seeming to contradict this, Jamali and Nikzad (2011) detected a negative, although insignificant, correlation between citations and title length. But articles with a colon in the title received significantly, at the 5 % level of significance, fewer citations in six PLoS journals. Van Wesel et al. (2014) find citations to decline with title length and increase with the number of authors in a range of disciplines. They also increase with the use of a colon in applied physics, decline in general and internal medicine and to have no impact in sociology. To some extent these differences between different studies may be explained by differences between disciplines.

It has also been argued (Glanzel and Thijs 2004) that citations increase with the number of authors in a number of disciplines, e.g. evolutionary psychology (Webster et al. 2009), biology and biochemistry, chemistry, mathematics and physics (Vieira and Gomes 2010) and information science and technology (Levitt and Thelwall 2009). However in clinical psychology, educational measurement, and management science, Smart and Bayer (1986) found little effect. Such an effect, if it exists, might be expected because multiple authorship may improve paper quality (Haslam et al. 2008). It may also expand the network of the authors (Frenken et al. 2005). There is also a greater potential for self-citation.

There is evidence that coauthorship increases the time it takes to write a paper, Hollis (2001) emphasising the increased amount of time spent on redrafting the paper into a mutually acceptable form. In addition, using panel data on 339 economists he finds a negative relationship between research output and teamwork. Bidault and Hildebrand (2014) also suggest that the costs may increase as the size of the team increases. This is consistent with a long established literature which argues for the decreasing returns to teamwork in general (Hackman 1990). Katz and Martin (1997) focus specifically on co-authorship and the extent to which it coincides with collaboration. They discuss both the costs and the benefits as they apply across disciplines in general. In terms of the former they first mention time costs, i.e. time spent in preparing a research proposal, carrying out the research and writing up the results. Time costs may be greater when travel is involved. With respect to writing up costs, it is emphasised that there may be disagreements over the results and their significance. Specifically they comment that “Differences of opinion are almost inevitable and time will be needed to resolve these amicably. Writing up results jointly may also take more time where there are disagreements over the findings and their significance, or over who should be included among the co-authors and in what order they should be listed” (p. 15). Given this, it seems likely that there will also be disagreement, and hence the need for compromise, about other aspects of the paper including the title. Katz and Martin further argue that these costs are likely to increase with the number of people involved. Cummings and Kiesler (2005) analyse the coordination costs with respect to researchers in multi-university collaborations, mainly in the sciences and engineering but also psychology and mathematics. They conclude that the more universities involved in a collaboration, the fewer will be the number of coordination activities and project outcomes.

There is also a literature on the impact exercises like the REF and the RAE have on academics, their mode of working and their publications (Butler and Spoelstra 2014). Much of this results from institutional pressure to be enterable. Thus Hodder and Hodder (2010), in a New Zealand context, note that in the run up to the end of the relevant assessment period, 2006, publications increased and then declined in the years immediately following. There was also a tendency towards more co-authorship. Harley (2000), in the context of the RAE, argues that the need to produce papers sufficient to count within the relevant time period may have a negative impact on academic work. This is intuitively plausible and consistent with economic theory which would suggest that the REF in changing the incentive structure facing individuals will also change their behaviour. With no RAE/REF constraints we assume the academic would seek to maximise the quality and impact of their publications. As is generally the case, they will tend to prefer publications now rather than later, i.e. future publications are discounted. Whether to publish a piece of work now or in the future is then a trade-off between the discount factor and potential improvements which could be made to the publication by devoting more time to it. The requirement by the REF that the number of publications in each round must be no less than four, constrains the optimisation problem. For academics who are close to the deadline and need additional publications, the Lagrangean multiplier pertaining to the current REF period constraint will be positive and this may lead to publications being brought forward. For other academics who already have the required number of papers, this Lagrangean multiplier will be zero as the constraint is nonbinding, but there may be value in delaying publication in order to help meet the constraints of the next REF. There is evidence for both of these possibilities in addition to the observations made by Harley (2000). Firstly, for example, with people rushing the publication of edited books in order to fit within the (RAE) timeframe (Thomas 2011) and trying to persuade journals to publish their paper rapidly (Wellington 2003). Secondly, the author of this paper has direct knowledge of academics who have delayed publication with a view to enhancing their future value in the academic labour market. We also have the example of a young researcher at Warwick who gives the following advice “if you know you won’t be submitted for the REF 2014, then it might be worth holding off from publishing that 4* piece of work until after November 2013!”⁴ In other words delay publication until the next REF period.

The impact of colons and question marks on citations is less certain. As discussed in the introduction, on the one hand they may increase the attraction factor in particular, but on the other hand, in constraining the title to a certain style, they may at times reduce the informational content. We anticipate that all of the above characteristics and impact will be reflected in the REF and hence can be tested with the REF data base. However, the extent to which they differ between different disciplines will also help inform us about those disciplines, and in some cases the problems of evaluating research across disciplines using the REF methodology. We return to these points in the conclusion.

We downloaded the data on REF submissions which were in Excel format. In all this gave rise to some 190, 962 outputs to analyse⁵ 155,552 of these were papers in journals. The data was then loaded into Stata where it was analysed. There were 26,719 books or book chapters and the remaining were neither published books nor papers. Books were more common in Panel D submissions, e.g. History. In this paper we focus solely on journal papers. Journal papers were identified as such by their having an ISSN number. The citations data was provided only to a limited number of sub-panels assessing the UoAs. It was obtained from the Scopus database. A final ‘snapshot’ of citation counts was taken shortly after the REF submissions deadline. These were then provided to the relevant sub-panels for use in the assessment.⁶ Title length was based on character length including spaces between words, colons, question marks, etc. An alternative is to use the number of words. In reality there is relatively little difference in the results when we use either of these options and, as would be expected, the correlation between the two is high at 0.965. In the work which follows the use of characters tended to yield slightly more significant results than words, and for this reason we choose to base the measurement of title length on characters. However throughout we indicate when the results significantly differ between these two options.

There are substantial differences between disciplines as can be seen from Table 2. We report the panel medians, rather than the means, as the latter are influenced by outliers of individual UoAs. Thus for Panel A, the title length of 109.85 is the median value of the six UoA means. The journal paper was very much the norm in health and the sciences, with over 99 % of papers in both panels being journal papers. In contrast, journal papers are less common in the social sciences and very much less common in the arts and humanities. The most parsimonious UoAs with respect to title length were philosophy, economics and classics. The least parsimonious were public health, clinical medicine and agriculture. In some cases titles played the role of mini abstracts. Not all disciplines reported citations. Of those that did the most were in clinical medicine, physics and public health. The number of authors differed substantially between disciplines. The fewest tended to be in the arts and humanities (Panel D) and the most in health and the sciences (Panels A and B). The figures are startling and the average for physics of 131.2 is an extreme, heavily affected by one or two outliers. But even so, there are still substantial differences if we take the 75th percentile. Differences between disciplines in the use of colons were also substantial. Clearly academics are not an homogenous group, at least based on this data.

There were also substantial differences in the average publication dates of the papers. The latest were in philosophy, music and classics. The earliest were for physics, earth systems and chemistry. The pattern of dates over time tended to be an inverted U shaped for the sciences and health (Panels A and B) as can be seen from Fig. 1. For the social sciences (Panel C) it peaked in 2012 and for the arts and humanities (Panel D) it carried on increasing throughout the period. The inverted U shaped patterns are suggestive that the REF and other similar such exercises are impacting on the timing of research and research publications, if not we would expect publication dates to be uniformly distributed and most importantly not to be lower in the initial and final years. It is consistent with researchers tending to plan new papers at the beginning of a REF period, with the normal research and publication lags leading to publications 2 or 3 years into the REF cycle. Once researchers having got their four good papers they might tend to reduce their research intensity, or at least delay publication to fit in with the next REF round, as suggested earlier would be the case. That this does not happen in the arts and humanities and social sciences is perhaps suggestive of longer publication lags.

Open image in new window

Colons were least common in mathematics, electrical engineering, biological sciences and aeronautical engineering, but could be found in abundance in english language, media studies and social work. There is a similar pattern for question marks. They are virtually absent in Engineering and also Mathematical Sciences, but are particularly prominent in Panel C relating primarily to the social sciences. This is particularly the case for politics, law and social work. We also noted in the introduction that question marks may constrain the style of the title. In support of this, we find that question marks tend to lengthen titles. In 15 of the UoAs, particularly in Panels C and D, the title length was significantly longer with a question mark than without at the 1 % level of significance, and in 22 UoAs at the 5 % level. This was the case whether title length was based on the number of words or characters. In no UoA was the title significantly shorter at the 1 % level and just once at the 5 % level.

The regressions which follow focus on individual UoAs, firstly with respect to the impact of the number of authors on a variety of dependent variables relating to the title and the year of publication, and secondly the impact of the title characteristics on subsequent citations. The results from Table 4 suggest that an increasing number of authors impacts on all the dependent variables, although in different ways. In these further regressions we just include the log of authors as this simplifies the analysis and focuses attention on the impact of this single variable and how it differs between UoAs. We log the number of authors as the impact of this variable is unlikely to be linear in any of the regressions. The regression with the length of title as the dependent variable was done by ordinary least squares (OLS). That for the year of publication by ordered probit, and the use of colons and question marks was analysed by binomial probit. In all cases the standard errors have been corrected for heteroscedasticity.