Introduction

Scientometric studies often focus on highly cited papers in a scientific field and scientists recognized as research stars. At the root of this research is a theory of citation and approach to measuring citations as symbols (Small 1978) or proxies of quality and intellectual influence (Cole and Cole 1967, 1972; Merton 1957, 1977, 1996; Zuckerman 1977, 1987; Soler 2007). For some time, this theory has been discussed and intensely debated (Moravcsik and Murugesan 1975; Porter 1977; Chubin and Garfield 1980; Cronin 1984; Case and Higgins 2000; also see Moed 2005). Merton’s (1957, 1977, 1996) original view of citations focused on the universal scientific norm of peer recognition (1957, 1977, 1996); however Gilbert (1977) rejected this notion by suggesting that works receive citations because they are more or less authoritative and persuasive. Zuckerman (1977) agreed with Merton’s position by stating that they are useful for measuring “the impact of research on subsequent scientific development” (p. 37); however, Cronin (1984) was decidedly sceptical of the normative view; arguing that the citation process requires further insight into the psychology of the scientist and interactions between institutional norms and the personal considerations of the scholar.

In the field of mathematics, a theory of citations requires special consideration because it is an area of science that tends to be “more highly individualistic or ‘anomic’ than other areas” (see Stern 1978, p. 132; Hagstrom 1965). Consider the use of citations in mathematics to measure the Journal Impact Factor (JIF). In many scientific fields, for example, the biomedical sciences, it makes sense to measure the JIF based on a 2 year time frame, because “most published articles receive most of their citations soon after publication…. Examining a collection of more than 3 million recent citations in mathematics journals (the Math Reviews Citation database) one sees that roughly 90% of citations to a journal fall outside this 2-year window” (International Mathematical Union 2008, p. 9). Rousseau (1988) compared mathematics journals to some life sciences journals and found that “pure mathematics journals would benefit from the use of 4-year impact factors” (p. 249). Korevaar and Moed (1996) carried out further tests with the JIF and found little difference in results for the field of mathematics when citations were calculated using short and long-term citation windows. For a 5-year, 9-year and 13-year citation window “the mean Journal Citation Score JCSm–mean Field Citation Score FCSm ratio” is “significantly higher for top journals compared to very good journals” (p. 124). Also, when experts were asked to assess both the top journals and top publication in mathematics, the results corresponded very well with the bibliometric indicators (i.e., field normalized IF of the journal).

Stern’s (1978) work is of particular relevance. Stern indicates that “it is by no means clear that citations have any validity in [the field of mathematics]” and conducts a study to show why they might still be used as a crude measure of quality (p. 129). In the introduction to her research, she raises a number of interesting points. First she suggests that “it may be that the most prestigious mathematicians are not, in general the most cited ones” (p. 129). She adds that specific works in mathematics “are considered really significant because they solve some heretofore unresolved problems—thus eliminating controversy (and even discussion) and reducing the number of mathematicians working in that area” (p. 132). A reference is given to Fisher’s (1967) study of The Last Invariant Theorists and David Hilbert’s work in Invariant Theory to illustrate the fact that a spectacular mathematical result can inhibit new work and lead an area to “a kind of ‘terminus’” (p. 132).

Mathematical specialties are so highly differentiated, and contain so few mathematicians, that if every paper published in a particular specialty in a given year were to cite a specific work, it would still receive a relatively small number of citations. Moreover, mathematicians often evaluate each others’ work on the basis of its aesthetic quality or ‘elegance’…If style and form are aspects of ‘quality’ in [some] mathematics [papers that] do not earn high citations, then citation counts would not reliably reflect mathematical achievement (p. 132).

Stern (1978) conveys many valuable insights, yet when she formally measures citation counts in mathematics, she finds that they ARE in fact a suitable measure of quality (though perhaps not the only one). The productivity and citation counts for 60 active ‘elite’ mathematicians (i.e., those with an honorary election to the US National Academy of Sciences) were compared to similar counts for a random sample of mathematicians on the faculty of PhD-granting institutions. Results showed that the mathematical elite tended to be cited more than twice as often as their university counterparts, even if they had produced less.

In this paper we will examine the Mathematical Review system (i.e., editorial reviews) to determine if review work is related to how well a journal article is cited within a 5 year citation window. By this we mean that if an article is published and reviewed in 1991 we count the number of citations it has received from the period of 1991 to 1995. In the following paragraphs we describe the role of the reviewer and the importance of status in mathematics research, and then introduce our hypothesis concerning the status of the reviewer and its potential impact.

The role of the reviewer is voluntary and best described in terms of cognitive support work. When a mathematician is asked by a Mathematical Reviews editor to review a paper, and voluntarily accepts (note: a refusal is possible), it means that he has acquired the respect of his peers, and has the reputation of being careful, reliable and knowledgeable in the specialized area to which the paper belongs (D. Trotman, personal communication, November 3, 2006). The purpose of a review is not to indicate whether or not a paper should be published (i.e., the referee system takes care of this), but to “serve researchers in the mathematical sciences by providing timely information on articles, books and other published material that contain new contributions” (see American Mathematical Society 2009) A signed review illustrates the value of a newly published article so that individuals may decide whether or not to read the original item, and perhaps cite the work in new research. In essence, a mathematics review functions as a type of endorsement. The Mathematical Reviews guide states that reviewers “may include a positive or negative evaluation of the item. Reviewers [are also] encouraged to include references to closely related work” (see American Mathematical Society 2009).

In all areas of scientific achievement, status and recognition are important. The reward system promotes a stratified vision of science, and many scientists, including mathematicians, look to this system while producing and promoting their research (see Hagstrom 1965; Merton 1968; Storer 1966). It is a sign of status in mathematics when a researcher is honoured with a prestigious award (e.g., The Fields Medal; American Mathematical Society Prizes), invited to become a member of the National Academy of Sciences (U.S.), and recognized by name due to a significant theorem—e.g., Milnor’s Fibration Theorem. We also know that excellent or high status mathematicians tend to receive more citations than their average counterparts (Stern 1978). Based on this knowledge, we assume that a reviewer of high status (i.e., strong publication record and highly-cited) will play a positive role in how well a new article is received and cited by others. The theoretical implication is that the cognitive authority of a reviewer can influence a mathematician’s motivation to cite, and that the review system is more than just a basic information system if it contributes in part to the mathematician’s reward system.

Bibliometric analyses of the reviewer’s role

Here we focus on a research community in mathematics known as Singularity Theory. This specialty, as it is known today, is largely based on the significant contributions of John Milnor, Vladimir Arnold, Heisuke Hironaka, René Thom and Stanislaw Lojasiewicz during the 1960s and 1970s (Trotman 1999). An in-depth analysis of the intellectual structure and social process of this specialty was previously carried out, based on a complementary set of bibliometric and qualitative research techniques. The results illustrated that this community functions like an invisible college (see Zuccala 2004, 2006). This specialty/invisible college also occupies a distinct set of codes in the 1991 and 2000 American Mathematical Society (AMS) classification system—e.g., Singularities (32Sxx; 1991-now) and Theory of Singularities and Catastrophe Theory (58Kxx, 2000-now).

With the MathSci ® index, we retrieved journal publication counts for a core selection of 85 mathematicians in Singularity Theory [Query: S AU = du Plessis, A? AND DT = Journal]. The Dialog RANK command associated with reviewer names (RANK RE) was then used to identify all mathematicians who have written signed reviews for a particular author. Table 1 presents a ranked list of mathematicians who have reviewed journal articles for A. DUPLESSIS. The highlighted names in the list are others from his specialty.

Table 1 Ranked reviewers for A. du Plessis (MathSci ®)
Full size table

A specific dataset was created in order to compare all journal publication counts with journal review counts for the 85 mathematicians. The dataset included the following: name, total publications, journal publications, journal reviews written, and journal reviews written for invisible college members. Figure 1 shows that journal reviews amongst the Singularity Theorists (n = 2002) have been less frequent than journal publications (n = 3593), yet some of the authors have reviewed as many journal articles as they have published (e.g., BRIESKORN, FUKADA, FUKUI, TROTMAN); while others have published less, and contributed more to the scholarly communication system as reviewers (e.g., CHILLINGWORTH, GIBLIN, STEVENS, WILSON, WAHL).

Fig. 1
figure 1

Journal publication counts compared to journal reviews in Singularity Theory (1974–2007). Authors ranked by total publication count (MathSci ®)

Full size image

A directed review matrix (UCINET 6; Borgatti et al. 2002) and network map (NetDraw 2.043; Borgatti 2002) was also constructed to determine which of the Singularity Theorists have contributed most to their specialty as reviewers (see Fig. 2). GIBLIN, DIMCA, STEVENS, WAHL and TIBAR have made the most review contributions in the past 33 years.

Fig. 2
figure 2

Principle components network map showing prominent reviewers in Singularity Theory marked by the larger circles “○” (MathSci ®)

Full size image

Although Fig. 2 confirms that many of the Singularity Theorists have been active reviewers, only a small percentage of the reviews for this specialty have been written by mathematicians from within the invisible college. Approximately 86% of the specialty’s publications have been reviewed by mathematicians from other research areas. The mathematicians in Singularity Theory who write reviews for each other also tend to cite each other regularly (see Zuccala and van den Besselaar 2007).

To determine the relationship between the reviewer of a journal article and a new article’s ‘citedness’, we sorted the 85 Singularity Theorists by journal review counts and retrieved the top 32 mathematicians who have written the most reviews for colleagues up to and including the year 2007: STEVENS—52; GIBLIN—46; JANECZKO—29; WILSON—29; WAHL—28; DIMCA—22; HAMM—27, TIBAR—16; RUAS—16; SHUSTIN—14; FUKUI—14; ROMERO-FUSTER—14; TROTMAN—13; FUKUDA—13; SEDYKH—12; ZAKALYUKIN—12; IZUMIYA—10; PAWLUCKI—10; COMTE—10; HOUSTON—10; YOMDIN—8; BRASSELET—8; ORRO—8; ARTAL-BARTOLO—7; PORTEOUS—7; TEISSIER—7; MILMAN—7; CHILLINGWORTH—6; DAMON—6; AROCA—5; LEDUNGTRANG—5; GIBSON—5.

We then used the MathSci ® database to identify 369 journal articles that were reviewed by the 32 mathematicians (named above) between the years of 1974 and 2003. A second dataset was created with the following information:

  • Journal article title (MathSci ®)

  • Journal name (MathSci ®)

  • Author name(s) (MathSci ®)

  • Date of publication and review (MathSci ®)

  • Reviewer’s name (MathSci ®)

  • Reviewer’s cumulative publication count up to review date (MathSci ®)

  • Reviewer’s cumulative citation count up to review date (SciSearch ®)

  • Citations to journal article 4 years after review, excluding author self-cites (SciSearch ®)

Each article’s citation count was obtained by typing the following query in SciSearch ®: S CR = DAMON J, 1991? AND PY ≤ 1995. The Cited Reference tag (CR) retrieved all cited articles published by DAMON in 1991. The Publication Year (PY) limited the results to all citing articles up to and including the year 1995 (i.e., 5 year citation window). With the final set of records (S1 = 11 citing articles in total) we examined all of the reference lists and matched the appropriate citation with the article in question. For instance, in 1991 J. DAMON published “On the number of branches for real and complex weighted homogeneous curve singularities” in Topology. An International Journal of Mathematics (reviewed by T. FUKUI). A cited reference to this article [CR = DAMON J, 1991, V30, P565, TOPOLOGY] was found in three different papers; once in 1992, once in 1994 and once by J. DAMON himself in 1991.

With a fully tabulated dataset, we categorized all reviewers according to their cumulative publication status (see Table 4, Appendix). A reviewer in Singularity Theory was classified as a junior if his/her publication count (in the same field or other cognate fields) was less than 10, a member of the specialty if his/her publication count was between 10 and 30, and a senior if he/she already had more than 30 publications. Juniors by definition are students or postdoctoral fellows at the early stage of their careers; while members and seniors generally have a higher position in their research affiliates (e.g., Associate and Full Professorships). From the period of 1974 to 2003 more members agreed to review papers (n = 186), followed by juniors (n = 92) and then seniors (n = 91). An analysis of citation rates from three separate periods demonstrates the changing composition of Singularity Theory and changing role of the reviewer relative to Mullins’ (1973) four stage model.

During the “normal” to “network stage” of the specialty (1973–1985), more junior researchers reviewed well-cited articles (see Fig. 3). During the latter “cluster” to “specialty” stage (1986–1995 and 1995–2003), many of the juniors became more successful; hence the regular members (former juniors) and seniors (former members) were responsible for reviewing the well-cited articles (see Figs. 4, 5).

Fig. 3
figure 3

Citations (1973–1985) associated with junior, member, senior reviewers

Full size image
Fig. 4
figure 4

Citations (1986–1995) associated with junior, member, senior reviewers

Full size image
Fig. 5
figure 5

Citations (1996–2003) associated with junior, member, senior reviewers

Full size image

The scatterplots shown in Figs. 6 and 7 compare the status of each reviewer (first by cumulative publication count, then by cumulative citation count) with the number of citations the reviewed article received within 5 years. Since the data for the scatterplots differ strongly in terms of their distribution, we see a kind of “stacking” effect along the x axes. There was no significant correlation between a reviewer’s status by citation count and number of citations to the journal article; but an unexpected small, but negative correlation (Pearson’s r = −.118; significant at the 0.05 level; 1-tailed), was found between a reviewer’s status by publication count and the number of citations to the reviewed article.

Fig. 6
figure 6

Reviewer’s cumulative publication count at time of review compared to number of citations the reviewed journal article received within 5 years (MathSci ®)

Full size image
Fig. 7
figure 7

Reviewer’s cumulative citation count at time of review compared to number of citations the reviewed journal article received within 5 years (MathSci ®).3

Full size image

The citation counts observed for the 369 journal articles in Singularity Theory (1974–2003) averaged at around 1; ranging from 0 to 12 cites within a 5 year period. Thirty-seven percent of the citations received by each journal article were author self-citations, but all self-citations were excluded from the analyses (note: self-citation means that a cited and citing paper has one author in common). Any article is likely to be cited on its own merit, due to outstanding work done by the author(s), thus it is critical to note that we do not account for this.

Qualitative analyses of the reviews

Given the nature of our bibliometric results, a follow-up analysis was included to examine the written reviews of a selection of highly cited and poorly cited articles. Our objective was to determine whether or not the descriptive language of the reviewer may have played a role in an article’s citedness. It was neither practical nor feasible to carry out an analysis of all 369 reviews; thus we collected a random sample of 20 articles (not reading the reviews beforehand), based on citation counts only—i.e., ten that received 6–8 citations within 5 years and ten that received 0 or less than 3 citations within 5 years (see Tables 2, 3).

Table 2 Reviewed articles (>5 citations) and reviewer’s identifiers or phrases
Full size table
Table 3 Reviewed articles (<3 citations) and reviewer’s identifiers or phrases
Full size table

For each article’s written review we looked for qualitative identifiers, or phrases, other than the description of mathematical proofs, which would convey the reviewer’s positive endorsement of the author’s finding. In most instances the language of the reviewer was factual, explaining simply what the author had shown, or what the result adds to previous work on the subject. Standard phrases such as “the author shows…” or “the author proves…” were the most prevalent; however, in a few of the highly cited papers, phrases such as “this well-written and interesting paper” or “the strength of this result” provide some indication of how a reviewer positive endorsement (see Table 2). Amongst the articles that were poorly cited or not cited at all within 5 years, the reviewer’s language was similar (see Table 3). Again, we observed standard neutral phrases to describe the published article, such as “the main result” or “in this paper it is proven that” or “the authors are concerned with”. Another observation: The average word count for the reviews associated with the highly cited papers was 281 and for the poorly or non-cited papers the average word count for a review was 164. Further notes regarding our observations appear in the concluding discussion of this research.

Concluding discussion

In this study of the Mathematical Review system we tested that a high-status reviewer (strong publication record and highly cited) would relate positively to how a newly reviewed paper is received and cited by others. We found no correlation between reviewer status by citation count and citations to a reviewed article 0–5 years after publication date, and a significant though very low negative correlation between reviewer status by publication count and citations to a reviewed article, 0–5 years after publication date. We will reflect on the reasons for this, but comment first on our general finding pertaining to the reviewer’s role.

Figure 1 demonstrates that a mathematician will act as a reviewer at different stages in his/her career. Junior researchers write reviews (e.g., COMTE; ORRO) as well as members with stronger publication profiles (e.g., GIBLIN; DIMCA). There is a small group of well-published senior researchers who do not, or at least have not done this type of cognitive support work (e.g., LOOIJENGA, PHAM, GORYUNOV), including an elite group of high status mathematicians clustered at the top of the publication rank (e.g., ARNOLD, WALL, MILNOR). Earlier we discussed a variety issues associated with review-writing including the idea that every mathematician will develop his/her own role-oriented profile in a speciality community (see Zuccala and van den Besselaar 2007).

Though we expected to find a positive relationship between high status reviewers and citations to a reviewed article, we are now left to consider why a number of lower-status mathematicians have written editorial reviews for Singularity Theory papers that have been well-cited. Perhaps these mathematicians were asked to review articles associated with ‘trendy’, or more accessible parts of mathematics—i.e., eloquent proofs that provide others in the community with expedient ideas for further research. By comparison, the senior mathematicians of much higher status may have only been asked to review exceptional papers of greater difficulty, and such works might not have been as easily understood and cited.

We also consider the stage of the specialty when the journal articles was first published, reviewed and cited. Singularity Theory was ‘blossoming’ in the early 1970s and 1980s with many interesting problems to solve. Perhaps a number of junior or lower status mathematicians were asked to review papers, because the specialty was new, or at its cluster stage. According to Mullins (1973) the cluster stage begins when students start to gather around key research figures, as well as new and exciting intellectual products. We examined the data used to construct Figs. 3, 4, and 5, and found that from the period of 1973 to 1983 more lower status mathematicians (juniors) wrote editorial reviews, and many of these papers from that time (n = 40) received higher citation counts (i.e., 2–5 citations) than articles published some years later when Singularity Theory was a more established specialty (note: from 1996 to 2003, 31 articles received 2–5 citations).

Earlier we described the informational function the Mathematical Review system and suggested that reviewers from a specialty, who endorse the work of others in that specialty, lend both their status and cognitive support. Our bibliometric data does not provide sufficient evidence to suggest that the status of a reviewer is strongly linked to citer motivation; thus we considered it important to examine the qualitative nature of some reviews. Upon reading a sample of signed reviews (n = 20) we found that the reviewer’s approach in many cases was to describe the mathematical work in factual terms. Nevertheless, we found a small difference in that the reviewers had used slightly more positive words to endorse the work of articles that were well-cited. Our analysis was not comprehensive in that we did not assess all 369 of the mathematical reviews; thus we cannot make substantial conclusions about this general finding. Instead, we would like to use our observations to suggest that further research pertaining to review writing and the length of reviews (i.e., word count) has potential to shed more light on particular reviewing factors that may influence the citation process.

Our bibliometric examination of the Mathematical Review system shows us that the citation process is more likely to rest with the interplay between the scholar’s perception of the work, that is, its aesthetic and cognitive quality, and norms within a specialty (as per Cronin’s 1984 theory), than it does with the authority of a signed editorial review (as per Gilbert’s 1977 theory of persuasion). We also believe that citations are somehow motivated by the stage of growth that a research area is experiencing, and the level of “trendiness” associated with the area. If a spectacular result in mathematics can lead to a specialty’s ‘terminus’, as it did with the Last Invariant Theorists, is it not just as likely to lead to an explosion of opportunities, as we have seen in Singularity Theory? Opportunities or research ‘trends’ can be psychologically motivating; hence the author agrees with Cronin’s (1984) view of the citation process, which depends on a certain degree of psychological insight. A theory of citation in mathematics requires insight into what constitutes typical activity surrounding a new specialty, and how mathematicians make decisions regarding their course of research.

This study opens the door for comparisons to other fields of science and scholarship. Is the Mathematical Review system unique or is it similar to review work in other fields of science, social sciences or humanities? Newly published books written by historians, for instance, are often reviewed. Is it more critical for an author of a text published in Irish history to be reviewed by a well-known or lesser-known scholar from the same research field than it is for a Singularity Theorist to be reviewed by a higher or lower status colleague? Little is known about the effect that editorial review writing (i.e., Mathematical Reviews; book reviews) has on the scientific communication system. We suggest that it deserves more attention, knowing that reviews deliver timely information to scholars on a regular basis and occupy a unique position between author and published text, leading to the citation process.