Abstract
Punctuated by joy, disappointments, and conflicts, research evaluation constitutes an intense, emotional moment in scientific life. Yet reviewers and research institutions often expect evaluations to be conducted objectively and dispassionately. Inspired by the scholarship describing the role of emotions in scientific practices, we argue instead, that reviewers actively define, display and manage their emotions in response to the structural organization of research evaluation. Our article examines reviewing practices used in the European Research Council’s (ERC) Starting and Consolidator grants and in the Marie Skłodowska-Curie Action’s (MSCA) Individual Fellowships. These two European funding mechanisms offer different perspectives on the organization of grant evaluation. We conducted interviews with review panel members and analyzed various institutional documents. By drawing on the sociological concepts of feeling rules and emotional work, we demonstrate that reviewers define rules concerning how emotions should be experienced and expressed to ensure the proper functioning of evaluation, and that reviewers experience the need to actively regulate their emotions to comply with these rules. We present four feeling rules concerning the experience and expression of: (1) excitement for novelty during individual evaluation; (2) respect for others’ opinions and the absence of anger in review panels; (3) attentiveness and interest, which are seen as missing in online evaluations. Reviewers also expect ERC candidates to (4) avoid pride and manifest modesty during interviews. These rules demonstrate that proposal peer review is governed by emotional norms, and show the influence of organizational settings and moral requirements on research evaluation.
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
Introduction
All of us have been involved in, or have at least heard about, situations of evaluation that could be considered “offensive, unfair or disrespectful” (Bloch 2016). Research evaluation is regularly marked by conflicts reflecting paradigm battles, methodological disagreements, and group rivalries. In addition, all of us have certainly experienced the striking moment of excitement, enthusiasm, and possibly envy, during the assessment of innovative research. Research evaluation is also fueled by the interest in unprecedented conceptual, methodological, and empirical developments. In other words, peer review is an intense, emotional moment of scientific life. Yet, reviewers and research institutions often expect evaluations to be conducted objectively and dispassionately (ERC 2019). How could their demands for “dispassionate criteria” of evaluation (ERC 2019) be compatible with the highly emotional outcomes of research evaluation?
An increasing number of works have demonstrated that emotions play a central role in scientific practices: the pleasure of field work and the emotional relationships with the objects of study energize the sometimes frustrating daily work of researchers; the excitement about new scientific ideas drives the development of research fields; and the management of pride and anxiety enables researchers to handle the asymmetric distribution of power in academia (Parker and Hackett 2014; Bloch 2016; Brunet et al. 2019). Authors have shown that peer review, like any other scientific practice, is imbued with various emotions. Whether in the role of assessors or assessed individuals, researchers experience a range of emotions, whose central place has been described in the contexts of research funding, applications for academic positions and publication of articles (Lamont 2009; Bloch 2016; Lorenz-Meyer 2018). Instead of considering that emotions disrupt the claimed objectivity and impartiality of evaluation, these works suggest that emotions enable reviewers to form their judgments, stay committed in the evaluation process, and regulate their exchanges with their peers.
Dating at least to the 17th century with the establishment of the first scientific societies, peer review is expected to provide a system of self-governance to the scientific community (Musselin 2013). In particular, previous research has shown that reviewers create and disseminate norms of academic excellence, and are expected to follow a set of evaluation norms to guarantee the legitimacy of their decisions (Merton 1973; Lamont 2009; Derrick 2018). One important and so far under-analyzed aspect of peer review concerns the role played by emotions in the implementation of evaluation norms. Indeed, the sociology of emotions has demonstrated that norms are profoundly emotional and often cannot be applied without eliciting emotions. Sociologist Arlie Hochschild (1983) proposed the concept of feeling rules to designate the often implicit norms defining how emotions should be expressed and experienced in different situations. In scientific practices, the study of feeling rules remains largely preliminary and challenges the idea that emotions should be excluded from scientific work to be impartial and objective. Some authors have showed that the norms of appropriate emotional expression vary across disciplines (Koppman et al. 2015), or that the respect for positive emotional norms enables specific scientific results to be made useful to decision-makers (Brunet 2022). Despite the preliminary analysis of the role of emotions in peer-review and the importance of emotions to apply norms, very few works have analyzed how the evaluation norms of peer-review function emotionally, and how feeling rules govern research evaluation for high-level funding.
This article combines the sociological study of emotions, Science & Technology Studies, and valuation studies to analyze which emotions reviewers articulate in relation to peer review practices in specific settings, which emotions they think they should be expressing, and how emotions that are out of place should be managed. We identify a number of feeling rules that are essential to the functioning of peer review. In particular, we examine how these feeling rules matter in four different evaluative settings: when reviewers assess proposals individually, when they are involved in in-person evaluation panels, when they write their evaluations in online asynchronous discussions, and when they evaluate applicants in interviews. To do so, we conducted interviews with reviewers from two primary European research funding institutions: the European Research Council (ERC) and the Individual Fellowships of the Marie Skłodowska-Curie Actions (MSCA). We mostly focus on various evaluative situations of the ERC, but also examine the specific online evaluation setting of MSCA. Our analysis contributes to previous studies on the role of emotions in peer review by showing that emotions are not straight-forwardly experienced, but are actively regulated to respond to specific feeling rules, and that the regulation of emotions constitutes an essential aspect of peer review.
Our article is structured as follows: after discussing the role of norms in peer review, we present the concepts of feeling rules and emotional work. We then introduce the conducted study and our method of analysis before describing the feeling rules involved in four evaluation situations. Reviewers define and apply successively: a rule to regulate excitement when assessing excellent proposals individually; a rule which prohibits anger and supports respect for other reviewers’ opinions in review panels; and a rule which promotes attentiveness and dissuades disinterest, which is seen as difficult to apply in online evaluations. Panel reviewers also formulate a rule intended to ERC candidates, which prohibits pride and encourages modesty during interviews. To comply with these rules, we show that reviewers must regulate their emotions intensely and can find themselves in contradictory emotional situations. We conclude our paper by discussing the consequences of examining feeling rules in grant funding, in relation to the organizational setting and to moral requirements of research evaluation. Importantly, our results demonstrate that emotions and emotional work, an essential but often neglected and silenced aspect of peer review, shape various evaluation practices of reviewers. These results invite research funding agencies and other institutions to consider emotions more thoroughly when developing peer review settings and processes and to explicitly address possible emotional challenges in their practices.
Literature Review: From Cultural Norms to Feeling Rules of Peer Review
Peer Review and Evaluation Norms
Perhaps no other scientific practice is as normative as peer review. By defining what is good research and exercising judgment according to established criteria, reviewers participate in creating and disseminating evaluation norms (Lamont 2009; Musselin 2013). Previous authors have argued that reviewers are “normative agents” (Lamont 2009) who enact “socialized norms of academic excellence” (Derrick 2018). Ideally, the ability of reviewers to apply their own, intra-scientific criteria of evaluation is supposed to guarantee the autonomy of research from non-scientific actors (Musselin 2013).
While reviewers diffuse norms of scientific quality, they are also expected to follow a set of procedural norms. Merton (1973) famously argued that science is regulated by four social norms defining how scientists should behave concerning the public character of scientific knowledge (communism); the evaluation of knowledge claims independent from race, class, gender, religion, or nationality (universalism); the absence of interest in personal gain (disinterestedness); and the rigorous scrutiny used to evaluate knowledge claims (organized skepticism). In Merton’s view, reviewers are required to evaluate research by debating its credibility (organized skepticism) without being influenced by personal preferences and benefits (disinterestedness) or by other non-scientific criteria (universalism). These four norms are often mentioned explicitly in the reviewers’ guidelines provided by funding institutions (see ERC 2019; MSCA 2018). When these norms are applied, the review process is expected to be fair and legitimate for applicants. Conversely, when they are not respected, the review process is seen as ‘biased’ (ibid.), and deviant reviewers can be excluded from future reviewing activities (ERC 2019).
However, the idea that universal norms regulate the functioning of science and peer-review processes has been broadly criticized. Since the early 80s, ethnographic laboratory studies have shown that scientists do not necessarily comply with these norms in practice (Knorr-Cetina 1999). In peer review, scientists have been found to be influenced by non-scientific criteria such as gender and to rely on idiosyncratic criteria of evaluation (Lee et al. 2013; van den Besselaar et al. 2018). For instance, reviewers have their own preferences depending on their “personal biographies, individual interpretations of specialized knowledge of their subject, the social network, and the local conditions under which they work” (Gläser and Laudel 2007). From that perspective, the norms of peer-review are not universal but historically and socially situated.
Lamont (2009) specifically exemplifies that peer-review is regulated by situated “intersubjective conventions and criteria of evaluation”, instead of pre-established, impersonal, and universal Mertonian norms. Drawing on interviews with more than 90 members of US funding institutions’ panels, she demonstrates the existence of a set of common “customary rules” shared across disciplinary panels that govern what is considered appropriate evaluation behavior (see also Chong 2013). For instance, reviewers respect other disciplines’ sovereignty by deferring expertise to qualified colleagues; they vote strategically by giving lower grades to some proposals in order to increase the odds for their own favorite proposals; and they support other reviewers’ preferences in the hope of future reciprocity. The application of these rules is expected to facilitate agreements in a limited period of time and avoid conflict while maintaining a shared belief of fairness among reviewers.
In the case of the European Research Council, Luukkonen (2012) conducted interviews with panel reviewers to understand how ERC evaluation criteria are interpreted and applied in practice. She identifies a set of “rules of interpretation” used by reviewers to turn abstract evaluation criteria recommended by the ERC (e.g., excellence, groundbreaking and risky research) into assessable categories. In particular, she reports that ERC reviewers assess feasibility and the “high-risk, high gain” of applications (ERC 2019) by evaluating, e.g., the capabilities of the applicants based on their CVs, the continuity of the proposal with applicants’ previous research, the planned experiments, and the mastery of experimental instruments.
Inspired by the previous scholarship on informal rules, our article proposes to find out how rules of evaluation do not only concern the interpretation of evaluation criteria (Luukkonen 2012) or the strategy of evaluative actions (Lamont 2009), but also involve another essential and normalized aspect of scientific life (Hochschild 2003): the emotions of reviewers and applicants. Indeed, emotions often remain silenced in peer review because of the persisting belief that emotions challenge the presumed objectivity and impartiality of research evaluation (Lamont 2009). Yet, as in any areas of scientific life (Parker and Hackett 2014), emotions are central to evaluation practices and are particularly important for the implementation of evaluation norms. The sociology of emotions has already demonstrated that norms are profoundly emotional and often cannot be applied without generating emotions (Hochschild 1983; Brunet 2021). In the case of research funding organizations, even the widely – but perhaps wrongly – consecrated Mertonian norm of disinterestedness suggests that the norms of peer review are essentially emotional. Disinterestedness, far from being straightforwardly achieved, often requires emotional management to give the impression of emotional detachment (Hochschild 2003). Understanding how norms can function emotionally is therefore essential to the study of research evaluation.
The Emotional Norms of Peer Review
Various science studies scholars have examined how emotional norms regulate scientific practices. Already Merton (1973) suggested that the four universal norms govern scientific activity through emotions. He encapsulated his idea in the concept of scientific ethos: “the affectively toned complex of values and norms which is held to be binding on the man of science” (1973: 268). Mitroff (1974) contrasted Merton’s idea by demonstrating the existence of counter-norms among Apollo moon scientists operating emotionally, such as a norm of emotional commitment opposing the norm of disinterestedness. Yet, the conceptual issues related to universal Mertonian norms are best resolved by the concept of moral economy proposed by Daston (1995). Moral economies designate affective complexes of norms and values governing scientific activities, which are, however, not fixed over time but “historically created, modified, and destroyed” (Daston 1995). Daston presents three moral economies which dominated scientific practices in 17th-century Europe: quantification relying on impersonality, integrity, diligence, fastidiousness, thoroughness, and caution; empiricism mobilizing trust, academic civility, and curiosity; and objectivity demanding self-control and detachment while rejecting pride and vanity. In this conception, norms are emotionally saturated and show how the desirability of some actions guide the behavior of individuals. The concept of moral economy highlights the existence of emotional, historically-situated rules of judgment and has been used to describe contemporary scientific practices, including in the case of article peer review (Lorenz-Meyer 2018).
Emotional norms can be further conceptualized by drawing on the work of sociologist of emotions Arlie Hochschild (1983). Hochschild describes how individuals actively regulate, manage, interpret, and reconstruct their emotions in various aspects of social life: “We do not simply feel, we think about our feelings, both individually and collectively” (1983). Instead of mere reactions to events and situations, she shows that emotions are mediated by norms and reflections over these norms. Hochschild coins the concept of feeling rules, which designates: “What we imagine we should and shouldn’t feel and would like to feel over a range of circumstances.” Feeling rules are “rules governing how we see situations” (Hochschild 2003) that tell us what, when, where, and how long to feel, and can be understood as specific types of emotional scripts. For instance, one usually is expected to show sadness during a funeral, and those who do not comply with this feeling rule may be judged for expressing inappropriate emotions. Applied to practices of peer review, feeling rules enable to understand how reviewers regulate their evaluative actions by shedding light on the emotions they consider appropriate to express and experience in specific situations.
Unlike Mertonian norms, which claim to be universal, feeling rules are, indeed, situated and can be more or less internalized depending on gender, profession, and other factors. For instance, Hochschild (1983) describes how women are preferentially hired as flight attendants over men because they are expected to better maintain a friendly atmosphere on planes and manage the anger of unsatisfied passengers. Furthermore, feeling rules are not applied similarly in different contexts; and what may seem an appropriate emotional expression in one situation, such as the workplace, may not necessarily be seen as acceptable in another context, such as at home. In peer review, feeling rules can therefore depend on the organization, structural support and moderation of evaluative settings. Additionally, feeling rules are expected to regulate interactions between people to ensure their proper unfolding. To comply with these rules, individuals conduct emotional work by adapting, controlling, or suppressing specific emotions. “Feeling rules are what guide emotion work by establishing the sense of entitlement or obligation that governs emotional exchanges” (Hochschild 1983). Inspired by a Goffmanian approach, the concept of feeling rules assumes that individuals try to obey the norms which social circumstances demand, and can therefore be considered in evaluative situations as an emotional type of the customary rules described by Lamont (2009).
Despite the clear role of the management of emotions for upholding normative conduct in peer review settings, there have been no studies on the role of feeling rules in peer review. Only a few authors have examined the management of emotions, but they have mostly focused on researchers instead of reviewers. In the case of a Czech bioscience laboratory, Lorenz-Meyer (2018) describes how researchers are energized by the hope of future publications and by joy of receiving positive reviews. She argues that these emotions help researchers handle moments of anger and exhaustion generated by negative reviews and a productivity-focused academic culture. Bloch (2016) explicitly addresses the subject of emotional norms in peer review and describes how emotions are managed to respond to competition and to the asymmetric distribution of power in academia. Drawing on interviews with various members of Danish academia, she describes how researchers feel anger when their work is criticized and may be tempted to respond with “destructive criticism” favoring their own interests. However, assistant professors, she argues, must maintain good relationships with their colleagues who may act as potential assessors (referring to the feeling rule of friendliness), and manage their anger after receiving negative reviews. Bloch (2016) also shows that feeling rules are gendered and explains that, more than men, women researchers respect another rule prohibiting the expression of pride from positive evaluations. Yet, the previous authors remain mostly focused on researchers receiving reviews and do not further explore how reviewers manage their emotions and enact feeling rules in evaluation.
Other works have analyzed the emotions of reviewers themselves, initiating a research agenda that could be further advanced by focusing on the feeling rules of peer review. In the editorial committee of a German journal of sociology, Hirschauer (2010) describes how editors can feel sympathies and antipathies for certain schools of thought and research topics. Lamont (2009) complements these findings by studying how members of evaluation panels for research funding conduct emotional work to respect each other and define customary rules to accept the diversity of conceptual and methodological interests. In the case of journals’ obituaries, an original form of postmortem peer review, Hamann (2016) demonstrates the existence of customary rules, which regulate the narrative consecration of researchers. He shows that authors value specific character traits, such as “modesty, discipline, and determination,” and omit others such as the psychological challenges faced by researchers during their professional lives. In another context than academic work, Rivera (2015) explains that evaluators for elite US firms use interviews to assess how candidates make them feel, and want to be excited by successful candidates. Our article contributes to these important works by focusing explicitly on the role of emotions in academic peer review and relating these emotions to broader mechanisms of emotional regulation. By studying conjointly various emotions and diverse evaluation settings, we describe how different feeling rules operate across review settings. These different rules neither generate the same emotional experience, nor require the same type of emotional work. To our knowledge, our article is the first dedicated study of feeling rules in peer review that examines how feeling rules support normative conduct in academic evaluation activities.
Methods and Analysis
Presentation of the Case Studies
To understand how peer review is regulated by feeling rules in high-level funding, we studied peer review in two important research funding programs in Europe: the ERC and the MSCA. Instead of systematically comparing the evaluations from both organizations, we explored how peer review can be organized in different settings, which involve diverse emotional processes.
Since 2007, the ERC has been funding scientists who conduct their research in the European Union for up to 5 years, with an annual budget of 2 billion euros in 2019 and a success rate of 10 to 15% for the proposals submitted. This article focuses on the grants designed for individual researchers at two different career stages: the Starting Grant (2 to 7 years after PhD) and the Consolidator Grant (7 to 12 years after PhD). Highly prestigious, these 1.5 to 2.5 million euro grants play a key role in advancing researchers’ careers. In both grants, the review process is divided into 27 thematic, interdisciplinary panels, each composed of two teams of 11 to 18 internationally recognized scientists, who alternate every year and who can be panel members up to four times, conferring panels a semi-stability over time. Panel reviewers are relatively free in their evaluations. They assess the “excellence” of proposals and the biographic trajectories of the applicants in two stages, regardless of applicants’ disciplines or nationalities (ERC 2019). Each reviewer individually assesses 30 to 50 proposals composed of CVs, a list of publications, and a five-page research project presentation. Next, reviewers collectively discuss the submissions at the ERC headquarters. In this second stage of evaluation, reviewers receive the complete versions of only 20 to 30 pre-selected proposals and consult external reviewers for assistance in their decisions. Afterward, the panel convenes in Brussels to interview the pre-selected applicants and rank the proposals for funding recommendations.
The MSCA fellowship has supported the training and mobility of researchers within and beyond Europe since 1996. It has an annual budget of around 1 billion euros, representing over 10% of the European funding Framework Programs. Among the different actions, we focus on the Individual Fellowships designed for individual research projects of 1 to 3 years, often conducted as postdoctoral research and awarded up to 8 years after the PhD. Since 2007, the Individual Fellowships have been increasingly popular among researchers, and proposal submissions have more than tripled, with a success rate of around 15%. Unlike for ERC, the funding of Individual Fellowships covers only the salary of one researcher, plus additional experimental and traveling costs. The evaluation process is divided into eight thematic, interdisciplinary panels covering a large scope of topics. As in the ERC, reviewers are expected to evaluate the excellence of the research projects and applicants, as well as project feasibility and the projects’ expected impacts on society. The evaluation is organized into two main stages. In the first stage, three external reviewers evaluate around 15 proposals individually without disclosing their identities. After the individual evaluations, the reviewers discuss their assessments and write a consensus report, reflecting their different views. In recent years, the consensus process has shifted from in-person to online asynchronous discussions. Only the chairs and vice-chairs of the thematic evaluation panels meet in Brussels to verify the quality of the evaluation reports and rank the proposals based on their scores.
We use the following abbreviations to refer to broad evaluation panels in the ERC: Physical Sciences and Engineering (PE), Life Sciences (LS), and Social Sciences and Humanities (SH). In the MSCA, we use: Information Science and Engineering (ENG) and Life Sciences (LIF).
A Method to Study Emotions in Reviewing Activities
In this article, we draw on interviews conducted with ERC panel members and MC reviewers, and on the analysis of different institutional documents. Studying peer review presents a methodological challenge because peer review often cannot be observed directly (Lamont 2009). Peer-review panels are confidential, and portions of reviewers’ work are conducted individually, in the privacy of researchers’ offices (Gläser and Laudel 2007; Brunet and Müller 2022). Most of the previous studies on peer review have therefore relied on interviews to describe how reviewers assess proposals (Lamont 2009; Derrick 2018; Kaltenbrunner and de Rijcke 2019; Musselin 2013). In our work specifically, we use ‘reflexive peer-to-peer interviews,’ a version of the semi-structured active interview, which considers that reflexivity is not only possessed by the social scientists conducting the inquiry but also by the interviewees discussing their working practices (Müller and Kenney 2014; Fochler et al. 2016).
Interviews offer a particularly appropriate medium in which to reconstruct feeling rules and related emotional regulation. First, interviews enable reviewers to reflexively analyze their emotional understanding and approach to their work, and to describe reflections and expectations about what they had to perform (see also Lamont and Swidler 2014; Müller 2014). Second, interviews allow examining more invisible, private, and long-lasting emotions, especially those regulated by feeling rules (Hochschild 2003), which might not be accessible through direct observations focused on visible, public and immediate emotions (Garforth 2012). Instead of examining the reactive, spontaneous and physical dimension of emotions, interviews enable us to gain a more nuanced understanding of how emotions are defined, displayed, and managed in peer review (Hochschild 1983; Lamont and Swidler 2014; Flam and Kleres 2015; Bloch 2016). Criticizing the laboratory studies of the 1980s, and their maxim stating that we should study “what scientists do and not what they say,” Garforth (2012) proposes to understand scientific practices beyond a sole focus on “doing and movement on one hand, or in terms of objects, visual representations and instruments on the other.” Instead, she shows that other things matter which are not visible practices, and that researchers resist the observational gaze by refusing to be watched constantly and by working privately (see also Lamont and Swidler 2014). This consideration is especially relevant for the study of feeling rules because feeling rules conceptualize emotions as actively regulated instead of passive, triggered reactions (Bloch 2016; Hochschild 1983). In peer review in particular, emotions are not always observable in the moment, but are mediated; may take a while to develop; and can be strongly regulated in their expression, especially to comply with feeling rules. In response to such challenges, interviews offer a method enabling interviewed reviewers to express emotional reflexivity and reflect on the intersubjective interpretation of one’s own and others’ emotions.
In our material, we reconstructed the feeling rules and related emotional work demanded to reviewers and applicants by drawing on multiple interviewees’ interpretations and analyzing written reviews and various reviewing instructions. We conducted 23 interviews with ERC panel reviewers, and 21 interviews with external reviewers (12) and vice-chairs (9) of the MSCA, often members of a similar panel, to inquire about their working experiences during different stages of the evaluation process (Brunet and Müller 2022). In interviews, we invited reviewers to reflect on their evaluation practices and their evolution over time. We also drew upon an analysis of the emotional contexts of reviewing and examined a set of documents, including instructions that had been provided to reviewers, in order to reconstruct how evaluations were conducted (ERC 2019; MSCA 2018). In the background of this study, we consulted 54 written reviews of ERC proposals, which enabled us to contextualize the feeling rules expressed in interviews at the level of written evaluations, and which are further analyzed in another article in preparation. Although we collected less material from MSCA, our analysis did not aim to systematically compare both review processes but intended to understand the role of emotions in peer review. MSCA enabled to reflect on another modality of peer review in online, asynchronous panels.
Drawing on Grounded Theory, we reconstructed the feeling rules of the peer-review settings analyzed and how these rules framed the reviewers’ experience (Charmaz 2006). After rounds of successive coding, we used feeling rules and emotional work as sensitizing concepts to examine how individuals attempted to make boundaries between what they considered appropriate and non-appropriate emotions (Charmaz 2006). In line with the scholarship on the sociology of emotions, we defined emotions as qualification of affective states, which correspond to existing labels such as excitement, anger, respect, attentiveness and pride (Hochschild 1983). In interviews, we looked for descriptions of emotions considered appropriate and focused on references to emotions as part of judgment criteria in written documents. As Hochschild (1983) explains, feeling rules are recognized when people assess their emotions as well as the emotions displayed by others, and the related sanctions and rewards related to these assessments (see also Lamont 2009). Moving from individual statements to a collective interpretation, we looked for shared understanding, intersections, and contradictions in how these rules were expressed and applied in different sources of our material. In addition, feeling rules were not always explicitly articulated by interviewees. In such instances, we reconstructed them through descriptions of appropriate and inappropriate actions, systematic associations of emotions to specific situations and attempts to regulate them through emotional work, and reported interaction failures due to the absence of some emotions (see also Chong 2013). By combining various sources and methods, we describe how four different moments of evaluation are regulated, and detail successively: how a feeling rule of excitement regulates individual evaluations in both ERC and MSCA; how a feeling rule of respect and absence of anger between reviewers emerges during panel discussions; how a feeling rule of attentiveness and interest is seen as not properly applied in online MSCA discussions; and how a feeling rule of absence of pride is prescribed for applicants during interviews for practical reasons in the ERC. While we consider these rules to be collectively enacted in response to diverse constraints related to peer-review work, we did not conceive of them as static but instead as adapted to settings we studied, and as partially transferable to other review settings.
Results
Individual Evaluation: Getting Excited about Novel Research
Reviewers of both grants expressed feeling pleased to be selected by high-level funding institutions to participate in evaluations, but also bored and annoyed because of the practical conditions of their work. Due to the short turnaround time for proposal reviews, most interviewees explained that they had to conduct their individual assessments concurrently with their professional commitments, often evaluating many proposals in a row outside of their regular working hours. As experienced researchers who have been involved in peer review for several years, our interviewees sometimes viewed proposal evaluations as a repetitive and tedious task and stated that they occasionally felt tired of the process. In addition, many reviewers expressed irritation with badly-written proposals and exaggerated elements─ especially in natural sciences, where sections on social relevance seemed overly standardized, buzzwords were overused, and possible applications of the results could be exaggerated.
To manage their annoyance and boredom, most of the reviewers focused on the energizing excitement from novel ideas that helped them cope with the demands of peer-review work. They described situations where excitement was legitimately experienced and expressed, thereby defining a feeling rule of excitement in individual evaluations. In interviews, reviewers from various fields of study demonstrated the existence of this rule by referring to an indispensable moment of excitement and amazement in their evaluation, either qualified as the ‘wow’ moment of the proposal (ERC1, SH) or described with the interjection ‘wow’ to report their excitement: “Wow, that’s so great!” (ERC2, LS). When reviewers were excited by a proposal, they reported evaluating it more thoroughly and extensively compared to others and could therefore defend it better in panel discussions (see 4.2). The statement of the following panelist shows how excitement can be legitimately experienced by reviewers when they evaluate novel and original ideas.
When I read a proposal, I am attracted when I feel a real sense of enthusiasm about the research subject. The proposal doesn’t read simply like a technical document. With the very best proposals, you literally feel excited about something you’ve never even thought of before, on a subject you know nothing about. You read the proposal, and you think at the end: wow, now this is really interesting and exciting. I would love to see this done (ERC 13, SH).
Here the panelist considers that novel research is a source of excitement, even when the proposal’s topic is outside of his specialty. This statement demonstrates that the experience of excitement is not neutral, but instead positively impacts the outcome of the proposals assessment. Yet, the panelist does not see a contradiction between his excitement and the supposed impartiality of evaluation. The experience of excitement is even considered a driver of research evaluation because it enables panelists to identify good proposals. Similarly, we could infer the feeling rule of excitement from the written reviews consulted in the background of this study, where panelists almost systematically associated the adjective ‘exciting’ with the adjectives ‘innovative,’ ‘novel,’ ‘ambitious,’ and ‘creative’ to qualify the research presented. In the reviews, panelists also recognized excitement as a positive attribute of proposals, and hence demonstrated that it was an emotion formally accepted in evaluation activities.
However, reviewers defined things they should not express excitement about, which demanded that they conduct emotional work to regulate their excitement. Symmetrically to the management of excitement, these reviewers also explained that annoyance had to be managed during their evaluations. They tried to avoid excitement regarding the applicants’ and supervisors’ institutions of origin, especially for applicants from Eastern European countries, and attempted to suppress the annoyance generated by buzzwords and poor English skills, especially in natural sciences. Instead of experiencing an outburst of excitement, these reviewers indicated controlling their excitement and looking for a sustained development of this emotion during the entire evaluation process. Indeed, feeling quickly excited about some proposals contradicted values mentioned and praised in reviewers’ guidelines (ERC 2019), such as fairness toward different candidates, openness to diverse disciplinary perspectives, and a responsibility to conduct an altruistic act for the scientific community. In contrast to the case of excitement, reviewers never presented annoyance as a way to identify bad proposals, but only explained how they regulated it. In the interviews, the experience of annoyance was less legitimately expressed than the experience of excitement, even though reviewers may have experienced both equally. A possible explanation of this difference is that reviewers probably considered that annoyance systematically contradicted values of fairness and openness.
In order to deal with the contradictory emotions experienced by reviewers and the different values praised in research evaluation, reviewers developed two strategies to manage their excitement and annoyance. Some reviewers explained that they had to “suspend judgment” in order to remain “fair” (MC6, ENG; ERC3, SH). Instead of being quickly excited (or annoyed) by some proposals, they had to go beyond their first impressions. Other reviewers reported the need to “balance judgment” (ERC 13, SH). Instead of being quickly excited by some proposals which looked like their own research, or annoyed by disciplinary and methodological differences, these reviewers attempted to go beyond situations that they often qualified as “academic rivalry” and “conflicts of interest” (MC6, ENG). Chairs at both funding institutions expected reviewers to be open to differing works and methodological approaches rather than judging proposals according to their own interests and to the standards of their research fields. When reviewers could not “balance their judgments,” they were invited to abstain from the evaluation. Due to the increasing dependence on external funding, some of these reviewers also explained that they had to behave in a fair and “empathetic” (ERC5, LS) manner to applicants whose careers might depend on their assessments.
The two strategies of suspending and balancing judgments underline the importance of reviewers’ emotional work to align their evaluations with a set of ideal values central to research evaluation. These strategies enable reviewers to comply with the feeling rule of excitement and maintain the legitimacy of the experience of excitement in very specific conditions, despite the fact that excitement may contradict the desired impartiality of research evaluation. Although excitement offers a resource to cope with the workload of proposals evaluation, its experience cannot be felt straightforwardly nor spontaneously but must be actively controlled. Reviewers are allowed to feel excited about novel and original research, but must suppress excitement regarding applicants’ university of origin or a proposal’s similarity with their own research. By contrast, reviewers seem to be much less allowed to be annoyed about badly written proposals, probably because the writing style of research proposals is a less important evaluation criterion than research novelty.
Panel Evaluation: Respecting Each Others’ Evaluations
After individually evaluating proposals, ERC panelists gather in Brussels for in-person panels. Collectively, panelists discuss their evaluations with other panel members, grade proposals, and select which proposals should proceed to the next stage. Contrary to individual evaluations, the purpose of panel evaluations is to compare reviewers’ viewpoints and reach a “consensus on whether the evaluation holds or not” (ERC4, SH). Bringing together panelists with varying preferences is expected to lead to better and more diverse evaluations. Yet, because panelists have different backgrounds, disagreements are frequent and can lead to conflicts.
In reviewing panels, conflicts often occur due to differences in theoretical commitments, methods, and disciplinary standards. For instance, when the ERC was founded, political scientists and anthropologists served on the same panels, but later had to be separated because of persistent conflicts over research methods. Conflicts can also happen if panelists have difficulties changing their opinions when the majority disagree with them, as explained by a reviewer (ERC15, PE) who expressed that panel members could decide to “absolutely love or hate an application.” When conflicts are not appropriately managed, disagreements can develop into intense disputes, which some interviewees qualified as leading to “animosity” (ERC9, LS) and “fist fights” (ERC15, PE) in the panel discussions.
Most interviewees emphasized the panel chairs’ role in avoiding conflict. In their view, panel chairs had to conduct emotional work to set up a cooperative, friendly, and joyful atmosphere between reviewers, also called “keeping the good mood” (ERC18, PE). The panel chairs we interviewed reported using different techniques to achieve this result, such as developing friendships with other panel members, carefully managing specific reviewers who were perceived as having “big egos” (ERC14, PE), de-escalating conflicts as soon as they started, and fostering a “team spirit” between panel members through organizing non-evaluation group activities (e.g., travel, housing, meals) (ERC15, PE). When emotional management was successfully conducted, a friendly and convivial atmosphere reinforced the bonds between panel members, as indicated by some reviewers who described their joy in meeting each other over the years. Yet, the emotional bounds connecting panel members could also create difficulties for new panel members to integrate into a well-formed group.
In addition to the chair’s emotional management, most panelists defined specific feeling rules that enable panels to function properly. According to them, panelists should express respect towards the other panel members and exclude anger from their exchanges. The purpose of this feeling rule was to enable panelists to disclose divergent opinions without creating conflicts. Respect maintained the possibility of co-existence of different regimes of valuation (Fochler et al. 2016) among researchers from various disciplinary perspectives. When panelists disagreed with other panel members, they were encouraged to formulate their disagreement simply, without depreciating the other panelists and expressing anger towards them. Most panelists expected their peers to let everyone speak without monopolizing the floor and to remain open to different ideas and scientific approaches.
To comply with the feeling rule of respect, many panelists reported having conducted emotional work and having used their expression of emotions strategically during panel discussions. For that purpose, they could apply three strategies of emotional management: over-emphasizing specific emotions, contextualizing the emotions expressed by others, and being simultaneously emotionally engaged and disengaged from the proposals. Some panelists anticipated that fewer conflicts would occur if they could over-emphasize excitement or disdain in an attempt to “increase or lower the enthusiasm” of the panel (ERC10, SH), especially when focusing on very specific aspects of the applications (novelty, methods, track record, passage of the proposal, or an applicant’s potential). Indeed, experienced panelists who participated in panels several times argued that proposals needed to be defended to resist criticism and be selected. By preparing their arguments before panel discussions, these panelists attempted to create enthusiasm for specific aspects of the proposals rather than supporting or rejecting the whole application. These panelists used enthusiasm strategically to comply with the feeling rule of respect to convince their peers without creating direct confrontations and conflicts in a situation emphasizing excitement.
However, some panelists also criticized the exaggerated expression of emotions by their peers who were systematically over-enthusiastic or over-disdainful. These panelists highlighted the need to become familiar with other panel members to understand how they expressed their emotions and find out whether they over-emphasized their emotions for strategic purposes. The following panelist described how she tried to get familiar with new reviewers on ERC panels to understand if they were over-disdainful or not.
There are also reviewers who are putting down everyone. You get to know them over time, but with a new panel, you can still be surprised. If the panel has been working for six years, then you already know who is spraying poison (ERC2, LS).
As formulated in the expression ‘spraying poison’ by the former interviewee, panelists used the emotions expressed by other panel members, here excessive disdain, to contextualize judgments and find out who was always highly critical or highly positive. In that way, they could avoid engaging in heated debates possibly leading to conflicts with panel members who were thought to exaggerate their evaluations. By contrast, when panelists already knew other panel members, our interviewees generally agreed that quicker agreements and compromises could be made in the panel evaluation. Yet, a few of these panelists, mostly from natural sciences, considered confrontation helpful in further elaborating their arguments and producing better collective evaluations of the proposals. To engage in confrontation without creating conflicts, these panelists managed their emotions intensively. According to them, they needed to engage in arguments to defend their favorite proposals, but they also had to accept being wrong and backing down from debates quickly, especially when they turned out to be holding a minority position. Hence, they recognized the need to be simultaneously emotionally engaged and disengaged from the proposals. This paradoxical emotional situation could only be achieved through intense emotional work, which demanded the reviewers’ time and energy. At the end of the panel’s evaluation period, panelists often reported preferring to avoid arguments, implying that the order of evaluation could influence the willingness of panelists to challenge majority opinions, and therefore impact how proposals were evaluated (Lee et al. 2013).
The three strategies of emotional management presented in this section offered possibilities for panelists to avoid conflicts, or at least allow them to occur in a controlled manner, complying with the feeling rule of respect and absence of anger between panel members. Hence, emotions were regulated to maintain interactional dynamics between panel members and enable the proper functioning of these panels (see also Lamont 2009).
Asynchronous Online Evaluation: Unfelt Attentiveness and Commitment Needed for Collective Evaluation
Contrasted to the ERC’s in-person panel evaluations, Marie Skłodowska-Curie Actions (MSCA) evaluations are organized online in successive, asynchronous rounds of exchange between reviewers. Online evaluations are less costly to organize and demand less time for reviewers by eliminating the need to travel to one place for a few days. In MSCA, the number of applications for individual fellowships increased dramatically in recent years and reached a total of 10,000 proposals (MC6, ENG). Therefore, contrary to the earlier years of the funding program, MSCA staff members decided to shift the review process online in 2016. Studying MSCA online evaluation practices is timely because many funding organizations shifted to online evaluations during the COVID-19 pandemic. However, while most funding organizations only moved their panel meetings online without changing the evaluation process (e.g., in the ERC), the MSCA evaluation is specific due to its online asynchronicity. In MSCA, three reviewers receive the same proposal, evaluate it individually and submit their evaluations through an online platform. One of the reviewers is then in charge of collecting the reviews, comparing them and writing a ‘conciliation report.’ This report is expected to draw on the three reviews, but should also propose a final decision on the value of the proposal, especially in the case of divergent reviews. After that, the three reviewers discuss the ‘conciliation report’ through asynchronous messages. During this asynchronous discussion, reviewers are expected to confront their evaluations and resolve possible divergences rather than averaging their individual grades (MSCA 2018). At the end of the discussion, the evaluation report is validated by vice-chairs, who supervise the entire evaluation process.
According to MSCA staff members, online asynchronous evaluations provided practical solutions to reviewers and facilitated the evaluation process. Staff members expected reviewers to be able to organize their work with more flexibility and have more time to reflect on evaluations and the other reviewers’ comments. Online reviewing was also supposed to be more inclusive, facilitating the participation of reviewers who were less involved in in-person debates due to weaker rhetorical skills or less familiarity with diverse research fields.
Yet, most of the reviewers interviewed, including but not limited to those who attended former in-person panels, criticized the new peer review process. In this section, we argue that one key issue with online evaluation is that reviewers have difficulties conducting emotional work and complying with the evaluation’s feeling rule of attentiveness and commitment. According to our interviewees, these emotions are missing and create a dissatisfying evaluation experience. By comparing MSCA reviewers’ scores before and after the shift to online evaluation, Pina et al. (2018) concluded that the level of agreement did not change and that “face-to-face consensus meetings do not guarantee a better consensus.” Our results complement this important study by showing, however, that reviewers seem to perceive a difference between both settings, and that other aspects could be analyzed (content of messages, discussion’s duration) to further understand the consequences of asynchronous evaluations.
Indeed, some interviewees who were involved in former in-person MSCA evaluations considered themselves to have been less committed in the online process because of its asynchronous and remote organization. Instead of being able to dedicate their entire days as they had done in Brussels, reviewers who stayed in their countries kept being disturbed and occupied by their daily work. In their view, exchanges between reviewers were scarcer and more superficial because reviewers often lived in different time zones and could not exchange directly, but also because reviewers felt less embarrassed about presenting poorly written reviews to their peers online than in person. A former reviewer who had been involved in the process for many years and had acted as a vice-chair summarized his frustrating experience: “It is as if the system had broken down when it shifted to entirely remote because people just do not feel engaged or invested in the process” (MC5, ENG). For this reviewer, the online asynchronous evaluation could not generate the emotional commitment that he considered essential for evaluating proposals.
Additionally, many reviewers reported that they could not really engage in debates and defend their favorite proposals because the online process prevented a synchronous conversation from taking place. Reviewers had three weeks to discuss their proposals with their peers and mostly exchanged non-synchronous messages a few days apart. Sometimes, when they received the answers to their previous comments, they admitted that they had forgotten the main reason for their disagreement and were not willing to spend time getting back into the proposal. When they disagreed with other reviewers, the online discussion had to be typed, which demanded more time than an oral discussion. Consequently, these reviewers not only felt less involved in the process but also believed that they agreed more easily with their peers in order to avoid confrontations. A reviewer summarized these issues, reporting their difficulties with maintaining attentiveness and conducting the emotional work of disagreeing with other reviewers:
At the end, nobody really fights for their applications, and you tend to agree with the other people’s arguments. Nobody really wants to defend their proposals. It’s difficult to give elaborate and detailed arguments through the internet. It is done by text, but nobody is doing it at the same time. Sometimes you write a response, but the person on the other side is busy with other things and will respond to you only two days later. After a while, you want it to be over. You have lost the dynamic. (MC2, LIF)
Furthermore, a majority of MSCA interviewees recognized that they had to comply with a feeling rule of respect and wanted to reach collective agreements, but they also found conflicts much more difficult to resolve online. In Marie Skłodowska-Curie, vice-chairs were expected to mediate conflicts between reviewers and facilitate the writing of the ‘consensus report.’ Despite their emotional management and attempts to resolve divergent views diplomatically, the vice-chairs interviewed found that debates could escalate rapidly and that disagreements could lead to personal attacks between reviewers. In their experience, reviewers could get suspicious and angry, refuse to log in after an argument, and write insults they would never have said in person.
Two possible explanations can be advanced to understand why conflicts seem to be particularly difficult to control online. First, in contrast to in-person panels where reviewers usually tried to find solutions and preserve each other’s face, emotional work may be more difficult to conduct online. As several reviewers stated, they did not have access to the “variation in tone and speech, body language, and eye contact” (MC6, ENG), which were critical clues to understanding other reviewers’ reactions and regulating the interaction in the moment. Second, asynchronous online interactions may not elicit similar feeling rules and emotional work than in-person interactions, because consensus building processes can be rarer and exchanges can be more contentious on internet. In the absence of professional socialization for online interactions, reviewers may draw on feeling rules derived from other online spaces. They may then have more difficulties applying a feeling rule of respect and leading their interactions toward the resolution of problems, as explained by the following reviewer.
Things escalate very quickly. In the remote situation, they can say: ‘I don’t agree, and I will never agree. I’m not going to continue discussing it,’ and then the process stops. Whereas when everyone is present physically in Brussels, the process doesn’t just stop; it continues until it’s resolved. Online, no one feels invested in the resolution. As a vice-chair, sometimes you will contact an expert and say: ‘this is not appropriate behavior; you need to try to be constructive.’ At that moment, rather than saying: ‘I understand, I will be more reasonable’ - they do what people do on the internet: they turn around, and they accuse the vice-chair of being biased, of trying to pressure them to change their scores. The conflict resolution process is being warped by remote review. We very rarely get to really resolve problems because the people are not around. They are typing into a chatbox. It’s very hard to understand what consensus even means in that context. (MC5, ENG)
The reported difficulty in displaying and producing the emotions needed to foster enough commitment in the evaluation to trigger the proper amount of debates and confront opinions without initiating devastating conflicts (i.e., the feeling rule of respect) highlights a non-production of the emotions required for the evaluation process to feel appropriate. Online, reviewers admitted that they could not feel attentiveness and commitment in the evaluation and experienced distraction and indifference. These un-felt emotions are emotions that reviewers expect to experience in an evaluation, but cannot feel in an asynchronous, online process. In their absence, the perceived failure of the interaction suggests that these emotions are seen as indispensable for the evaluation. This section therefore illustrates how a feeling rule cannot only be understood by explicitly expressed emotions, but also by the emotions which are missing from the situation of evaluation in order for it to function properly.
Interview Evaluation: From Pride to Modesty
In the second round of ERC evaluation, panel reviewers interview about 20 to 30 candidates and select about ten for funding. Candidates usually have 15 minutes to present their proposals and 15 minutes to answer questions. Panelists can ask for clarifications about the proposals and invite applicants to defend their theoretical and methodological choices. The interview plays a crucial role in the evaluation: candidates located at the top of the preliminary ranking can be downgraded and end up without funding, while candidates with the lowest grades before the interview may be funded.
As in the situations of evaluation described previously, panelists reported trying to remain fair and open-minded, but all stated becoming exhausted by 30 successive interviews in two to three days. Indeed, these panelists had to understand and evaluate many proposals that they rarely read entirely and which were often outside of their research fields. In an emotionally challenging atmosphere generated by time constraints and the repetition of standardized talks, some panelists reported being moved by original presentations and being influenced by the presentations’ order. However, panelists did not focus on the role and regulation of excitement to manage their tiredness as in the individual evaluation. Rather, they used the interview as an occasion to assess who the applicants were, and defined a feeling rule concerning how applicants should present themselves.
Many panelists defined a feeling rule of modesty, humility, and absence of anger, overconfidence, and arrogance. They reported that they examined how applicants reacted to criticism during this intensely stressful situation in which only one-third of the applicants would be funded, and their careers could depend on their success. Panelists generally agreed to look for applicants who could respond to criticism well and defend their choices without anger. According to them, applicants had to show determination and command, but should not express arrogance or overconfidence. For instance, a panelist explained that she rejected an applicant who “was very confident,” “couldn’t manage to end his presentation on time,” and “replied very aggressively to some questions” (ERC8, SH). Similarly, many other panelists indicated that complying with the feeling rule of modestyFootnote 1 while being criticized illustrated whether applicants could work in a group, respect power hierarchies among researchers (Hochschild 1983), and respond well to unexpected difficulties in their research projects (see also Jasanoff 2003).
The feeling rule of modesty involves the rejection of the expression of pride, also qualified as the prohibition of “self-eulogy” (ERC2, LS) or as “one should not praise oneself” (ERC18, PE). According to these panelists, the expression of pride reflected values that were not compatible with collective research, such as egoism and narcissism. When the feeling rule was not respected, these reviewers admitted to feeling “irritated” and negatively evaluating applicants who presented themselves as the “best of their generation,” as the only ones in the world to know something, or as publishing an exponential number of articlesFootnote 2 (ERC14, PE; ERC1, SH). The existence of this feeling rule was also confirmed in the written reviews consulted in the background of this study, where panelists could criticize applicants who expressed too much pride in their previous achievements and over-confidence in superior research abilities, as highlighted by the following review written in response to a very confident proposal:
I would have preferred to read more about how they would actually tackle these instrumental setups, which would have given me more confidence that they would be developed in a satisfactory way. Instead, it’s written more as a “trust me, we know what we’re doing” sort of manner. (review ERC, LS)
A few panelists reported stories of applicants who stated that Europe would “lag behind” if they were not selected (ERC18, LS) or who described themselves as the inventor of a new idea by displaying a picture of themselves (ERC1, SH). The emphasis by researchers on self-achievement relates to a neoliberal turn in research policies, where researchers have been incited to manage their careers in a quasi-entrepreneurial manner (Shapin 2009; Müller 2014). Hence, ERC applicants had to conduct careful emotional work to give an ambitious, forward-looking image of themselves and of their research but refrain from expressing pride to show openness and cooperativeness in collective research. The feeling rule of pride reflected the tensions generated by transformations in scientific careers concerning how applicants should present themselves. The regulation of modesty was experienced by panelists as a way to assess the socialization of applicants and their ability to work in groups, as well as to verify their awareness of the unexpected nature of research work.
The feeling rule prohibiting the expression of pride also revealed differences in how gender could be taken into account in ERC evaluations, confirming previous findings on the gendered character of some feeling rules (Hochschild 1983; Bloch 2016). As women could be disadvantaged in accessing scientific careers, the ERC introduced different evaluation recommendations to improve the representation of women among grantees, such as additional application time for women who have children. Before conducting their evaluation, panelists now receive a video sent by the ERC which informs them about ‘unconscious bias’ in the evaluation and warns them that the independence and competence of women are often more contested than those of men (see also van den Besselaar et al. 2018). Different panelists, especially in the natural sciences, considered that these biases could be reinforced by the gendered expression of emotions, where men are expected to express pride and use “self-selling” and “boasting” strategies, while women are supposed to be more modest and to stay more often in the background. When these panelists were not experts in a research field, several of them indicated the temptation to favor male applicants who expressed pride and confidence because their scientific achievements looked more important, as opposed to women who were more modest and careful.
Consequently, many panelists considered that controlling the expression of pride was also a way to make the evaluation of researchers uniform across genders. In response, some panelists indicated negatively evaluating male applicants who did not regulate their expression of pride. For instance, a reviewer (ERC7, SH) explained that he advised his former PhD student, who was “a bit macho,” to be more modest in his presentation. The panelist contrasted the failed presentation of the previous applicant, who did not take into account his advice, with the successful presentation of a woman who expressed confidence and modesty the same day. In some LS and PE panels, panelists even applied the feeling rule of modesty by changing the organization of interviews. To better calibrate their evaluations (see Brunet and Müller 2022), these panelists interviewed all the women before the men. The feeling rule of modesty was therefore not only applied to select specific values among applicants, but was also expected to normalize how applicants presented their achievements across genders.
Conclusion
Peer review has previously been analyzed as a normalized and normalizing activity (Merton 1973; Lamont 2009; Musselin 2013; Brunet and Müller 2022). In particular, Lamont (2009) described how customary rules of evaluation were collectively created and adhered to in panel evaluation while prescribing appropriate behaviors in the evaluation process. Emotions are an important but often neglected aspect of normalization, and are intensely regulated and governed in social life (Hochschild 1983; Bloch 2016; Brunet 2019). In this article, we have extended previous research on emotions in science (Parker and Hackett 2014) and peer-review (Lamont 2009; Bloch 2016; Lorenz-Meyer 2018) by describing how peer review is organized in various situations and governed by different feeling rules. Importantly, the study of feeling rules shows how specific situated emotional norms are internalized in research evaluation and influence the implementation of evaluation criteria.
We have identified four feeling rules which highlight the appropriateness and expectations regarding the experience and expression of emotions in specific evaluative situations. Reviewers are allowed or expected to: 1) feel excited by new ideas to get energized during their individual evaluations and mitigate the boredom and annoyance of this repetitive, time-constrained work, but should not feel excited about the institutions of the applicants and annoyed by the proposals’ writing style; 2) feel respect for other reviewers’ judgments and enable the co-existence of different evaluation regimes without creating destructive conflicts; 3) feel attentive and committed to the evaluation process to maintain a sufficient level of collective engagement and manage possible conflicts. Reviewers also prescribe a feeling rule for applicants who should 4) express modesty and avoid expressing pride to demonstrate different qualities and facilitate evaluation across genders. These rules concern the emotions that reviewers and applicants are allowed to legitimately express and influence the emotions that can be experienced (Hochschild 1983). To respect these rules, reviewers must engage in emotional work and actively regulate their emotions.
Our analysis of feeling rules builds upon and extends the existing scholarship by providing a focused and nuanced picture of the role of emotions in different review settings, showing how these emotions can both facilitate and hinder the practice of peer review. The feeling rule of excitement echoes previous findings on evaluators in charge of hiring employees for elite US firms, who want to feel excited by candidates (Rivera 2015). In research evaluation too, excitement seems to drive the work of reviewers, but contrary to the private sector, the experience of excitement is strictly regulated to maintain fairness towards candidates from Eastern European countries. The feeling rule of respect particularly demonstrates that emotions are not merely the outcome of specific evaluation strategies, as in the case of customary rules (Lamont 2009; Hamann 2016). Evaluation strategies are themselves intrinsically emotional: panel members guarantee the proper functioning of evaluation panels by defining how emotions should be experienced, and deliberately act on their emotions and those of their peers. The feeling rule of attentiveness and commitment complements previous studies on the role of emotions in peer-review (Lamont 2009; Bloch 2016) by demonstrating that evaluation is also conditioned by un-felt emotions, i.e. missing emotional states needed for the evaluation to proceed correctly. Furthermore, by showing that the emotional work conducted to preserve one’s face (Lamont 2009) is also important in online settings, we open an unexplored research agenda on the emotions in online and asynchronous evaluations. Finally, the feeling rule of modesty extends studies conducted in other professional contexts, which demonstrated that evaluators use interviews to assess how candidates make them feel (Rivera 2015). We contribute to these works by showing that, contrary to the private sector, the emotions experienced by panelists during candidate interviews are gendered and their expression can be normalized to create more equity between genders in research funding.
The study of feeling rules in peer-review highlights three consequences of examining the appropriateness of emotions in grant funding evaluation. First, our article demonstrates that reviewers see emotions as indispensable to conduct their work, and that they have to manage them constantly for pragmatic purposes. According to the organizational settings of evaluation, reviewers define and apply different feeling rules which are expected to ensure that they can work properly despite numerous constraints. In the individual ERC evaluation, reviewers cope with the demands of reviewing by getting energized and excited by novel ideas. In the ERC panel evaluation, the feeling rule of respect is seen as responding to researchers’ pragmatic needs to find agreements rapidly with limited time for interdisciplinary exchanges (Müller 2021). By managing their anger, reviewers can confront their often-divergent views of research quality without creating destructive conflicts, and enable evaluation processes in interdisciplinary settings. Peer review is therefore not only a practice generating diverse emotions; instead, the emotions experienced in peer-review are essential to its proper achievement, and this emotional experience has to be strictly managed.
The need to manage emotions and respect feeling rules is a hidden aspect of peer review and could be more explicitly acknowledged by research managers and reviewers because of its potential epistemic implications. Emotional work requires time and energy, and may not be equally performed at different moments of the evaluation or in different settings. When panel discussions reach the end of the allotted time, reviewers may be less inclined to engage in possible conflicts by defending some proposals against other reviewers, and may prefer to be more consensual and accept their peers’ opinions directly. Furthermore, emotional work seems to be more easily conducted in synchronous and in-person evaluation processes than in online and asynchronous evaluations. During the first year of the pandemic, the ERC attempted to organize synchronous online evaluation panels, but ERC officers found that debates were less lively and decided to reorganize in-person panel evaluations as soon as they became possible. The organization of online synchronous evaluations raises a first challenge for reviewers, who may not be able to be equally devoted compared to when they are present in person; but the use of asynchronous online evaluations presents yet another setting and may generate additional challenges for sustaining the commitment of reviewers. In times marked by pandemic and ecological crises, online evaluation may offer answers to acutely pressing problems. At the same time, our results highlight the importance of thinking carefully about how online processes can alter the emotions that reviewers experience as well as their ability and willingness to manage these emotions.
Second, our article shows that the feeling rules of peer-review and the emotional management related to them respond to a contemporary transformation of the emotional culture of academic work. Despite funding institutions’ and reviewers’ efforts to organize research evaluation in the best conditions possible, peer review may be constrained by broader issues, such as productivity and precarity (Müller 2014). In the ERC individual evaluation, novel research may offer reviewers a moment of excitement contrasting with the emotional tiredness fostered by overwork. Furthermore, the feeling rule of modesty and absence of pride reflects the tensions generated by the precarity of scientific careers. Authors have usually discussed modesty as a founding value of modern science associated with the rise of objectivity, where scientists are expected to disappear behind facts (Daston 1995; Shapin 2009). In the ERC interview evaluation, the regulation of the expression of pride is expected to enable assessing researchers’ socialization and making the evaluation of researchers uniform across genders. Yet, the transformation of scientific careers to the model of entrepreneurial research (Müller 2014; Shapin 2009) creates a conflicting situation for applicants who have to remain modest while generating enough excitement for their projects and scientific potential. Surprisingly, this contrasts with the advice given by coaches to applicants that emphasizing research achievements can improve chances of success (see James and Müller in prep.).
Third, our article demonstrates that emotional work enacts the figure of the good reviewer in the moral economy of research evaluation. Indeed, emotions are managed to respond to ethical needs and give the impression of an impersonal and fair evaluation, advancing previous findings (Lamont 2009; Hamann 2016). When reviewers and applicants comply with the previously exposed feeling rules, they show their adhesion to specific values considered necessary for members of the scientific community such as respect, openness, inclusiveness, and originality. Part of these values are explicitly defined in the guidelines provided to reviewers (ERC 2019; MSCA 2018). To make their work comply with such values, reviewers conduct emotional work and attempt to not be excited directly by some applications, or to evaluate equally both male and female applicants by regulating the expression of pride. Peer review can therefore be viewed as a moral activity inscribed in a moral economy, where values are emotionally saturated (Daston 1995).
In contrast to other works which emphasize the value of ‘reciprocity’ between grant reviewers (Lamont 2009), or between journal editors and reviewers (Kaltenbrunner et al. 2022), our article shows that the moral economy of evaluation enacted in the ERC and MSCA mobilizes a diversity of values and emotions. In both grant evaluation settings studied, reviewers have to respect some of these values to be seen as good reviewers by their peers. In particular, the ability to conduct emotional work is seen as essential to the figure of the good reviewer. Reviewers who cannot manage their emotions properly, or who systematically over-exaggerate them, are criticized and discredited by their peers. When reviewers respect the four feeling rules presented, they enact a specific subject, who appears to be a self-motivated researcher, overbooked yet still burning for science, but also a mild, calm and dispassionate researcher, who is able to control his or her emotions in the name of the collective advancement of science. The figure of the good reviewer is therefore a paradoxical subject. It demands researchers to alternate between contradictory emotional states, and to govern themselves in the absence of direct prescriptions of feeling rules, which are rather acquired through the reviewers’ socialization.
To conclude, we invite research funding agencies to acknowledge the central role played by emotions and emotional work in peer review, instead of excluding or downplaying their importance. Emotions motivate reviewers: they help them handling working conditions marked by boredom, tiredness (excitement) and conflicts (respect), and enable them to stay focused in asynchronous and online reviewing processes (attentiveness, commitment). These results contribute to dissipating the common view that emotions should be set aside from peer review, or at least silenced and not discussed, because they disturb scientific judgments and influence research evaluation in detrimental ways. On the contrary, our article shows that reviewers actively define when and how emotions should be experienced and invites to be more aware and open about the role of emotions in research evaluation. Research funding agencies could therefore consider these emotions in their organizational work and carefully analyze when, how, by whom and for what purpose emotions are expressed and managed in evaluation activities. For instance, research funding agencies could change the feeling rule of respect by encouraging a higher degree of conflict in peer review panels and giving each reviewer the possibility to select their favorite proposal, even if the other panelists disagree (see Langfeldt 2001). In addition, reviewers are often only informally socialized into the emotional norms of research evaluation, and they may not all have the same skills to manage their emotions and comply with these norms. During meetings at the beginning of the evaluation process or in written summary documents, research funding agencies could speak more explicitly about the emotions that can be experienced by reviewers and describe how these emotions are usually regulated. Our suggestions could then enable a more thorough consideration of the role emotions in peer review, as essential mechanisms for the formation and regulation of judgments.
Notes
This contrasts with the advice given by coaches to applicants (see James and Müller in prep.).
References
Brunet, Lucas, Isabelle Arpin, and Taru Peltola. 2019. Governing research through affects: The case of ecosystem services science. Science and Public Policy 46(6): 866–875.
Brunet, Lucas. 2022. Transposing emotions to conserve nature? The positive politics of the metrics of ecosystem services. Science as Culture. https://doi.org/10.1080/09505431.2022.2151426
Brunet, Lucas. 2021. #Flygskam: le pouvoir de la honte de prendre l’avion pour gouverner le changement climatique. Lien Social Et Politiques 86: 54–70.
Bloch, Charlotte. 2016. Passion and paranoia: Emotions and the culture of emotion in academia. London: Routledge.
Brunet, Lucas, and Ruth Müller. 2022. Making the cut: How reviewers use evaluation devices to select applications at the European Research Council. Research Evaluation 31(4): 486–497.
Charmaz, Kathy. 2006. Constructing Grounded Theory. A Practical Guide for Qualitative Analysis. London: Sage.
Chong, Phillipa. 2013. Legitimate judgment in art, the scientific world reversed? Maintaining critical distance in evaluation. Social Studies of Science 43(2): 265–281.
Daston, Lorraine. 1995. The moral economy of science. Osiris 10: 2–24.
Derrick, Gemma. 2018. The evaluators’ eye: Impact assessment and academic peer review. Springer.
European Research Council. 2019. Guide for peer reviewers.
Flam, Helena, and Jochen Kleres, eds. 2015. Methods of exploring emotions. London: Routledge.
Fochler, Maximilian, Ulrike Felt, and Ruth Müller. 2016. Unsustainable growth, hyper-competition, and worth in life science research: Narrowing evaluative repertoires in doctoral and postdoctoral scientists’ work and lives. Minerva 54(2): 175–200.
Garforth, Lisa. 2012. In/visibilities of research: Seeing and knowing in STS. Science, Technology, & Human Values 37(2): 264–285.
Gläser, Jochen, and Grit Laudel. 2007. Evaluation without evaluators. In The changing governance of the sciences, eds. Richard Whitley and Jochen Gläser, 127–151. Dordrecht: Springer.
Hamann, Julian. 2016. “Let us salute one of our kind”. How academic obituaries consecrate research biographies. Poetics 56: 1–14.
Hirschauer, Stefan. 2010. Editorial judgments: A praxeology of ‘voting’ in peer review. Social Studies of Science 40(1): 71–103.
Hochschild, Arlie. 1983. The managed heart. University of California Press.
Hochschild, Arlie. 2003. The commercialization of intimate life: Notes from home and work. University of California Press.
James, Mallory, and Ruth Müller. In preparation. “The EU glitter”: Coaching, training, and the professionalization of European research excellence.
Jasanoff, Sheila. 2003. Technologies of humility: Citizen participation in governing science. Minerva 41: 223–244.
Kaltenbrunner, Wolfgang, and Sarah de Rijcke. 2019. Filling in the gaps: The interpretation of curricula vitae in peer review. Social Studies of Science 49(6): 863–883.
Kaltenbrunner, Wolfgang, Kean Birch, and Maria Amuchastegui. 2022. Editorial work and the peer review economy of STS journals. Science, Technology, & Human Values 47(4): 670–697.
Koppman, Sharon, Cindy L. Cain, and Erin Leahey. 2015. The joy of science: Disciplinary diversity in emotional accounts. Science, Technology, & Human Values 40(1): 30–70.
Knorr-Cetina, Karin. 1999. Epistemic cultures: How the sciences make knowledge. Cambridge, MA: Harvard University Press.
Lamont, Michèle, and Ann Swidler. 2014. Methodological pluralism and the possibilities and limits of interviewing. Qualitative Sociology 37(2): 153–171.
Lamont, Michèle. 2009. How professors think. Cambridge, MA: Harvard University Press.
Langfeldt, Liv. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science 31: 820–841.
Latour, Bruno. 1987. Science in action: How to follow scientists and engineers through society. Harvard University Press.
Lee, Carole J., Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology 64(1): 2–17.
Lorenz-Meyer, Dagmar. 2018. The academic productivist regime: Affective dynamics in the moral-political economy of publishing. Science as Culture 27(2): 151–174.
Luukkonen, Terttu. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices. Research Evaluation 21(1): 48–60.
Marie Skłodowska-Curie Actions Individual Fellowships Programme. 2018. Manual for evaluators.
Merton, Robert K. 1973. The sociology of science: Theoretical and empirical investigations. Chicago, IL: University of Chicago Press.
Mitroff, Ian I. 1974. Norms and counter-norms in a select group of the Apollo moon scientists: A case study of the ambivalence of scientists. American Sociological Review 39(4): 579–595.
Müller, Ruth. 2014. Postdoctoral life scientists and supervision work in the contemporary university: A case study of changes in the cultural norms of science. Minerva 52(3): 329–349.
Müller, Ruth. 2021. Time as a judgment device. How time matters when reviewers assess applicants for ERC starting and consolidator grants. In Inquiring into academic timescapes, ed. Filip Vostal. Bingley: Emerald Publishing Limited.
Müller, Ruth, and Martha Kenney. 2014. Agential conversations: Interviewing postdoctoral life scientists and the politics of mundane research practices. Science as Culture 23(4): 537–559.
Musselin, Christine. 2013. How peer review empowers the academic profession and university managers: Changes in relationships between the state, universities and the professoriate. Research Policy 42(5): 1165–1173.
Parker, John N., and Edward J. Hackett. 2014. The sociology of science and emotions. In Handbook of the Sociology of Emotions, vol. II, eds. Jan Stets and Jonathan Turner, 549–572. Dordrecht: Springer.
Pina, David G., Ivan Buljan, Darko Hren, & Ana Marušić. 2021. Meta-research: A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018. Elife 10: e59338.
Rivera, Lauren A. 2015. Go with your gut: Emotion and evaluation in job interviews. American Journal of Sociology 120(5): 1339–1389.
Shapin, Steven. 2009. The scientific life. Chicago, IL: University of Chicago Press.
van den Besselaar, Peter, Hélène Schiffbaenker, Ulf Sandström, & Charlie Mom. 2018. Explaining gender bias in ERC grant selection—Life Sciences case. In STI 2018 Conference Proceedings (pp. 346–352).
Acknowledgments
We are grateful to our interview partners, who gave us their time and shared their experiences with us. We acknowledge and are grateful for the important work of Kay Felder, who conducted many of the interviews which this article is based on. We further want to thank the anonymous reviewers and the editors for their thoughtful remarks and contributions to improving the manuscript.
Funding
Open Access funding enabled and organized by Projekt DEAL. Research for this article was supported by the Gender & Diversity Incentive Fund of the Technical University of Munich. It has been conducted in conversation with and intellectually supported by the Deutsche Forschungsgemeinschaft Research Unit 2448 ‘Practicing Evidence—Evidencing Practice’.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Brunet, L., Müller, R. The Feeling Rules of Peer Review: Defining, Displaying, and Managing Emotions in Evaluation for Research Funding. Minerva 62, 167–192 (2024). https://doi.org/10.1007/s11024-023-09518-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11024-023-09518-8