Background

We live in an evaluative society or “evaluation” society (2011) in which all aspects of our activities are measured and evaluated. The current scientific debate shows a bipolarization between those that compare, discuss and introduce an increasing number of new and more sophisticated bibliometric quantitative methods and those that are sceptical towards metrics and quantitative assessments, due also to all the intrinsic and extrinsic problems they traditionally have (see e.g. the recent book edited by Biagioli and Lippman, 2020). On the one hand, the methodological discussions, based on comparison of quantitative methods, are typically exposed to various objections. One problem is their inability to sufficiently discriminate between researchers who have the same quantitative bibliometric parameters (e.g. the same h-index) but differ in the quality of their research (discriminatory lack). A second problem is that they treat complex dynamics as if they were simple ones (simplifying lack), often on the basis of moral principles that are implicitly assumed to be true without being publicly discussed (lack of moral transparency). A further problem is the proliferation of different types of quantitative assessments without having a criterion and a framework (Daraio, 2017) that enables us, in turn, to assess them (lack of assessment criteria). On the other, those that are critical towards the quantitative assessments do not consider an important question that relates the “good evaluation” of research.

If we face the evaluation problem without prejudices, the question of how to make a good evaluation appears as a very important one. In order to address this question, the fundamental conceptual problem of what research is and what it means to do good research should be addressed firstly. Without having a criterion to guide the selection between different quantitative criteria, any public debate concerning their use and function is often reduced to ideological contrast between those who support the importance of evaluation and those who deny it. Not having an axiological criterion, i.e. the notion of “good” evaluation, the discussion about quantitative methods is often reduced to a sterile and ideological opposition between the fanatics of evaluation and its opponents.

The aim of this paper is to bridge this gap by introducing the notion of good evaluation of research practice. In order to defend this methodological thesis, the paper takes as its starting point a previous and more fundamental question concerning the nature of the object of the evaluation, i.e. scientific research. Using the well-known definition of social practice introduced by the moral philosopher A. MacIntyre, we argue that scientific research is a form of social practice that is defined on the basis of peculiar internal goods, i.e. research objectives and the criteria of excellence that concern them, and of the psychological characteristics of the researchers that make them possible (MacIntyre 1985). Using McIntyre enables us not only to describe but also to broaden the scope of the research evaluation, which does not only coincide with science but covers any kind of research practice.

Our paper uses notions drawn from philosophical ethics following their standard definitions. According to the extensive literature on the subject, we define normative ethics as that part of moral philosophy that formulates and justifies principles of conduct and concepts that are conceptually connected to some notions of moral good. Basic normative principles and concepts govern our self in two ways. They help us (1) to make the right sorts of decisions (practicality requirement), and (2) to form a correct evaluation of other’s behaviour (evaluative requirement). What is then the connection of normative ethics with evaluation? We use normative ethics as a perspective to redefine and clarify the notion of good evaluation of research practice. We try to understand what are the fundamentals underlying the bibliometric evidence and in doing so we propose a paradigm shift, from the methods of evaluation to the content of evaluation.

Following this track, the paper aims to fill a significant methodological gap in the field of the evaluation of research. We argue that the evaluation of research activities, including research projects and programs, together with their outcomes, should not be limited to assess the products or quantitative aspects of the production and dissemination of recorded information, but should also take into account the psychology of the actors involved in this process (authors, readers, etc.), including their motivations.

The paper unfolds as follows. The next section introduces existing research on the topic of the paper. The following section describes the main aim of the work. Method and material are then presented in the following section. After that, an outline of the questionnaire to assess researcher virtues, the main result of our work, is illustrated. Discussions and further developments are then illustrated. The final section concludes the paper.

Existing research

Our proposed description of the object of the evaluation clearly differs from that defended by the sociologist of science Robert Merton (Merton 1973), allowing our approach to be immune from the objections that were addressed to him. According to our proposal, the focus of the evaluation is not on scientific research, understood as a unitary discipline inspired by the exact sciences method, but on different research practices. In this way, not only do we propose to broaden the focus of evaluation to practices that had been excluded so far, but we also intend to make evaluation sensitive to the internal characteristics of each research practice.

From a methodological point of view, our approach aims to use tools and concepts of normative ethics, to clarify the notion of good evaluation and to enable our evaluation model to take into account the motivations of participants in various research practices. Using concepts of moral philosophy to supplement the evaluation of research is not a completely new idea. Furner (2014) argued that evaluative bibliometrics should use concepts of moral philosophy as this way it can develop fair criteria for informing decision making in the distribution of rewards. Furner argued, for example, that it would be fruitful to place quantitative evaluation methods within a broader perspective that, modelled on John Stuart Mill's classic utilitarianism, assesses their impact on society's overall well-being. Resnik (2012), on the other hand, proposed an integrated method that values both ethical principles and the psychological dispositions of researchers who interpret and respect those principles. For Resnik (2012), it is only by attaching value to both aspects that the integrity of research can be promoted.

Before these approaches took off, Robert Merton, in his fundamental work on the sociology of science (Merton, 1973) pointed out that the character traits of researchers, i.e. virtues, are essential input in the process that generates the outputs of research. Merton famously illustrated the link between traits such courage, self-confidence, resilience, taste and the recognition from one’s peers along with the capacity to produce excellent quality research. These studies, however, while showing the importance of the intersection between philosophical ethics and scientometrics, are very partial in their results. Indeed, Merton describes only the peak of scientific research and Resnick is not interested in using that intersection to develop a new model for evaluating research.

Our goal is to orient the pre-existing literature, summarized in Fig. 1, towards an understanding of what a good research practice is considered as a requirement to be able to implement a good research evaluation. Figure 1 shows the main existing contributions that we use to develop our framework (see the building blocks in the bottom panel) and other related works connected to our approach. As will become clear later, one of the strengths of our contribution is that it allows us to shift the focus from the evaluation of the outputs to the evaluation of the process oriented towards the achievement of good practice including the dimension of learning.

Fig. 1
figure 1

Building blocks of our framework and other related literature

Full size image

We follow a different approach than the ethics of evaluation which establishes principles and rules for the evaluator and the evaluation process (e.g. Furner, 2014; Jayawickrama and Strecker 2015; UNEG 2008a and 2008bb; ALLEA 2017; NASEM 2017). We do not deal directly with how research outputs are evaluated by evaluators, but we start from the definition of what research is and what is a relevant unit of analysis to analyze and evaluate it. In our approach, we do not deny the existence of general ethical principles of evaluation and research, but we claim the role of virtues to understand motivations and behaviours, to be able to distinguish good practices from bad ones and make a good evaluation. Furthermore, virtues are also useful for correctly interpreting the very general principle of evaluation (they are complementary).

Approach and aim

Our goal is to understand how a good evaluation can be made. To achieve this, to do this we start by defining research practice, then we move on to that of a good research practice and finally we claim that a good evaluation must take into account the characteristics of good research. The unit of analysis that we propose is that of "research practice" as it seems to us the most appropriate and the one that best fits with the characteristics of the activity carried out by the researchers. We propose a new level of research analysis based on the concept of "research practice" understood as social practice, according to the definition proposed by MacIntyre (1985). This concept allows us, on the one hand, to enter the research activity, going through the different disciplinary areas each of which has its own specificity (Whitley 1984). On the other hand, it allows us to capture what connects all research practices as social practices oriented towards the achievement of their own “internal goods”. It is the universality of virtues for the achievement of the internal goods, the traits of the character and the motivations of the people involved, who share all research practices.

Taking McIntyre’s notions of virtue and social practice as well as Nussbaum’s notions of human capabilities (Nussbaum 2006), we set out an evaluative model that enables us to make the notion of good evaluation of research practice operational. We argue that this central notion can be developed from that of good research practices. This is because a good evaluation takes into account the constitutive elements of a good research practice. So our proposal is to start from a general notion of a “good social practice”. From this, we specify the notion of “good research practice” and from the latter we specify that of “good evaluation” of research practice.

This line of thought involves different moves that can be schematically summarized in three points. First, clarifying the notion of “social practice” explaining what it means to comply with its rules, and which elements of our psychology can account for its emergence. As it will become clear later on, to comply with social practices requires agents to develop specific traits of character which unable agents to grasp, produce and further the “internal goods” of the practices they join. These traits identify those who excel in following the practice. They are exemplary figures that the other participants in the practice want to emulate. Second, examining how the practice affects the life of those who inhabit it. Finally, setting the standards in the light of which assessing the overall effects of practices on society as a whole.

Although our framework partly echoes Merton’s approach of considering virtue as an important input into the process that yields research products, it differs significantly in two crucial points. First, he uses individual qualities to explain and justify differences in capacity to acquire outstanding achievements between future Nobel laureates and average researchers. We instead use virtues to understand the difference between the activity of researchers, whose motivations cannot be described independently of the intrinsic (non-instrumental) desire to acquire the “internal goods” of the practice—e.g. the peculiar pleasure of undertaking new line of research, excellence in analytical skills, a particular taste for problem raising, etc. -, and those who participate in the research practice mainly out of desire to acquire goods external to the practice, e.g. power and wealth. Second, unlike Merton, who merely mentions the excellences in character, we want to present a characterization of virtuous psychological traits that highlights their constitutive role in producing a “good” research practice. We argue that a “good” practice is characterized, among other things, by the fact that its participants have an intrinsic (non-instrumental) interest in seeking the “internal goods” of the practice together with the capacity to grasp and appreciate them. In line with MacIntyre’s approach we shall argue that the possibility of achieving these “goods” depends on whether participants in the practices have, cultivate and teach others certain virtuous character traits. In the following, we will provide a detailed description of what “internal” and “external” goods of the practice are. Figure 2 outlines the main logical steps of our theoretical contribution. To understand 4 (good evaluation of research) we need to clarify the nature of 1 (research), focusing on research practice as level of analysis, then move on to 2 (good research) and identify the features of good research, these features yield normative requirements for an appropriate evaluation (evaluation fitness for purposes of the good research). 3 complement “informed” peer review with a Questionnaire on researcher’s virtues.

Fig. 2
figure 2

Logical steps of our theoretical contribution

Full size image

In order to make the notion of “good evaluation” of research practice available, several more basic notions need to be clarified. These notions are not on the same logical level, but form a chain where the most complex elements follow from the simplest ones. The notions must therefore be explained in a certain logical order.

First, we clarify the notions of “good” and “social practice”. Following McIntyre’s hypothesis, we argue that researchers need to develop specific traits of character which enable them to grasp, produce and further the “internal” and the “external goods” of the practices they join. Those who fully possess these traits are those who excel in following the practice. They are exemplary figures that the other participants in the practice want to emulate.

Second, we explain the notion of “good practice” as one in which a sufficient number of participants have full possession of the virtues. These make available the internal and external goods of the practice and educate the other participants to follow in their footsteps. We argue that the possessing of some virtues is typically incompatible with the possession of others. Therefore, good research practice should be composed of researchers who have different personalities and experiences capable of hosting different groups of virtues. To this end, we propose an organization of the practice that includes three different types of researcher, i.e., leader, good and honest, each of whom is characterized by the possession of certain virtues.

Thirdly, the elements of good practice impose normative constraints on the potential conceptual resources to evaluate the practice: any potential evaluation must be appropriate to the elements of the practice. Finally, in the light of the constraints, we propose what we believe to be appropriate effective tools to develop a good evaluation of research practices.

Method and material

Premise

The achievement of our aim will be possible by an integrated method, which uses philosophical conceptual analysis (MacIntyre 1985; Hursthouse 1999; Nussbaum 2006; Swanton, 2003, 2007; Snow 2010), but integrates it with the empirical method of the social sciences (Resnik, 1998; Walumbwa, 2008; Ferrero and Sison, 2012; Bazeley 2010; Snow 2012). The method of analysis applied in this work consists in the application of argumentative logic, that is, conceptual and philosophical argumentation and clarification. Our study has a mainly theoretical character, aiming to identify a new qualitative dimension in the evaluation of research. We propose to employ primarily philosophical notions to elaborate a new description of the research that makes it suitable to be characterized as good or bad.

The way we use “good” is a technical one, which comes from a philosophical perspective. Following Peter Geach’s analysis (Geach 1956), we argue that “good” is an attributive and not a predicative adjective. Predicative are the adjectives that we can separate from the object to which they are referred and make predicates: when we say that this notebook is blue we can also say that there is something that is a notebook and that thing is blue. The “blue notebook” can be broken into two predications. That doesn’t count for “good”. When we say that this is good practice, we cannot break “good practice” by saying that there is something that is a practice and that is good, because we have no information about goodness that is independent of the fact that it is attributed to practice. The meaning of good depends on the fact that it is attributed to a practice and would have a different meaning if it were referred to a pen. A practice is good in relation to the various functions for which it has been built, such as the ability to produce certain internal and external goods. We argue that “good” is not an all-nothing property but a property that can be owned in degrees. Our model therefore enables us to make comparative judgements between practices, capable of differentiating them on the basis of how good they are.

Along with the good, the notion of practice will be the other key concept employed in our study: we shall examine research qua social practice. We do not intend to argue that this is the only way to characterise research, but we do argue that social practice is a constitutive aspect of the concept of research. As in the case of good, we use practice as a technical term according to the use made of it by the philosopher A. McIntyre in his After Virtue. More precisely, our hypothesis is that a scientific/academic research is a form of social practice a la MacIntyre.

MacIntyre defines social practice “as any coherent and complex form of socially established cooperative human activity through which goods internal to that form of activity are realized in the course of trying to achieve those standards of excellence which are appropriate to, and partially definitive of, that form of activity, with the result that human powers to achieve excellence, and human conceptions of the ends and goods involved, are systematically extended” (MacIntyre 1981 first ed.; 1985: 187). Following MacIntyre’s formulation we define a academic/research-practice as “any coherent and complex form of socially established cooperative human research activity through which its participants, through the exercise of a set of refined human psychological qualities or virtues (called “human powers” or virtues by MacIntyre), contribute to the advancement of the body of knowledge that is constitutive of that practice and commit themselves to promoting any positive effects this has on the well-being of society.

We define good academic practice as one in which these aims are actually realized, having its participants developed the virtues related to the production of those goods and their work being divided according to their skills and experiences. That scientific/academic research can be described according to the McIntyre model is strongly justified by the well-known definition of research practices offered in the Frascati Manual. According to this document (OECD 2015, p. 44) Research and experimental development (R&D) “comprises creative and systematic work undertaken in order to increase the stock of knowledge—including knowledge of humankind, culture and society- and to devise new applications of available knowledge”. For an activity to be an R&D activity, it must satisfy five core criteria. The activity must be: 1. Novel, 2. Creative, 3. Uncertain, 4. Systematic, 5. Transferable and/or reproducible.

The first element of a good practice. Internal and external goods

A good evaluation of research practices is then an evaluation that is able to take into account the different elements that characterize a good practice, that is, both its outcomes (which can be classified in internal and external goods, following MacIntyre) as well as the virtues of the different types of researchers. Let’s focus on the former, leaving the examination of the latter to the next section.

We characterize internal goods as both the outcomes of research and the subjective experiences related to participation in the research practice, which does not necessarily translate into outputs. We call external goods the positive and measurable effects of research results or outcomes on society as a whole. Let us now examine in detail the nature of external and internal goods as well as how this distinction affects the plurality of standards that constitute good evaluation practice.

Since its products are both internal and external to scientific practice, having an impact outside the research community that potentially affects the well-being of society as a whole, it is advisable to use different styles of evaluation to assess each of them. Therefore, in addition to MacIntyre’s ethical perspective, our framework will also use notions from Nussbaum (2006)’s theory of capabilities, and from utilitarianism (as discussed in the next section).

To better characterize the notion of social practice that we are using to describe scientific research, it is useful to articulate further the distinction put forward by MacIntyre between internal and external goods.

Internal goods to a practice are high quality outcomes of the practice that (a) can only be specified in terms of some specific practice, as for example the way of conducting an empirical experiment; the practice of university teaching through lessons, seminars, and individual tutoring activities; the practice of interpretation and problematization of the texts of classics in the humanities; etc. and (b) can only be identified and recognized by actually participating in the practice in question. Those who lack the relevant experience are incompetent thereby as judges of internal goods (MacIntyre 1985: 189). Internal goods are reachable by those participants in the practice who practice it as an end in itself and not merely as a means to get something else, e.g. money, power, prestige. According to MacIntyre, these goods include the following three kinds of outcomes. They are.

  • the high quality in performance (e.g. ability to question a text; ability to ask relevant questions during an experiment; ability to motivate one’s own research group or students in class, etc.);

  • the high quality of the outcome itself (e.g. articles, books, research projects, discoveries, etc.);

  • the great value that comes from living a certain kind of life—the fact that occupying a certain professional role in a research practice contributes to the unity and value of the researcher’s life.

The last point needs more articulation. The idea is that those who participate in a practice by acquiring its internal goods are likely to consider it as something that makes their lives meaningful. They will tend to describe their lives as those of the participants in a certain practice and this will give a unitary character to the different parts of their biography.

Unlike internal goods, external ones are only “externally and contingently attached” to the practice by social circumstances and typically includes prestige, status and money. There are always alternative ways for achieving such goods, and their achievement is never to be had “only” by engaging in some particular kind of practice (MacIntyre 1985: 201). Moreover, external goods, when achieved, they are always some individual’s property—i.e. the more someone has of them, the less there is for other people. They are characteristically objects of competition in which there must be losers as well as winners. On the contrary, internal goods include the outcome of competition to excel, but also positive externalities. This means that their achievement is a good for the whole community who participate in the practice (e.g. Bowlby's attachment theory has transformed the way of seeing the relationship between mother and child by reducing trauma in hospitalized young children; Moore’s naturalistic fallacy argument has helped expose many fallacious arguments in philosophical reflection).

A narrow interpretation of McIntyre might lead to underestimate the value of external goods by arguing that their search threatens to corrupt non-instrumental motivations to follow a social practice. This may be explained by the fact that one tends to give an overly reductive interpretation of external goods which makes them coincide exclusively with material goods of the participants to the practice. This is not a necessary move. Indeed, there are goods that are contingently connected to practices but have a positive effect on large scale communities, or strengthen the link between practices and internal goods, or a combination of both. Moreover, we believe that nowadays research practices require hybrid forms of combination between internal and external goods. Different factors can explain this transformation, including the changes in the way in which science is produced and interacts with society (Scott 2003).

In the light of these factors, we propose to reconsider McIntyre’s distinction between internal and external goods. Specifically, we identify internal goods with the quality of its outputs, the way in which they are achieved (in accordance with the rules that constitute the practice), and the impact that following the practice has on researchers’ life plans. Moreover, expanding McIntyre’s approach, we identify external goods with the positive consequences that the following practice has on values protected by the democratic constitutions in which the practices have taken hold. That means that in order to evaluate a practice it is not only necessary to consider the internal goods, but also to assess whether the practices produce outputs that are in conflict with interests such as freedom, equality, health, respect for the environment, human dignity, and sociability.

In the light of this twofold requirement, we believe it may be helpful to interpret the two types of goods in the light of the capability approach developed by Nussbaum (Nussbaum 2006; for another interpretation of Nussbaum’s role in this debate see Hicks et al. 2018).

Specifically we holds the view that internal goods can be described as follows:

  1. 1.

    To use the senses, imagination and rationality in a typically human way, informed by adequate education. To be able to use imagination and thought in connection with our experience and produce works that are the result of our autonomous and reflective choices (reinterpretation of Nussbaum’s point 4: 95);

  2. 2.

    To be able to pursue the objectives of research without ulterior purposes but as intrinsic ends. To be able to have fun and play with activities related to the practice. Moreover, to be able to acquire and use specific mental capacities connected with the exercise of the practice such as the ability to apply the rules of the practice to completely new and unexpected contexts, ability to grasp the saliences of the situation required to act in accordance to the practice, etc. (reinterpretation of Nussbaum’s point 9: 95);

  3. 3.

    To be able to have attachments to people involved in the practice and to the outcome of research; to experience gratitude towards teachers and masters and justified anger towards those who betray our trust and violate our intellectual property. To be placed in conditions where one’s potential and development is not hindered by fear and anxiety (reinterpretation of Nussbaum’s point 5: 95).

Following the same approach, we argue that external goods are not only money, power or the reputation of the research institution and its capacity to attract investment, but also the impact research practice has on what Nussbaum has called the “human capacities necessary to live life worthy of human dignity”. These capacities, should include:

  1. 1.

    Life, Bodily Health Being able to live to the end of a human life of normal length— i.e. not dying prematurely, or before one’s life is so reduced as to be not worth living— and being able to have good health, including reproductive health, and to be adequately nourished (Nussbaum’s point 1 and 2);

  2. 2.

    Affiliation Being able to live with and toward others, to recognize and show concern for other human beings, to engage in various forms of social interaction; to be able to imagine the situation of another. Protecting this capability means protecting institutions that constitute and nourish such forms of affiliation, and also protecting the freedom of assembly and political speech. Having the social bases of self-respect and non humiliation; being able to be treated as a dignified being whose worth is equal to that of others. This entails provisions of non discrimination on the basis of race, sex, sexual orientation, ethnicity, caste, religion, national origin (Nussbaum’s point 7: 96);

  3. 3.

    Other Species Being able to live with concern for and in relation to animals, plants, and the world of nature (Nussbaum’s point 8: 96);

  4. 4.

    Control over One’s Environment (a) Political. Being able to participate effectively in political choices that govern one’s life; having the right of political participation, protections of free speech and association. (b) Material. Being able to hold property (both land and movable goods), and having property rights on an equal basis with others; having the right to seek employment on an equal basis with others; having the freedom from unwarranted search and seizure. In work, being able to work as a human being, exercising practical reason and entering into meaningful relationships of mutual recognition with other workers (Nussbaum’s point 10: 96).

The second element of a good practice. The virtues of researchers

In order to take account of these goods, the evaluation of research practice must also be able to assess the ability of researchers to obtain them. To this end, the virtues of the participants in the practice should also be taken into account. Following once again the MacIntyre setting, we define virtue as “an acquired human quality the possession and exercise of which tends to enable us to achieve those goods which are internal to practices and the lack of which effectively prevents us from achieving any such goods (Macintyre 1985: 191)”.

A potential issue arising from taking this approach concerns the relationship between moral virtues and the virtues that are relative to those who practice scientific research. This is one aspect of the more general issue which concerns the possible tension between the traits of character that make us good as human beings and those that make us efficient as occupying a particular social role. For example, it may be argued that the ability to take a certain detachment from suffering may be a necessary trait in a physician that allows him to make crucial decisions by looking only at facts objectively without letting himself be clouded by emotions. This same trait, however, is not desirable within family relationships where the ability to participate in the emotional life of loved ones is a fundamental part of relational life. Likewise, although a professor’s loyalty to his pupils can have the useful function of creating a close-knit group that works efficiently and does the good of research. This same trait could lead the teacher to misbehaviour when, in assigning a public job, he prefers one of his students to a more competent one. These are, of course, simplifications, and one could argue that the more detailed the example becomes, the more so-called conflicts are mitigated. Mitigated as it may be, however, it could be argued that some dose of conflict between the virtues of participants in social practices and moral virtues exists. And if this is true, what is the point of arguing that philosophical ethics can help us define the virtues of the academic researcher?

Our thesis is that moral virtues can be interpreted in ways that allow a typically non-conflictual relationship with the role-specific virtues. According to our proposal, which follows the general lines of Swanton (2007)'s analysis, the relationship between moral virtues and role-specific ones runs in two directions. On the one hand, role-specific virtues allow moral virtues to be given content, which otherwise would be too vague and generic to offer a practical guide to action and to assessing the conduct of others. This is to say for example that the virtue of courage acquires its content only when it is grounded on the paradigmatic cases of courage that human beings find themselves living in their concrete social interactions as parents, friends, members of a community, etc. On the other hand, what virtue requires in different social circumstances is delimited by the general meaning of virtue. To return to the example of courage, what is required of a brave friend is partly defined and circumscribed by the fact that courage should not be confused with recklessness and disregard for danger.

Having clarified this preliminary question, let us now examine what possible virtues would be desirable to be distributed among participants in a research practice. This is not intended to give the necessary and/or sufficient conditions for a practice to be a good one. We just aim to propose a possible list of virtues that would have chances to allow researchers to obtain the internal and external goods of the practice in which they take part.

The list we propose is the result of an elaboration of different sources, both McIntyre himself and more specific literature, for example business ethics and scientific research. Virtues are not presented in a lexical order. However, we can expect an empirical survey to reveal that some virtues are possessed with greater intensity by some types of researchers (e.g. leaders) and not by others.

  1. a

    Justice is the disposition, required above all by the evaluators of the performances and outputs of others, to treat others “in respect of merit and of the desert according to uniform and impersonal standard” (MacIntyre, cit., 191–192). The virtue of justice also requires us to apply this same standard to ourselves in relation to others, i.e. not to favour ourselves over people more meritorious than ourselves.

  2. b

    Courage is the capacity to risk damage or danger to oneself when individuals, values, goals that are crucial to the existence of the practice are at stake. Courage is therefore a way of showing that our attachment to these elements of the practice is genuine (McIntyre, cit., 192).

  3. c

    Resilience together with pride, this ability is indispensable to move forward in the research. It allows us to leave behind failures (paper rejected, unfunded projects, etc.) and to focus on future projects (Hormann 2018).

  4. d

    Empathy, Benevolence In line with extensive literature, by this term we mean the human ability to feel the emotions and feelings of other people through a vicarious feeling that is similar to that of the person with whom we sympathize. We do not believe, however, that empathy in itself is a virtuous capacity in research practices. Since empathy is an instrument for reading the other’s mind, it can also be used to manipulate others researchers in malicious ways. Empathy must be cultivated in such a way that it is rooted in the benevolent tendencies of human beings (Batson 2017: 2). In this way, empathy can allow the creation of a climate of trust between those who work within research institutions. Indeed, mutual trust is an indispensable component in these practices given the fundamental fact of the asymmetry of power that characterizes those interactions (Baier 1991).

  5. e

    Pride is evaluative attitudes towards ourselves (Ardal 1966; Cohon 2008; Taylor 2015). Unlike other emotions, which simply motivate us to pursue or avoid objects, this traits of character fix our attention on persons, casting a positive or negative light on them. If I am proud of my child’s success at school, my pride does not fix my attention on the ‘merits of my child,’ and still less on ‘me in the role of father,’ but on the whole of myself. As Cohon has rightly said, “when I feel pride, I am proud of something in particular [its cause]... But the attitude of pride is a pleasure or satisfaction not in that particular accomplishment or possession, but in myself in my entirety” (Cohon 2008: 166). We believe that the pride associated with one's own achievements in research and the consequent approval of one's peers or superiors is a fundamental spring that drives researchers to perform at best in their area of research (Tangney, 1999).

  6. f

    Prudence is the capacity to sacrifice the satisfaction of less important pleasures closer in time than the satisfaction of more distant but more important pleasures. Where the degree of importance is defined with respect to the long-term objectives that characterize our lives (Parfit 1984).

  7. g

    Humility is the ability to accept the authority of the standards related to the rules that define the practice. I have to recognize that other participants know rules and know how to apply them better than I do. I have to be willing to learn from these people and accept their criticism (MacIntyre 1985: 193).

  8. h

    Patience is the ability to curb one's own urge to complete a research in order to obtain as soon as possible the gratification of a positive result. To be able to wait and to be guided by a cautious scepticism that prompts us to control accurately the different steps of our investigation.

  9. i

    Accuracy is the disposition that consists in the care with which the individual researchers collect data that will constitute the pool of information shared in the research practice (Williams 2002).

  10. l

    Sincerity, Honesty is the disposition to tell the truth to others and, when this does not happen, it is the capacity to indicate good reasons why this did not happen where good refers to the fact that these reasons have a constitutive reference to the interests of other people (McIntyre 1985; Williams 2002).

  11. m

    Integrity: is the willingness to behave in such a way that our actions are the outcome of our deepest values and commitments and that we tend to refuse making them hostages to imposed obligations or duties that we do not endorse on reflection.

  12. n

    Creativity is the ability, which finds expression both in our social interactions with others and in the products of our research, to produce something that not only has value but is characterized by the elements of novelty and the capacity to arouse surprise in others. (Swanton 2003: 162, 165).

  13. o

    Practical wisdom is a kind of super-virtue essential to make each virtue effective. It enables the virtuous agent to acknowledge and respond properly to the items in the field of the research practice, choosing the appropriate means for their own ends. (McDowell 1979). Moreover, it also allows the different virtues within an individual’s character to operate and develop harmoniously with each other.

We argue that these virtues are those traits that enable researcher to acquire the internal goods of research practices. We also argue that the link between virtue and internal goods is not instrumental but conceptual: internal goods are not fully intelligible or achievable except through the exercise of the virtues mentioned above. The situation is different for external goods. Even if the possession and exercise of the virtues by researchers can allow them or the institution in which they work to obtain them, this also depends to a considerable extent on other factors. In particular, by the institution’s relations with other companies and organisations and by its ability to communicate and sell its results externally (Scott 2003).

Three typology of researchers

In the aim section we argued that good practice is underpinned by different types of virtues that cannot be allocated in a single character. This means that good practice will be composed of different types of researchers who, having different personalities and experiences, are capable of hosting different groups of virtues. From MacIntyre’s notion of research practice, we highlighted an organization of research based on a division of labour and/or distinctive roles that make it possible to innovate in the tradition of the practice. In this organizational structure we identify a basic three typology of researchers composed by the leader, the good researcher and the honest researcher. Each type of researcher is characterized by the possession of certain virtues.

The internal organization of team’s knowledge production has been studied in many works and with different perspectives (Latour and Woolgar 1979, 1999, 2001). However, we still don't know why the research team organizes itself in a certain way and according to a specific configuration. Our model, based on the notion of MacIntyre practice, helps to understand why knowledge production depends on collaboration between different types of participants whose roles emerge spontaneously according to their specific skills.

Let us describe in details the main features of each type of researcher. The good researcher is a typical participant of a good research practice defined so far—she/he participates in the practice learning and developing the virtues of the practice. The good researcher employs typically human qualities to respond in the best way she/he can to the problems that are typical of that practice enabling her/him to creatively advance a particular stock of knowledge. Together with the good researcher, we identify two other types of researchers: the leader researcher and the honest researcher. The leader researcher is one who achieves an outstanding level in the development of creative and social virtues enabling her/him to produce excellent outputs and to be a motivating leader in research group. Finally, the honest researcher is the one who does not produce outputs that are contrary to good research practices. More precisely, the honest researcher typically exemplifies the researcher who has completed her/his PhD and is at the beginning of her carrier. Within research institutions, this figure mainly carries out her/his activity in the service of more experienced researchers. Within the university, she/he carries out her teaching activity mainly as a tutor not having yet her/his institutional course during one of the terms. The figure of the honest researcher generally progresses towards that of a good researcher and, in some cases, becomes a leader. However, this may not be the case. In such a situation, an honest researcher is one who, despite having a permanent job as a lecturer or researcher for many years, continues to carry out the tasks she/he was carrying out at the beginning of her career. She/he, however, grasps the research practice in which is involved, with its “internal goods”, and fulfils his/her role of being at the service of the practice.

We argue that the three types of researchers should be present in good research practice. A research practice consisting of all leaders or all honest researchers may not be good practice as it may generate harsh discussions among members about the strategy for carrying out the research work. In addition, a practice consisting only of good researchers or honest researchers may have difficulties in developing and thriving as it does in good research practice.

Analysis and results

In this section, we attempt to develop a quantitative assessment of the virtues that a researcher involved in a “good” research practice should have. In addition to the works cited in the previous section, several strands of literature have treated these topics from different points of view.

Resnik (1998, pp. 48–61) proposes 12 principles of ethics in science which relate to the research process. The most important ones, related to the virtues cited in the Sub-section “The second element of a good practice. The virtues of a researcher” are included in our Table 1 below. Sagiv et al. (2017) report a comprehensive review of the numerous existing studies on personal values, integrating different streams of research in psychology, sociology, management and political science. They define values as “what is good and worthy” and individual or personal values as “broad desirable goals that motivate people’s action and serve as guiding principles in their lives”.

Table 1 Towards a Preliminary Questionnaire for an Evaluation of Virtues in Research Practices
Full size table

In their study, Sagiv et al. (2017) state: “Individuals act in ways that allow them to express their important values and attain the goals underlying them. Thus, understanding personal values means understanding human behaviour (Sagiv et al. 2017). In contrast to the numerous studies investigating the consequences of values, much less is known about the origin of values.” By summarizing existing literature from different streams of works, Sagiv et al (2017) conclude that “understanding personal values means understanding human behaviour” and identify 10 basic human values. We attributed this list of values to the list of virtues we proposed in the Section “The virtues of a researcher involved in a good research practice” (see Table 1).

Åkerlind (2008) on the base of a literature review and following a series of interviews conducted for his study, shows that there are 4 qualitatively different ways of understanding being a university researchers, ranging from fulfilling academic requirements, establishing oneself in the field, developing oneself personally and enabling broader change. Åkerlind (2008) found also a dimension of the research experience that was not present in previous studies which relates to different range of feelings associated with each way of understanding being a researcher. She found a graduation of feelings, ranging from “Anxiety to satisfaction” for those that consider research as “fulfilling academic requirements”, from “Frustration to joy” for those considering research as “establishing oneself in the field”, feelings of “interest and enthusiasm” for those that are doing research for “developing oneself personally”, up to “passionate engagement” for those that are doing research for “enabling change”. Taking into account these different feelings, she concludes, may be helpful to understand differences in the behaviour of researchers belonging to the same field or similar field of study.

Developing further the previous consideration, Bazeley (2010) proposes a conceptual model of research performance including six main components that are engagement, task orientation, research practice, intellectual process, dissemination and collegial engagement. These items are included in our list of items to consider for developing a Questionnaire for a quantitative assessment of virtues (see Table 1).

The managerial literature on virtue ethics in business has received a recent development. See Ferrero and Sison (2012) for a review. In particular, Walumbwa et al. (2008) put forward a quantification of leadership comprising self-awareness, relational transparency internalized moral perspective and balanced processing. We integrate this perspective in our list of virtues proposed in the Sub-section “The second element of a good practice. The virtues of a researcher” to show that our framework is flexible enough to embrace different streams of literature. See Table 1 in which the dimensions proposed by Walumbwa et al. (2008) are integrated in our framework.

It is important to point out that the list of items reported in Table 1 is not exhaustive. It represents a first rough base for developing a preliminary questionnaire for attempting a quantitative assessment of the virtues of researchers. Snow (2014) has recently launched a promising line of research based on the elements of psychology that characterize the virtues. In its perspective virtue is composed of the following three elements: (1) intelligence, which highlights the fact that virtue proceeds from a set of cognitive and emotional mental states that enable us to be sensitive to some morally relevant features of the situations in which, really or imaginatively, we find ourselves (Snow 2014, pp. 4–5. See also Snow 2010 and 2012); (2) dispositionality, that refers to the fact that this state is a trait of the personality of the agent and is not only an occasional element of her/his psychology; (3) behaviour, i.e. virtue typically manifests itself in the actions and other behavioural responses of the virtuous person (Snow 2010, pp. 4–5).

Snow argues that each of these characteristics of virtue can be measured and she outlines a model that consists of three measurement criteria. First, the agent’s performance must be taken into account, i.e. the presence of the virtue in question must be verified from the agent’s ability to repeatedly perform a given behavioural pattern in the different situations that constitute, so to speak, the field of action of a specific virtue. Secondly, Snow believes it is crucial to take into account the reports that agents make of their emotional and cognitive life during the performance of actions that they consider virtuous. To facilitate this task, Snow believes it is desirable that, on the model of some US colleges, research institutions make available to their participants special apps that can be downloaded on any electronic device, allowing them to collect the results of the self-observations of agents. Gathering the products of introspection, in addition to offering a useful material to those who are called to assess the presence of virtues in others, also allow agents to take into account the health of their virtues and measure any flexing or, on the contrary, increases in their readiness and effectiveness in responding to the pressures the world exerts on them. Finally, Snow argues that it is important to connect these data with those that impartial observers, in the figure of external evaluators, can collect in the course of annual surveys covering both the outputs of the research and the way in which the researcher dwells in different spheres of social interaction with other participants in the practice.

A further problem to be addressed is which questions to introduce in the questionnaires. These must be sufficiently diversified to allow the evaluators to answer not only the dry question about whether or not there is a virtue, but also to determine the quantum of it. Snow suggests four levels to be introduced in the questionnaires. The first verifies the presence in the agent of receptivity to the stimulus that typically activates virtue. The second examines her/his ability to recognize the virtue appropriate to the given circumstance. The third verifies the most complex ability to generate a virtuous response. The fourth, finally, measures the agent’s ability to generate the virtuous response across a variety of situations.

Following the four levels questions introduced by Snow, it is possible to measure on a scale from 0 (minimum) to 4 (maximum) the researcher’s mastery of virtue (over a spectrum ranging from (1) the ability to understand the importance of the problem to which virtue constitutes an answer, to (2) the ability to recognize the virtue in question, to (3) the ability to express virtue occasionally, to (4) the ability to manifest it in all situations that constitute the scope of that virtue.

On the basis of our characterization of the different types of researchers, it can also be expected that some virtues will be typical of one group and not of another. Leaders, for example, will tend to show pride. While humility will be typical of honest researchers.

Our implementation of the questionnaire, built on Snow’s recent works, includes several tools that may allow us to introduce a quantitative dimension in the evaluation of virtue. This may allow us not only to evaluate more concretely the ability that a researcher has to contribute to the achievement of internal and external goods to the practice, but also to verify how a certain environment is conducive to the cultivation and transmission of virtues.

Let us briefly dwell on some of the immediate benefits of an evaluation of research that includes these instruments. Firstly, the evaluator can take into account not only the outputs, but also the potential of the researcher. This information is crucial both when deciding on the advisability of giving the researcher new job opportunities and when making a choice between researchers who may have reached the same scientific achievements in terms of outputs. Secondly, these tools allow to identify the reasons why a research group/institution/department produces certain results and not others. Being able to verify, through an empirical survey, whether the group discourages, for example, empathy among its members or does not tolerate expressions of pride and self-esteem, unless leaders makes them, exhibit flaws in the practice and suggests at the same time remedies to improve the group’s performance.

Having said that, it is worthwhile to say a few words about other more general advantages deriving from the cultivation of virtues within research practices that do not directly concern the positive effects they have on outputs, but rather the function they perform on the psychophysical health of researchers. According to a model that was originally formulated in an empirical study on the well-being of adolescents in an US colleges, good schooling must not only aim to ensure a high level of learning, but must take into account how programs and school life affect the overall flourishing of the character of their students (Seligman 2011, 2012; White 2014, pp. 2–3). According to this research, individual flourishing depends on the satisfaction of at least five elements: positive emotion, engagement, relationships, meaning, and accomplishment (PERMA). The achievement of these goods, says Seligman, is based on the cultivation of the virtues of character. Although there are still no empirical studies in the field, it seems plausible to argue that a similar model can be used to explain the satisfaction that comes from performing one’s duties within research practices. A satisfaction that reaches its peak when we gain the awareness that the achievement of the internal goods of the practice are functional to the achievement of its external goods, i.e. the human needs and prestige of the institution in which we work. However, as has been mentioned several times, it is only through the cultivation of virtue that we can hope to achieve internal goods and create a stable balance with external ones.

Discussion

One of the central theses of this essay is that virtue can be transmitted within research practices from the most experienced components or leaders to the new generations of researchers. This transmission is a crucial fact. As we have repeatedly argued, the peculiar notions of social practice and internal goods used in our evaluative model imply that it is not possible to acquire the virtues that realize those goods if not within the practices in which these goods emerge.

At a general level, this is a plausible thesis, which does not give rise to any particular objections. It could be argued, however, that a virtue-oriented evaluation model should engage in a stronger thesis and it is precisely at this point that difficulties may arise. What should be indicated are the criteria that allow us to quantify this transmission, allowing us to distinguish the situations in which it occurs effectively from those in which the transmission is defective. In order to make this distinction effectively, it would be necessary to measure the possession of virtue in young researchers, thus making it possible to obtain indirect indications as to which circumstances are most favourable to their transmission within the practices. It is precisely this additional requirement that seems to raise objections. Virtue, it has been argued, seems to consist of a special sensitivity that escapes empirical measurement (Murdoch 1998).

Recently, however, some scholars have tried to undermine this pessimist assumption by giving hope to those of us who seek to develop a model of evaluation of research that also includes a virtue-based element. It has been argued that although in some cases pessimism may be plausible, in others it is not. It is likely in exceptional cases, i.e. far above or far below the average threshold. But in cases involving the behaviour of most of us, pessimism is not plausible: in cases falling within the average, it is conceivable to give an account of the psychological states that underlie our virtues, and it is also possible to do so in ways that do not escape empirical investigation.

Indeed, Nancy Snow recently claimed that extraordinarily virtuous people can be compared to individuals who have a special artistic talent. In both cases, their abilities are clearly not attributable to specific elements of their psychology. Likewise, when we are dealing with people who while not having received an adequate moral education or, worse, have been morally corrupt, strive to behave affectionately with others, it is difficult to say whether they simply act out of a sense of duty or have acquired a minimum interest in others (Snow 2014, p. 3). Consistently, our framework and the typology of researchers we propose does not include “Genius” as we consider them as the limit of the spectrum of possible researchers that is difficult to assess (Kaufman 2013). In other words, genius should be detected and treated differently. What we consider as leader, identifies instead a clear set of dispositions of the character that is possible to teach and learn and hence, at a given extent, can be measured. On the contrary, it is not possible to teach and learn how to be a “Genius”.

Implications and further development

The preliminary questionnaire proposed in Table 1 represents a first attempt to quantify the virtues of researchers involved in research practices. It should be extended, tested and consolidated. Another very important aspect concerns education. That is to investigate how virtuous behaviours are taught, communicated and learned through models, exemplars (e.g. Zagzebski 1996; Nakamura and Condren 2018). All these aspects are left for future work.

The conceptual framework developed in this paper allows us to identify (define) the good evaluation as the evaluation that is able to discriminate between good and bad research practices. Having characterized the research practice through “internal” and “external” goods, offers us the possibility to deepen our understanding about what is a good research practice and what is the role played by researchers in it. Reflecting on what is a “good” research practice, on what is the role of researchers in it, according to the typologies of researchers we propose (leader, good and honest researcher), may be extremely useful for many reasons.

Firstly, it offers a self-assessment tool for researchers, to understand the functions of their research activities, their motivations and where they are in their research practice. This is an important step towards the improvement of research practices and individuals involved in it.

Secondly, it helps institutions to collect and describe the main functions of the research practices (highlighting their special features) developed by its researchers, and their motivations, to include them in their strategic plan. This is a further important step for the development and improvement of the organizations involved. A “good” evaluation should take into account all the building constituents of a “good” research practice and should be able to discriminate between good and bad research practices, while enforcing the functions of good research practices.

These reflections, although at their infant stage, may be the starting point for a paradigm shift in the evaluation of research practices. From an evaluation focused on products towards an evaluation focused on the functions of research practices. This new way of evaluate might also contribute to improvement of the research practices itself, stimulating new innovative solutions thanks to the self-assessment of the research community, providing clearer views of the strategy, missions and functions of the groups involved in the research practices.

Further studies should deepen the articulation and development of the concept of “good” evaluation and the development of a theory of good, which explains the good items, and a theory of right action, which explains what amount to a right action within a research practice.

In this paper we aimed to take the first steps to provide a notion of good evaluation of research guided by a McIntyre’s ethics that is based on the viewpoint of those who possess the virtues that sustain social practices. Further studies and research are needed to explain how this perspective, while internal to practice, is sufficiently detached from particular practices to allow a critical viewpoint on them.

Concluding remarks

We attempt to deal with clarifying and discussing what it means to make a “good” research evaluation. We maintain that the most appropriate level of analysis to make a good evaluation of research is to refer to “research practices” according to MacIntyre’s definition. Resnik (2012, p. 341) identifies the need of further research “to describe the virtues that operate in science, explore how scientists learn moral virtues, and determine the extent to which virtues have an impact on scientific thinking and behaviour.” We have worked on several levels to contribute to this lack of knowledge: by developing a general model centred on the notion of social practice; by identifying three different types of researchers; by proposing a list of virtues; by indicating the elements on which to build a questionnaire that can measure the intensity of the mastery of virtues in researchers. An important feature of our proposal is that we do not only use epistemic virtues (which concern the ability to know/reach the truth), but also moral virtues, which concern the way we take into account the well-being of others and which allow us to coordinate and cooperate better with them.

Our contribution is also to try to raise the very heated debate on research evaluation policies, moving the discussion from the ideological sphere (based on being in favour or against evaluation regardless of the evaluation context) to that of contents. This means analysing what we are measuring and why, before discussing how and when to measure it.

Speaking of good evaluation and good research practice also contributes to shifting the focus of evaluation from the outputs of research practices to the functions performed by participants in research practices on the base of their virtues. This would indeed represent a “paradigm shift” in the field of research evaluation.

The model we propose can be used for different types of evaluation and offers a self-assessment tool for researchers, to understand the functions of their research activities, their motivations and where they are in their research practice. This is an important step towards the improvement of research practices and individual behaviour involved in it. This can help institutions to collect and describe the main functions of the research practices (highlighting their special features) developed by its researchers, and their motivations, to include them in their strategic plan. In the last sections, we discussed the possibility of evaluating the characteristics of the individuals participating in research practices.