Quality is integral to scholarly research and publishing, but we have struggled to agree on how to define it. This is due, in part, to the complexity and dynamic nature of research and publishing. For example, a graduate student may come out of a high-quality program (based on published rankings), produce low-quality research (based on a professor’s assessment), then improve the manuscript’s quality through peer review, only to publish it in a low-quality journal (based on Impact Factor), but then get picked up by the media and distributed to a wide audience, thus resulting in high readership and citation numbers (an indicator of quality). Additionally, from an administrative perspective, the unique, field-specific expectations of scholarly knowledge production create a complicated landscape in which to create policies intended to guide individual and institutional research objectives. Then, to add to the complexity, scholarly knowledge production is now a global affair, and we still face an “inclusion/exclusion divide in the global knowledge pool” (p. 37) with Anglo-European perspectives dominating [1].

To the extent that scholars have addressed research quality, they tend to focus on peer review [2, 3]—a means of quality assurance that is implemented after research is conducted [4]. By contrast, this study examines how research quality is conceived by diverse stakeholders in knowledge production before peer review—an area that has not been extensively explored through empirical methods [5]. In this study, stakeholders refer to individuals who are involved, in some manner, with the production of scholarly research. Our sample includes researchers both inside and outside of academic institutions, writing and publishing consultants, publishers, and academic librarians. Importantly, we also included scholars working in a variety of geographic settings to better represent the epistemic diversity of global scholarly knowledge production [1].

1 Theoretical framework

Our theoretical framework is based on Harvey and Green’s [4, 5] five conceptualizations of quality: exceptional, perfection (or consistency), fitness for purpose, value for money, and transformative. Within this framework, there is great care to separate quality from quality assurance—with quality assurance defined as “checking the quality of a process or outcome,” and quality as “the conceptual tool through which these purposes are implemented” [7]. Harvey further differentiates quality from standards, placing quality as the driver for how processes are carried out, as opposed to standards, which are how those processes are measured or evaluated [7]. Additionally, quality is differentiated from qualities, which should be considered attributes, rather than “as essential embodiment of character” [7]. Harvey and Green [6,7,8] have identified academic standards, standards of competence, service standards, and organizational standards as the areas of higher education that could be assessed through their conceptualizations. None of these areas account for research and knowledge production in higher education. In fact, quality in research in much of the work using Harvey and Green’s framework is relegated to assessment through peer review and recognition [7]. However, given the multi-faceted understanding of quality presented by the framework, we believe Harvey and Green’s conceptualizations of quality can be used to frame an exploration of quality during scholarly knowledge production. This seems especially appropriate for applying the conceptualizations of quality to a global knowledge environment.

Harvey and Green’s conceptualizations of quality were first presented in a 1993 article, but they have evolved since then, as summarized below.

1.1 Perfection or consistency

In this conceptualization, quality is differentiated from standards—it is something which conforms to a specification that in itself is not “assessed against any standards” (p. 16). Specifically, there is a focus on committing zero faults at each stage of the process, rather than relying on a final inspection (such as peer review) to identify flaws. Additionally, in this conceptualization, quality becomes a relative concept, allowing flexibility for disciplines and individuals to determine their own standards of zero faults [7].

1.2 Fitness for purpose

Quality under this conceptualization is one of the most widely used in evaluating certain aspects of higher education [7, 9] and is related to the “purpose of the service or product” [6]. Under this definition, quality is equated with functionality, and whether a product or service does what it was intended to do from the perspective of the individual or institution that is using the product or service. Harvey [7] notes that fitness for purpose is a particularly troublesome measure of quality because it allows for each individual object, person, or institution to have multiple purposes, whereas Green [9] argues that purposes change constantly and are often unclear, making it difficult to measure and evaluate quality under this conceptualization.

1.3 Transformative

In this way of approaching quality, there is a process of change and growth through learning. Harvey and Green [6] position transformation in relation to research production through understanding that the researcher situates their work in the broader context of their field to build and transform the body of knowledge in which their work is contextualized.

1.4 Value for money

Quality as value for money is conceptualized as customer-centered (e.g., tuition-paying students) and embraces competition as a driving factor for effective use of resources [6, 10]. Effectiveness, in turn, is gauged in terms of research assessment, control mechanisms, and quantifiable outcomes [6].

Harvey and Green’s [6, 7] conceptualizations derive from a survey of ways of understanding quality that have been prevalent in higher education literature. Although quality will always be somewhat elusive, Harvey and Green’s model has offered scholars some terms and definitions that have helped pin down this slippery subject. Their model has been especially influential in studies of quality in teaching and learning [e.g., 11–15]. This study extends this conversation by applying Harvey and Green’s model to discussions of quality in scholarly knowledge production and addresses implications for higher educational policymakers tasked with guiding scholarly knowledge production efforts at their institutions. Notably, examinations of quality in the knowledge production process through empirical studies are scarce [5], positioning this analysis to offer important insights about the indicators of quality as knowledge is produced through research in academic settings. Additionally, we approach this exploration from a standpoint of acknowledging and celebrating epistemic diversity, which calls for inclusion of diverse “hermeneutical horizons of experience and reservoirs of knowledge to inform scientific production” [16].

To explore the nuances of quality in scholarly knowledge production, we address the following research question:

RQ1:

How can Harvey and Green’s conceptualizations of quality orient our understanding of quality in global scholarly knowledge production?

As elaborated in the results section, our analysis reveals a disconnect between institutional and individual perspectives on quality. Scholars’ perceptions of quality when they refer to their own research practices align most closely with Harvey and Green’s conceptualizations of quality as ‘consistency or perfection’ and quality as ‘transformation.’ However, participants suggest that the indicators of quality most readily embraced at an institutional level are ‘fitness for purpose’ or ‘value for money,’—perspectives they view as challenges to maintaining or achieving quality in the knowledge production process. These findings contribute to the previous research outlined in the next section by offering first-hand perspectives from multiple stakeholders involved in global scholarly knowledge production.

2 Quality in scholarly knowledge production

Arguably the most important marker of success in academia is the production, and ultimate publication in peer-reviewed outlets, of scientific knowledge. Yet most knowledge production occurs before peer review takes place. In many Euro-American research environments, there was a movement in the late twentieth century to address quality in research, primarily in response to public funding allocation requirements [9]. But despite an attempted emphasis on research quality, a gap remains in our understanding of quality in scholarly knowledge production [17, 18]. For example, while Kajaste’s [18] study on Finnish research quality noted the dearth of research examining quality in research production, their study instead focused on quality assurance processes [7] through an examination of the quality audit reports of Finnish institutions, rather than knowledge production.

A few published frameworks have begun to provide criteria or contextual understanding of quality in research production [19, 20]. As for the history of quality, qualitative research seems to have been examined more closely, as outlined by Welch and Piekkari [20]. In general, articles outlining mechanisms for protecting or increasing quality mostly reflect a field- [21] or method-specific perspective [19] and often address ethical conduct as a placeholder for quality [22,23,24]. One frequently mentioned aspect is the role of senior faculty in mentoring and guiding junior researchers by modelling ethical research behaviors and hands-on guidance [25,26,27]. Additionally, some recent literature touts Open Science (OS) for its potential to increase transparency and reduce over-sensationalism [28, 29]. In brief, OS principles encompass open data, and other transparency mechanisms such as pre-registered reports, open software or coding, and open peer review [30, 31].

Challenges to producing quality research are frequently discussed from a field-specific perspective as well. Such challenges include questionable research practices [32], methodological challenges [33, 34], proximity between researchers and subjects [35, 36], and interdisciplinary research [37]. A few scholars have addressed ethical issues in specific fields, including authorship, participant recruitment, conflicts of interest, and accurate representation of data [e.g., 38], or assessing threats to social validity [22]. Still others have argued the quality of scientific knowledge has long been diminished by racism, sexism and colonialism [39] and is often manipulated to increase potential impact through exaggeration [28, 29].

A particularly salient challenge to scientific knowledge production is the persistence of colonial ideology [40] in the ways information transfers from the global North to the global South,Footnote 1 leading to undervaluation and marginalization of knowledge from the global South [40,41,42,43]. On a broader scale, Harvey and Williams [17] provide an overview of scholarly examinations of quality for a decade in one journal dedicated to the subject, noting that scholars around the globe have dealt with growing demands for accreditation, despite mounting evidence that external evaluations of quality do not necessarily increase output quality, and that faculty increasingly distrust the institutions they serve.

Literature on policy related to quality in scholarly knowledge production is sparse. Bogue and Hall [44], for example, argue that quality cannot exist without integrity in personnel, policies, and programs, and that policy development aimed at increasing or maintaining quality needs to incorporate multiple stakeholders in all areas of the institution. Decades ago, Neave [45] noted that national policies that seem unrelated to educational policies (e.g. anti-protest laws, financial austerity procedures) actually filter into educational policies and impact how institutions of higher education evaluate their research production activities.

In sum, although academic quality has received a great deal of attention, it remains elusive and difficult to operationalize, quantify, or automate. Much of the previous research consists of opinion pieces or editorials, or examines narrow, easily quantifiable standards for quality improvement or field-specific challenges. To date, few empirical studies have explored the perspectives of knowledge producers. In addition, all too often, markers of success for research are the endgame—publication in a peer-reviewed journal or citation counts—further contributing to a gap in empirical knowledge regarding quality in the process of knowledge production. Finally, whereas Harvey and Green’s five conceptualizations of quality have been applied to teaching [11, 12], institutional assessment [13], program evaluation [14], and peer review [3], this study applies them to scholarly knowledge production. Given that this study explores the perceptions of individuals working as members of organizations, participant perspectives may reflect both internal and external components of quality—something that is not addressed in the current literature.

3 Methodology

A total of 48 in-depth interviews with scholars, administrators, librarians, and other publishing professionals were conducted between March and October, 2020, as part of a larger project examining scholarly publication ethics, funded by the National Science Foundation (NSF; Award # 1926348, 2020). Participants were from 17 countries of origin, and 18 countries of residence. (See Tables 1 and 2).

Table 1 Summary of participant demographics
Full size table
Table 2 Detailed participant demographics
Full size table

Interviews covered perceptions related to quality in knowledge production, and research integrity, along with views about predatory publishing, Open Science, and Open Access (see Appendix 1 for full interview guide). The data presented here represent information directly relevant to quality in scholarly knowledge production. Scholars represented a wide range of STEM and social science disciplines. Participants were recruited through personal networks, ListSERV’s, pertinent organizational announcements, and snowball sampling. Given that the research team hails primarily from the media and communication and STEM disciplines, this resulted in a high number of participants who are active in the media and communication and STEM fields. This nonprobability sampling allowed us to access participants who would have otherwise been difficult to access, especially those outside the Euro-American academic environment [46], but did result in a sample that falls heavily in the social science and STEM disciplines.

Interviews were transcribed verbatim, and then coded using NVivo qualitative research software. Initial coding was conducted by the first author, followed by review and discussion with the research team to discuss emerging themes and findings. (See Appendix 2 for detailed codebook). Data were analyzed following Owen’s [47] principles of repetitions (i.e., the same theme being mentioned by multiple interviewees in the same way), recurrence (i.e., the same theme being discussed by interviewees in different ways and from different perspectives), and forcefulness (i.e., an argument that is mentioned by few interviewees, but the idea is compelling) to select themes.

3.1 Compliance with ethical standards

The interviews conducted for this study were guided by IRB protocols at Texas Tech University. The interview protocol was approved by the Texas Tech University IRB (IRB2019-238) committee in accordance with the Exempt category, which includes research that involves minimal risk and follows with the 1964 Helsinki Declaration. Participants were given written information prior to beginning an interview, and were given the opportunity to give their consent, or to withdraw consent at any point during the interview process, or to decline answering any question(s) during the interview. All data was immediately de-identified after collection.

4 Results

Participants’ perceptions of quality fall into two broad categories: defining characteristics and challenges. As elaborated below, comments in the former category align most closely with Harvey and Green’s [6, 7] conceptualizations of quality as ‘perfection or consistency’ and quality as ‘transformation,’ and comments in the latter category align most closely with Harvey and Green’s conceptualizations of quality as ‘fitness for purpose’ and ‘value for money.’ In our analysis, Harvey and Green’s conceptualization of quality as ‘exceptional’ is removed from consideration because its focus on operationalizing high standards of students’ academic achievement in the classroom is not applicable in the production of research. Because we focus on knowledge production through research, rather than through academic processes in the classroom, we excluded the ‘exceptional’ dimension from this discussion.

4.1 Defining characteristics

The defining characteristics of quality, as identified by our participants, were coded into the themes of ‘following scientific protocol,’ ‘committing to ethical behavior,’ ‘dedication to transparency,’ ‘knowing your field,’ and ‘research that has been deeply thought about.’ In viewing these results through the lens of Harvey and Green’s [6] framework, it became clear that individuals in our study conceptualized the defining characteristics of quality research production in ways that correspond to some extent with Harvey and Green’s conceptualizations of either perfection/consistency or transformation (See Fig. 1).

Fig. 1
figure 1

Alignment of emergent defining characteristic themes with Harvey and Green's (1993) conceptualizations of quality. See Codebook (Appendix 2) for full Emergent Theme descriptions

Full size image

4.1.1 Quality as perfection or consistency

The themes of ‘following scientific protocol,’ ‘committing to ethical behavior,’ and ‘dedication to transparency’ all align with Harvey and Green’s [6] conceptualization of quality as a process where certain specifications should be met with zero defects and where the process is done correctly the first time. By focusing on the context of scholarly knowledge production instead of institutional mechanisms, our interview data heighten awareness of individual agency, revealing a contradiction: On the one hand, individual scholars are deeply trusted to ensure the quality of the knowledge they produce, but on the other hand, conventions of scholarly publishing such as the expectation of a detailed methods section require them to document that they have behaved properly in producing knowledge. Thus, we also gain valuable insights into some key values the are pertinent in scholarly knowledge production, even across the epistemic diversity that exists among our interview sample.

Following scientific protocol Using specific technical mechanisms to follow scientific protocol was the most frequently mentioned way of producing quality research. Some participants broadly connected quality to following scientific protocol (P02, P24), while others delved into specific mechanisms they associated with quality. Rigor was frequently mentioned (P09, P14, P16, P31, P41, P48), along with replicable methods (P02, P08, P12, P19, P25, P31, P38, P46, P48), as ways to ensure scientific protocol was being followed, as summarized by this participant: “I realize quality is a judgment call, but to me it is about…very clear methods, very clear controls, very clear descriptions of all your methods, because ultimately in science, it's supposed to be repeated” (P08). Other participants noted the importance of relying on peers to help ensure quality before journal submission (P02), modelling openness in the research process for junior scholars (P03) and having honest discussion about authorship early in the research process (P08, P09, P26, P33, P40). Participants represented in this theme represent a broad spectrum of our sample—researchers, librarians, and publishing professionals from nearly all the regions represented in our sample seemed to agree on the need for specific technical mechanisms that ensure quality in the knowledge production process.

Committing to ethical behavior Committing to ethical behavior when no one is looking was discussed in a range of ways, from statements relating to values such as, “Just behaving, being a good citizen in the scholarly world and not doing bad things with regard to your own research” (P05) and “the moral value of honesty is essential for the scientific process to progress” (P08) to more concrete suggestions related to research practices, “not tampering with data” (P19) and “don't go and steal someone else's work” (P30). Participants touched on the need for ethical behavior in every part of the research process, from going through the “IRB [Institutional Review Board] process” (P02), to “respecting your subjects” (P10), to “present[ing] your results as it is” (P02). Ultimately, committing to ethical behavior was best summed up by this participant who said, “you have to be dedicated to the truth, period, no matter what” (P08). Across all the disciplines represented by the participants in this study, there seemed to be a sense of personal responsibility to do the right thing—and this notion of personal responsibility did not vary among participants working in different regions or professions.

Dedication to transparency After ‘committing to ethical behavior’ and ‘technical mechanisms,’ a dedication to transparency was the next most frequently mentioned attribute of quality in scientific knowledge production. Some framed transparency as “being willing to show your work” (04) whereas others positioned it as “giving appropriate credit to the people who were involved” (P09) or mentioning specific mechanisms to increase transparency such as “preprints… because you can see how the work evolved” (P12). As one participant noted, “The more open you can be about [your processes], the more likely you are to pick up what went wrong, what went right, share it, build on it” (P21). In short, showing a dedication to transparency refers to a wide array of conventions in scholarly publishing that require a scholar to demonstrate that they have behaved properly. As with the other themes that align with perfection or consistency, participants from numerous professional and national backgrounds advocated transparency as a core marker of quality in scholarly knowledge production.

4.1.2 Quality as transformation

The themes of ‘value mechanisms,’ ‘know your field,’ and ‘research that has been deeply thought about’ all align with Harvey and Green’s (1993) conceptualization of quality as transformation (see Fig. 1). Specifically, participants emphasized that the goal of knowledge production is transformation, accomplished by advancing the disciplinary conversation to which one contributes. As in the previous section, interview data suggest a contradiction in the scholarly knowledge production system: scholars are trusted to behave properly and have a great deal of autonomy to make their own ethical decisions, but at the same time, scholarly discourse conventions guard against actions that would erode the quality of knowledge.

Value mechanisms The most frequent sub-theme under the ‘value mechanism’ theme was ‘being critical of your own work,’ expressed through statements such as, “be more critical on yourself than anybody else is” (P08) and the suggestion to ask, “How can I self-evaluate more effectively?” (P12). Participants also pointed to the “moral value of honesty [as] essential for the scientific process to progress” (P08), a sub-theme that acknowledged the conundrum of needing morals in science because “science itself depends on that moral value but morality isn't science…you can't measure morality units…there's no morality meter” (P08). The final sub-theme discussed by our participants was ‘respect,’ which included mentions of respecting subjects (P10, P34, P41) along with the research environment (P10, P21) and colleagues’ work (P10). While the quotes shared here represent mostly researchers from our participant pool, in the whole corpus of data, participants from nearly every geographic region and represented discipline echoed similar perceptions of the need for strong value mechanisms to protect quality in the scholarly knowledge production process.

Know your field Because reproducibility is considered essential to science, our participants were adamant that a scholar who knows their field will be better equipped to produce quality research, stating “Academic research… would ask you to have both knowledge of the past but also build new things for the future” (P31). In this sense, quality was couched as “fitting your study in with the rest of the field” (P08) and “a good quality article should be able to build on some sense of some people in the past and have their own new ideas that help us to understand the phenomenon or explain the phenomenon” (P31). In essence, quality research “grounds itself,” (P07) is “well fitted into the literature” (P09), and contributes to “building up incrementally” (P18) the body of knowledge. All the comments presented in this theme reflect the processual nature of research inquiries. Putting new research into conversation with the existing body of knowledge, aligning with Harvey and Green’s notion of transformation, was a core principle of quality that seemed to resonate with participants from all regions and professions in our sample.

Research that has been deeply thought about This theme shows a dedication to the internalized process of knowledge creation, where “the researcher has to be curious on something” (P27) and “also a lot to do with a kind of passion” (P34). Additionally, there was a sense that researchers needed to take the time to produce research that “has been thought over and discussed” (P06) and that “Ultimately [the researcher] can still stand behind what [they] would have wrote then 10 years ago” (P34) so that the research “can create a dialogue with the existing literature” (P46) and continue to build upon the existing knowledge base. This notion did not appear to have regional boundaries, suggesting that individuals who pursue scholarly research as a career are dedicated to embracing deep thought processes, regardless of their ethnic background or specific regional affiliation.

4.2 Challenges

When participants discussed the challenges associated with producing quality research, their responses fell into a broad range of categories, including: ‘Producing knowledge in a global environment,’ ‘inconsistencies in evaluation methods,’ ‘getting rejected,’ ‘needing to sensationalize findings,’ ‘willful disregard of ethical practices,’ ‘valuing quantity over quality,’ and ‘pressure to publish’ (see Fig. 2). In contrast to the defining characteristics of quality addressed above, the challenges related to producing quality scientific knowledge align more closely with quality as ‘fitness for purpose’ or ‘value for money’ as defined by Harvey and Green [6]. Through these challenges, as identified by our participants, we begin to see the rift between institutional perceptions of quality and individual researcher conceptualizations of quality. This rift, as presented through our interview results, expands Harvey and Green’s conceptualizations, which have largely been conceived, developed, and applied from institutional, rather than individual, perspectives. Furthermore, in participants’ perceptions of challenges, we begin to see important differences among scholars working in different regions.

Fig. 2
figure 2

Alignment of emergent challenges with Harvey and Green's (1993) conceptualizations of quality. See Codebook (Appendix 2) for full Emergent Theme descriptions

Full size image

4.2.1 Quality as fitness for purpose

Conceptualizing quality as fitness for purpose emerges from the user’s perspective, revealing multiple, constantly changing purposes that can be at odds with each other. These conflicting purposes lead to challenges addressed by our participants in the five themes elaborated below and summarized in Fig. 2.

Producing knowledge in a global environment In this theme, participants discussed the challenges of navigating conflicting disciplinary expectations on national, regional, and global scales. A Southeast Asian academic researcher summarized this perception when he noted that “quality is associated with the WesternFootnote 2 world” (P41), adding that while he was not necessarily opposed to this global view, other regions are producing high quality work that is not being recognized because “they don't meet the kind of processes that the Western publication industry is familiar with” (P41).

This same issue was also addressed by one European participant who stated, “international publishing means Western publishing, and I'm very critical against this stuff” (P35) and then offered the example of Latin America, a region this participant suggested has:

…their own research cultures, and they are very good in building their own databases and own publication networks… in Latin America, they developed a very good and working network… we can speak about Latin American research integrity because these articles meet their standards. When you are not Latin American, it will be different for you. (P35)

‘Resources’ emerged as a significant sub-theme, capturing the challenges faced by entire disciplines or scholars within a specific country (P18). For example, a researcher in an African country said she relies on a colleague in the U.S. to send her research articles because her institution does not provide access to journals she needs to cite in her research (P03). This notion of inequities in resources was also addressed by a North American researcher who asserted that even a supposedly global organization like the International Communication Association (ICA) often perpetuates biases (P34). Most participants who addressed the challenge of producing knowledge in a global environment were working in, or were originally from, regions outside the Euro-America sphere. The few Euro-American participants who mentioned the challenges of producing knowledge in a global environment tended to be publishing or consulting professionals who routinely worked with researchers or journals in non-Euro-American regions and had witnessed the challenges faced by scholars in other regions.

Inconsistencies in evaluation methods Academia, by its very nature, encompasses all fields, meaning that one approach that works for a certain field may not work for another (P08). But a bigger problem identified by several scholars from various parts of the world, expressed succinctly by a participant from Eastern Europe, was that “there is some assessment system, but nobody cares” (P35). Another scholar from Southeast Asia noted their national system of ranking publications “only look[ed] at the ratings” (P19), before continuing to describe how this has lowered the production of quality research because so much focus has shifted to journal rankings.

Some participants also discussed the intricacies of their national standards and the influence of the government in scholarly research practices. For example, one scholar in a STEM field who has worked in Africa and Asia compared the influence of government policies in two countries, noting that in the African country there was an increase in publications in predatory journals after a new policy emphasized quantity of publications, whereas in the Asian country, “they have a ranking…, which is developed by [the National Academy of Science]…So just publishing in a predatory journal, your articles is not going to be recognized … So, people here avoid predatory journals because of that” (P10). Here again, it was primarily participants working outside the Euro-American sphere who noted the differences in evaluation systems and how it affects their ability to produce quality scholarly knowledge. This observation seemed especially salient for individuals who have worked in multiple regions.

Getting rejected This theme captures rejection as a constant occurrence in academic life and its impact on research conduct. Although it may seem like this fear of rejection would encourage quality in knowledge production, one participant noted it may have the opposite effect, as it could lead desperate scholars to turn away from traditional, reputable publishers, and instead, “just rely on predatory journal” (P03). Getting rejected unfairly was especially salient for scholars working outside Euro-American regions, as summarized by an Eastern European scholar who stated, “when you submit your paper from Nigeria, and the same paper is submitted claiming that it is from Harvard, the referees will extremely overvalue the paper [from Harvard]” (P35). Almost all the participants who shared perceptions of unfair rejections based on national affiliation were from non-Euro-American regions.

Conversely, scholars from Euro-American or affluent Asian regions tended to couch rejections in a more positive light: “even if we didn't publish at that journal that rejected us, we used that feedback to make the article better and better” (P41), a viewpoint agreed upon by another participant who acknowledged that despite the emotional difficulty of rejection, reviewer comments often help improve the manuscript (P48). As with other themes related to quality as fitness for purpose, regional differences emerged in how the participants viewed the challenges they were facing.

Needing to sensationalize findings This theme captures the potential of over-sensationalizing findings as a challenge to producing quality science (P15, P38)—in one participant’s words, a “preponderance of people just wanting cool findings that you just throw at whatever high-impact journal” (P45). One senior publishing professional working in Europe spoke of the need to instill a sense of personal responsibility in junior scholars “not to oversell their studies, to be honest about what their study really showed, not the things that they wished it would have shown you” (P09). A European researcher agreed with this sentiment, urging junior scholars to not succumb to “temptations to maybe make the story a bit more streamlined…, or to make it a bit more simple, a bit easier to understand” (P38).

But the trend of only publishing research that supports hypotheses or that is statistically significant is not just an issue for authors to address—it is a problem for journals and publishers as well. In the sub-theme ‘importance of null results,’ participants spoke of the additional pressure from journals to only publish work that supported hypotheses, rather than understanding that “sometimes negative results can be very important as well” (P26) and that “a lot of times an advance comes off something negative, not off something positive” (P08). One participant said that feeling pressured to sensationalize their work and not publicize negative results is “the most harmful thing probably for scientific progress… Because a null finding can be much more revealing than a finding” (P48). This theme emerged from comments throughout our participant sample, in contrast to other challenges that appear to have regional differences.

Willful disregard of ethical practices Aside from the disciplinary and institutional pressure expressed by participants in our study, an added complexity that a handful of individuals discussed was the reasons why scholars engage in unethical research practices. Participants in the Euro-American regions tended to couch unethical behavior in softer terms, saying “people make mistakes. But there's a lot of … intentionally unethical behavior, because there's so much pressure to chase the money, and so if you're under a lot of pressure to chase money, yeah, you cook your data” (P25). Conversely, a participant from Africa chalked it up to “laziness,” sharing the practice of “faculty members here in [my country], they just pick their student's project work and modify it as a journal article and submit,” noting that “sometimes they put the name of the student along, sometimes they don't” (P42). One South American participant shared a story of a colleague who knew they were submitting to a predatory journal, but did it anyway, ending the anecdote by saying:

My impression is that perhaps at that moment he was so unhappy with the system. He was so unhappy with the way reviewers were dealing with his submissions. …I think that when he found that somebody was appreciating his work …he simply was happy to have those ideas spreads through that journal. I think that sometimes they know that there is something wrong, but the fact that somebody accepted the work, … people like to have their work published. (P13)

Ultimately, despite participants from different regions giving different reasons for researchers to disregard ethical practices, there was universal agreement that the human element of honesty could fail during knowledge production, damaging the credibility of knowledge.

4.2.2 Quality as value for money

For our participants, challenges coded as ‘valuing quantity over quality’ and ‘pressure to publish’ align most closely with Harvey and Green’s conceptualization of quality as value for money (see Fig. 2). As elaborated below, these themes reflect an understanding of quality as something that can be quantified and brings value to the institution.

Valuing quantity over quality The complicated interaction between personal ethical values and institutional pressures was especially salient as participants discussed the trend in academia to value the number of publications, rather than the social benefit, practical application, or other less quantifiable markers of quality. This concept was addressed from a range of angles, including scholars from multiple regions (P02, P06, P08, P22, P24, P32, P38, P47) noting cultures of publishing in which the publication venue does not matter, summarized by one North American participant’s statement that “in some areas, unfortunately, they're mainly counting publications not thinking about quality and so you can just publish a lot and you're considered productive” (P08). An Eastern European scholar shared arguments from his students, noting when he tried to guide them towards publishing in reputable journals, “they say, ‘Oh, why should I take 100 or 1000 h for developing a good paper, when I can just write something in low quality, but my university will accept it as international? Why should I care?’” (P35).

Others spoke about the focus on quantity from a resource perspective, noting “there are really perverse incentives in academia…with this idea of just maximizing a very poor proxy for quality, just because it's quantifiable,” an idea they continued with by saying, “quality has become…a question of numbers…the number of publications per academic” (P22).

Despite the vastly different publishing expectations across the globe, participants from multiple regions mentioned the growing emphasis on numbers of published manuscripts because of the relative ease in measuring indicators such as journal impact factor and citation counts.

Pressure to publish The valuation of the number of publications, rather than social benefit or practical application of fewer articles has led to a culture where scholars constantly feel they are in a race to “fling [manuscripts] to a journal… rather than taking the time and the effort to really prove yourself wrong” (P45). This pressure to publish was especially salient to some participants whose national systems required them to publish before their degree could be conferred (e.g., P03, P15), placing publications as a “kind of the currency of the field” (P08). The pressure to publish original research was a challenge articulated by participants from nearly all our represented regions, suggesting it is a global phenomenon in academia.

In terms of systemic challenges, the so-called ‘replication crisis’ (P47) emerged as a powerful sub-theme, with both researchers and publishers noting that placing more value on replication studies could increase the quality of research. One participant noted researchers’ wasted time because “people just keep doing the same stuff because they only have the articles published that did work” (P43). A publishing industry professional emphasized that journals should be more open to publishing replication studies (P38), a perception balanced by another participant who noted the difficulty journals face in providing all the materials scholars need to reproduce studies (P08). This challenge was expressed almost entirely from scholars in the Euro-American regions, suggesting that for scholars working in the highly competitive and often well-funded regions, they did not feel empowered to replicate studies to build on existing knowledge—rather they felt pressured to continually find new and unique ways to conduct research. Ultimately, many felt this was a disservice to the scientific method because attention was continually being diluted to new projects, rather than working on deeper understandings of specific issues.

5 Discussion

Previous studies provide valuable insights into research quality, but these are fragmented by disciplines or geographic regions. Our study provides a more comprehensive examination of the defining characteristics of quality in scholarly knowledge production, as well as challenges to quality, through the lens of Harvey and Green’s [6] conceptualizations of quality.

In assessing the defining characteristics of producing quality research, participants’ comments aligned with some previous literature, such as ethics and rigor in Tracy’s [19] ‘Big-Tent’ criteria. Interestingly, the defining characteristics of quality were largely consistent across our participants from various regions and professions. This suggests that despite vast differences in profession, national affiliation, or even stage of career, some core principles of quality in scholarly knowledge production were identified by our participants. In contrast to previous research, and expanding on Harvey and Green’s conceptualizations of quality, our participants delved more deeply into the human elements of producing quality research, calling for depth of disciplinary knowledge and willingness to spend time thinking deeply and critically. This suggests that researchers understand the “human element” of producing quality scholarly knowledge necessitates a combination of technical and value mechanisms.

Participants’ comments about defining characteristics of quality align most closely with Harvey and Green’s conceptualizations of ‘quality as perfection or consistency’ or ‘quality as transformation.’ This is important because these conceptualizations entail processes, rather than outcomes, and require that institutions trust the individuals who produce knowledge because it is challenging to standardize and assess such a fluid and relative process [7]. For example, the processes that participants emphasized vary greatly among, and even within, disciplines. Furthermore, some characteristics, such as transparency, can be difficult to standardize or evaluate, although the concepts behind Open Science may provide a guide for institutions desiring to place a greater importance on transparent research practices [31] as a way to increase quality in knowledge production. Thus, despite a growing trend toward institutional assessment, and evermore-complex methods of evaluating researcher productivity, our participants adhered to ephemeral aspects when discussing characteristics of quality knowledge production. This finding suggests a need for more attention to non-quantifiable components of knowledge production in graduate student education and emphasizing the value of ethical and transparent research activities, aligning with Harvey and Green’s [6] supposition that quality is multi-dimensional and requires consideration of numerous factors when considering scholarly knowledge production. Participants’ insights into defining characteristics of quality deepen our understanding of Harvey and Green’s conceptualizations by revealing a contradiction: on the one hand, scholars are trusted to adhere to conduct research properly, but on the other hand, scholarly discourse conventions such as the expectation of a detailed methods section in a scholarly article require scholars to prove that they have behaved properly.

As for challenges to producing quality knowledge, some differences emerged between our findings and existing literature, where challenges are often siloed into field-specific discussions [22, 23, 25, 29, 31]. Instead, our participants spoke of challenges more broadly, drawing attention to difficulties of global knowledge production, inconsistencies in institutional or departmental evaluation methods, and challenges related to an ever-increasing pressure to increase publication and citation numbers. In terms of Harvey and Green’s conceptualizations, the challenges to producing quality scientific knowledge aligned with either ‘quality as value for money’ or ‘quality as fitness for purpose.’ Interestingly, those two conceptualizations of quality are most frequently used by institutions to evaluate outcomes, including satisfaction of services delivered [e.g., 10, 48, 49]. Yet Harvey [7] argues that these two conceptualizations, despite their wide use, are more about quality assurance, and are outcome based, rather than process based. Therefore, our study suggests a disconnect between institutional perspectives of quality and perspectives of individuals who create new knowledge. Specifically, participants expressed that they feel limited by institutional evaluation methods, whereas institutions feel compelled to standardize assessment rubrics to facilitate fairness and equity [e.g., 50]. Furthermore, regional differences emerged when participants discussed challenges to producing quality research. For example, only participants outside the Euro-American regions spoke of challenges in terms of ‘producing knowledge in a global environment’ and ‘inconsistencies in evaluation methods.’ Conversely, regarding ‘getting rejected,’ participants from outside the Euro-American sphere were quick to identify the negative and possibly discriminatory aspects of rejection, whereas participants in Euro-American and affluent Asian countries took note of the positive side of rejections and how criticism of one’s work by peers can ultimately make it better. Participants uniformly agreed on an underlying need to sensationalize their work, largely due to an institutional emphasis on quantity over quality that has resulted in a universal perception of a constant pressure to publish. When participants addressed why researchers may knowingly disregard ethical practices when conducting research, participants from different regions offered up different reasons, suggesting a need to explore this phenomenon further. Also noteworthy was the Euro-American scholars’ emphasis on a “replication crisis” in scholarly knowledge production (e.g., journals rejecting replication studies in lieu of new original research). This challenge was not addressed by researchers outside the Euro-American regions, suggesting this may be a regional challenge and/or perception.

6 Conclusion

By moving from field-specific conversations to a broader conceptualization of quality in scholarly knowledge production, our study emphasizes that quality cannot be easily quantified or relegated only to morals and values. Quality in scholarly knowledge production, for our participants, requires mechanical and technical expertise, and a commitment to high standards of honesty and integrity—even when no one is looking and there is little chance of getting caught. Although scholars are generally trusted to adhere to these ethical standards during the conduct of research, scholarly discourse conventions, such as the expectation for a detailed methods section, enforce such adherence by requiring scholars to demonstrate that they have behaved properly. In addition, our findings indicate a rift between institutional and individual perceptions of quality. The stakeholders who produce new knowledge value processes as quality indicators, aligning with Harvey’s supposition that “quality is dynamic and about change,” and believe that the transformation that occurs with knowledge production cannot be captured with the “static state evaluation” [7] inherent to Harvey and Green’s other conceptualizations of quality. Thus, we conclude that the multi-dimensional conceptualization of Harvey and Green’s notion of quality does help orient our understanding of quality in the scholarly knowledge production process. Through applying Harvey and Green’s conceptualizations, we can see that the producers of scholarly knowledge (researchers and graduate students) appreciate and embrace the processual nature of quality as explained through Harvey and Green’s dimensions of quality as ‘consistency or perfection’ and quality as ‘transformation.’ Conversely, the producers of scholarly knowledge feel that the indicators of quality most readily embraced at an institutional level are ‘fitness for purpose’ or ‘value for money,’—perspectives they view as challenges to maintaining or achieving quality in the knowledge production process. This suggests the need for additional research to align the indicators of quality at the individual and institutional levels. While the mechanisms of Open Science (e.g., open data, preprints, open peer review; 43) may begin to heal that rift, this study shows that there is still much work to be done.

Study limitations include sample size; 48 is hardly representative of the millions of scholars producing knowledge across the globe. In addition, we only accepted participants who could be interviewed in English. It is hoped that future research will address these limitations through qualitative or quantitative research that continues to illuminate research quality as conceived by the stakeholders who produce new knowledge.