Introduction

If you are reading this, this paper, like many, many others, has probably passed a so-called plagiarism scanner. Plagiarism scanners are text mining algorithms that check for overlap between a given text and a corpus of reference texts, and they are widely used in academic publishing. In addition to plagiarism scanners, there is a diverse set of other tools and algorithms that screen papers for potential issues with research integrity: Algorithms that scan images for duplicated areas (Acuna et al. 2018), R packages that check for errors in statistical analyses (Nuijten et al. 2016), or tools that flag problems in the reporting of research protocols (Menke et al. 2020). These tools promise to improve the integrity of the academic literature, foster research quality, and help preventing scientific misconduct. Their purpose is to monitor and enforce adherence to rules of good scientific practice and research integrity, which makes them similar to other forms of algorithmic regulation (Hildebrandt 2018; Yeung 2018) or algorithmic governance (Beer 2009; Musiani 2013; Katzenbach and Ulbricht 2019). As Introna remarked for the case of plagiarism scanners, they contribute to “the algorithmic governance of academic writing on an unprecedented scale” (Introna 2016: 31). Used widely, they hold the potential to shape and re-structure the body of published academic literature, and as such carry far-reaching normative implications. Gaining a better understanding of the technological, epistemic, social, and normative preconditions and implications of automatic screening in academic publishing thus seems highly relevant.

Currently, however, while plagiarism screening of students’ papers in the context of university teaching is widely discussed (e.g. Introna 2016; Stuhmcke et al. 2016 Stapleton et al. 2012; Zwagerman 2008), we still do not know a lot about how automatic screening tools are used by academic journals (but see Zhang and Jia 2012). In particular, analyses from Science and Technology Studies or the Sociology of Science of how and to which effects automatic screening tools are used in academic publishing are still missing. Given their wide-ranging potential as technologies of governance, this lack of scholarship is somewhat surprising. The present contribution aims to address this by taking a closer look at the normative implications of screening tools used by editors of academic journals. It builds on Introna’s (2016) seminal work on the plagiarism software “Turnitin” as an algorithmic actor that contributes to the overall governance of students’ academic writing practices. As technologies of government, screening tools are embedded in normative concepts and ideals and reflexively also contribute to these concepts. Employing the analytical lens of the “digital imaginary” (e.g. Mager and Katzenbach 2020), this contribution investigates the normative concepts that play a role in journals’ use of automatic screening. It draws on data reflecting both publishers’ normative expectations about the use of automatic screening (publisher policies, training materials, and COPE discussion documents) as well as journal editors’ accounts of their employment of automatic screening tools or their opposition to it (open-ended survey answers). As such, this contribution is not concerned with the actual prevalence of research integrity violations such as plagiarism in the academic literature (e.g. Gupta et al. 2021; Kalnins et al. 2015), nor does it seek to evaluate whether editors’ perceptions accurately reflect this prevalence (for a critical discussion of the concept of prevalence, see Hesselmann et al. 2014), or how their use of screening tools could be improved (e.g. Taylor 2017). Its goal, again following Introna (2016: 31), “is not to take a normative view on the issue of plagiarism [and other forms of misconduct] as such”. Instead, the contribution traces how editors normatively situate automatic screening by developing two opposing digital imaginaries of academic publishing: One that portrays academic publishing as a closely-knit community of trustworthy people, and one that sees it as a vast and dangerous space. Within these two imaginaries, editors carve out different modes of control that entail different positions of power for themselves. These imaginaries are shaped by the social and epistemic characteristics and the predominant publication cultures in different academic fields.

Digital Imaginaries and Automatic Screening Tools for Research Integrity

Following Introna (2016: 33), automatic screening tools are approached as “mundane technolog[ies] of government”. As governmental technology, automatic screening tools do not stand alone, but are connected to specific discourses, visions, and rationales. They are embedded in “changing discursive fields within which the exercise of power is conceptualized, the moral justification for particular ways of exercising power by diverse authorities, notions of the appropriate forms, objects, and limits of politics, and the proper distribution of such tasks […]” (Rose and Miller 1992: 175). These discursive fields and moral justifications constitute “sociotechnical imaginaries” (Mager and Katzenbach 2020; Jasanoff and Kim 2009, 2013) that are defined as “collectively imagined forms of social life and social order reflected in the design and fulfillment of nation-specific scientific and/or technological projects” (Jasanoff and Kim 2009: 120). Sociotechnical imaginaries are highly normative and comprise ideas about what is good and desirable for human welfare (Jasanoff and Kim 2009), as well as about who rightfully has authority and agency (Mansell 2017: 43). As collective visions of the “goods and bads” (Jasanoff and Kim 2009: 124) of technologies, they are, however, not about accurately assessing the specific risks or potentials associated with the development or use of technologies, but rather about articulating possible desirable futures for a social collective in relation to technological development. Sociotechnical imaginaries are often related to symbolic manifestations of the nation state (e.g. Jasanoff and Kim 2013; Burri 2015), or understood as “associated with active exercises of state power” (Jasanoff and Kim 2009: 123). However, they can also be seen as performative visions of other social communities besides the state (Moon 2015: 174). Building on this broader understanding of imaginaries, concepts such as the “digital imaginary” (Mansell 2017; Wahome and Graham 2020) or the “algorithmic imaginary” (Bucher 2017) specifically address the visions, understandings, and normative assumptions surrounding digital technologies and their everyday use.

The present case concerns the digital imaginaries that shape the use of automatic screening tools by journal editors. Importantly, these imaginaries revolve around questions of scientific norms, control, and governance. While not connected to a nation state or exercises of state power, they still concern visions of the academic community as a social collective and address questions of how social order in academia can and should be shaped by technology. Using screening tools in editorial practice both presupposes and produces specific conceptualizations of what kind of community academic publishing is and should be, what problems it faces and how and by whom these problems ought to be addressed. It is thus connected to and actualizes imaginaries that are located at the intersection between various strands of discourse about research integrity and scientific misconduct, academic publishing more broadly, and the automation of control. In this context, this contribution asks which imaginaries of academic publishing editors invoke when discussing their use of automatic screening (or lack thereof), how these imaginaries (de-)legitimize automatic screening, and how and where they assign authority and agency. The goal is to gain a better understanding of the political underpinnings and effects that come with the availability and use of automatic screening tools.

Data and Methods

This study employs a mixed methods approach using different types of data reflecting the perspectives of different actors in academic publishing. To capture the view of journal editors, it draws on data stemming from an online survey of international editors conducted in spring 2021. The survey sample was constructed based on the Scimago Journal Rank: Firstly, five subject areas (Arts & Humanities; Education; Engineering; Biochemistry, Genetics and Molecular Biology; Physics & Astronomy) were selected that represented the five broader fields of Humanities, Social Sciences, Engineering, Life Sciences, and Natural Sciences and that were roughly comparable in size. After excluding Conference Proceedings from the data, 500 journals were selected randomly from each area, resulting in a sample of 2,500 journals. Subsequently for the selected journals, the contact details (email addresses) of an editorial representative (typically the Editor-in-Chief) were researched via web search. Excluding missings (cases where no email contact could be identified, invalid emails, and bounces) the final sample consisted of 1,873 editorial contacts (one contact per journal) that were invited to participate in the survey. Of those, 225 respondents completed the survey, amounting to a response rate of 12.02%. However; for the qualitative approach of the present analysis, the sample was restricted to those 173 respondents who answered at least one of the two open-ended questions that are analyzed here. As such, this sample cannot be considered representative and should not be used to derive quantitative estimates about either the use of automatic screening or the quantitative distribution of attitudes towards automatic screening among journal editors per se.

The questionnaire (see Appendix II) contained questions relating to journal characteristics, questions about the type of automatic screening used at the journal, and questions about attitudes regarding the use or non-use of screening tools. In addition, it included two open questions: One following the question “Do you think that automatic screening should be used more widely to ensure research integrity?”, which asked respondents to briefly explain their response, and the other one as a field for further comments at the end of the survey. Overall, there were 254 separate answers from 173 respondents.

Additionally, to provide more contextual information about the use and regulation of automatic screening in academic publishing, publicly available policy and training documents about automatic screening by 5 major publishers (Elsevier, Springer Nature, Wiley, Taylor & Francis, Sage) as well as by the Committee of Publication Ethics (COPE) were collected. Publishers are important actors in the field who can be considered a major factor in whether and how individual journals use automatic screening. Publishers are also usually the ones providing access to screening tools for journals, especially since they mostly cover licensing fees and similar associated costs. Together with COPE, they also provide normative guidelines and frameworks for a legitimate use of screening tools by editors, and as such also shape the imaginaries around automatic screening that are expressed by journal editors. These 5 publishers were selected because they are commonly viewed as the biggest academic publishers, something that was also reflected in the fact that about half (52%) of the editors whose responses were analyzed here worked at journals that belonged to one of those publishers. Documents were collected by searching the publishers’ website for a general Publishing or Editorial Policy. For four publishers, an integrated Editorial or Publishing Policy could be found, while one publisher had several more specific resources on Publishing Ethics. Additionally, Google searches for “publisher’s name + iThenticate” and “publisher’s name + Similarity Check” were performed, and all relevant documents and files on the respective publishers’ websites were downloadedFootnote 1 (please see a full list of documents analyzed in the Appendix I).

All data were analyzed following a Grounded Theory methodology (Corbin and Strauss 1990): First, the open answers from the survey were coded inductively, resulting in a preliminary coding scheme. Selected accounts from all codes were then analyzed more in-depth following a line-by-line sequential analytic method (Maiwald 2005) and resulting in a refined coding scheme. Subsequently, applying a strategy of axial coding, codes were grouped into four and then two larger themes that related to the overarching question of imaginaries of the academic community. For the publisher and COPE documents, a content analysis was first performed to identify the topics covered by the guidelines and training materials, as well as to retrieve any information publishers provided about the deployment of screening tools at their journals. All relevant sections from the publisher and COPE documents were then analyzed with regard to the two themes previously identified in the survey data, in order to compare the perspectives of editors and publishers on automatic screening.

Automatic Screening Tools in Publishers’ Policies

Automatic screening tools in the publishing process are typically employed before or during the peer review process in order to flag submissions that contain grave errors or issues with research integrity. The goal is to identify and address potential problems before manuscripts get accepted and published. In principle, automatic screening tools for research integrity represent a highly heterogeneous group of software tools and programs that can be used to screen academic publications. While some screening tools aim to detect violations of research integrity such as plagiarism (e.g. Turnitin, Zhang 2010), others target more benign issues such as inadvertent statistical mistakes (e.g. Statcheck, Nuijten et al. 2016), or poor or intransparent reporting of research protocols (SciScore, Menke et al. 2020). The term encompasses different kinds of tools, such as text mining algorithms, image processing algorithms, or R packages.

In practice, however, the most common screening tool in editorial work is a program for detecting text overlap (often dubbed plagiarism detection) called Similarity Check, powered by the software iThenticate. This program was developed in cooperation between the software company Turnitin and the publisher initiative CrossRef. It is widely known and has become somewhat emblematic for automatic screening tools in general: Software to detect text overlap is the only screening tool that is explicitly mentioned in the publishers’ Publishing Ethics Policies analyzed here: Out of the five publishers, four explicitly state in their policies that their journals may use plagiarism software to check submitted manuscripts for overlap with previously published material, the remaining publisher did not have a comparable policy document. No other software checks are mentioned in any of the Publishing Policies.

It is typically through the publisher that a journal’s editorial team will get access to Similarity Check via the publisher’s membership in Crossref, and also the publisher who will cover the costs for the software. At many publishers, Similarity Check is integrated into the journals’ editorial management system, from where it can run automatically on submitted manuscripts. Additionally, it is also possible to access the tool manually through an individual account, and to (re-)analyze a manuscript at various stages in the peer review process as well as post publication. While publishers provide access to the tool to their journals, they still allow journals a number of options as to when and how they use Similarity Check:

“With plagiarism detection software (e.g. CrossCheck's iThenticate) manuscripts can now be checked quickly and any overlap can be quantified. Each journal with plagiarism software should consider how to use the software most effectively. Some check every manuscript, some only perform random checks, and others only perform a check when the possibility of plagiarism is raised by a reviewer. Given that these checks are quick to perform, our recommendation is to check each new submission.” (Wiley: Plagiarism and Libel)

“All new submissions to many Elsevier journals are automatically screened using Crossref Similarity Check within the editorial system.” (Elsevier: Plagiarism Detection)

Still, as can be seen in these quotes the publishers often implicitly or explicitly express a preference for screening all manuscripts upon initial submission and advertise tools as (supposedly) quick ways to check. Such an emphasis on the speed and objectivity of automatic tools corresponds both to more general discussions about automation and can also be found in the guidance on automatic screening in editorial practice by the Committee of Publication Ethics (COPE). In the COPE discussion document “Artificial intelligence (AI) in decision making” it says:

“The AI and automation tools being developed have the power to help with the speed and accuracy of peer review. Software created to detect text overlap provides a level of assessment, by cross checking millions of documents, that a human brain could not achieve.” (COPE 2021: 4)

This emphasis on the ability of automatic software to quickly process large volumes of data goes hand in hand with an image of the vastness and fast-paced growth of the academic literature at large. This idea of the academic literature as an almost unimaginably large space is prominently invoked in publishers’ information about Similarity Check, for example by stressing the amount of texts that Similarity Check screens against:

“Over 200 CrossRef members, including Elsevier, collaborate by donating full-text journal articles and book chapters to create a unique database of over 50 million articles. Note that even this database is not entirely exhaustive: research published by non-participating publishers or before the digital era may be absent. Elsevier's contribution consists of 10 million articles and 7000 books and is ever-increasing as all newly published articles are added.” (Elsevier: Plagiarism detection)

When integrated into the editorial management system, Similarity Check can often run automatically in the background without the users (e.g. editors) being able to stop the screening. However, once the screening is completed, there is no automatism or obligation on the part of the editors to act on the results. All of the recommendations issued by publishers and COPE in fact in some way or another emphasize the need for human involvement and interpretation when using automated tools:

“At this point in time in the development of AI and the evolution of mindsets, we recommend that if an editorial decision is made by AI that provides a final outcome for an article, such as acceptance or rejection, the decision should directly involve an editor. The decision cannot be made by an AI tool alone.” (COPE 2021: 6).

“The web-based tool can be used in the editorial process to identify matching text but it cannot, on its own, identify plagiarism. Manual examination of the matching text is still required and judgment used to identify if plagiarism has occurred or not.” (Springer Nature: Plagiarism prevention with CrossCheck)

Depending on the respective editorial management system, the screening is sometimes indicated in such an inconspicuous manner (as a single line in a long sidebar menu) that it is easily overlooked (be it willfully or accidentally), so that editors might easily simply ignore the screening results. Similarly, while publishers provide additional information and resources for editors on how to use Similarity Check and interpret its results, such as handbooks and user guides, as well as training videos and seminars, editors also aren’t bound to follow them. As such, while publishers can be considered a major factor in whether and how Similarity Check is used in the editorial process, journals and editors retain a degree of freedom as well. Here, the imaginaries associated with the use of automatic screening tools are highly relevant, as they at least in part shape if screening tools are actively used within the peer review process, as well as provide a normative context for editors’ interpretation of the tools’ results.

Automatic Screening Tools in Editors’ Accounts

As discussed above, considering the relatively low response rates, the editorial survey should not be treated as representative. The quantitative results should not be interpreted as reflecting the use of and attitudes towards automatic screening among journal editors in general, but only as a description of the sample at hand and as a contextualization of the qualitative analysis.

For the purpose of this analysis, the sample was restricted to those 173 journal editors who answered at least one open question. This sample consists of 33 (19.08%) editors from Humanities, 49 (28.32%) editors from Social Sciences, 34 (19.65%) editors from Engineering, 19 editors (10.98%) from Life Sciences, and 38 editors (21.97%) from Natural Sciences, making the Social Sciences overrepresented, while Life Sciences were underrepresented in the sample. 141 respondents (81.50%) were the Editor-in-Chief of their journal, reflecting the search strategy that focused on EiCs as the primary contact. The median time respondents have been in their editorial position was 6 years (mean 7.45 years).

With regard to automatic screening tools, 127 (73.84%) respondents answered that some type of automatic screening was used at their journal, either by the respondents themselves, or by a member of their editorial team. 125 respondents said that plagiarism software was in use at their journal (either by the respondent themselves or by a member of their editorial team). In contrast, software for screening images was used at only 21 journals, and software to screen statistics only at 17 journals. Respondents were also asked the name of the specific tool(s) they used. Here, the use of Similarity Check stood out in particular: 87 of the 100 respondents who provided information here named this specific tool, with all other tools being mentioned less than 5 times each. While it is important to keep in mind that these numbers cannot be considered representative, taken together with the information on the publishers’ websites it seems safe to say that plagiarism screening via Similarity Check is the most common form of automatic screening in editorial practice, while other tools are rather marginal phenomena.

The use of screening tools varied by academic field. While a vast majority of journals from Engineering (91.18%), Natural Sciences (89.47%) and Life Sciences (89.47%) used some type of screening tool, tools were somewhat less common at journals in Social Sciences (62.50%) and much less common in the Humanities (45.45%).

When asked whether they thought that automatic screening tools should be used more widely, a majority of respondents (75.44%) answered yes. Respondents at journals that already used some type of screening tool were more positive about increasing use (82.68% answered yes), but even respondents who currently did not use screening tools slightly favored expanding the use of screening (53.49%). Given the limitations outlined above, both the high rates of current use, as well as the generally positive future outlook on automatic screening must be interpreted with caution, however, as it is probable that editors who responded to the survey were more positive towards automatic screening tools compared to those who did not answer the survey.

Imaginary I: A Safe Community

“Our field is a fairly small sphere of scholars and there has never been a serious case of plagiarism. People have a general sense of who is working on what. So if we received an essay on bus drivers in New York City during the 1970s then our reviewers would have a very good sense of who had previously written on that subject, meaning the ability of someone to pass off someone's previous work as their own is just highly improbable.” (ID 95, Humanities)

The first imaginary that can be identified refers to the vision of a safe academic community. This community is typically described as small and cohesive, almost like a friendly neighborhood where residents all know each other. Respondents who oppose an increased use of screening tools exclusively refer to this imaginary. It is found most frequently among respondents from the Social Sciences and especially the Humanities (see Table 1), who are also much less likely to report screening tools being used at their journal.

Table 1 Distribution of Imaginaries between Academic Fields

Here, use of phrases like “our field”, “in our humanistic field”, “our journal”, “journals such as mine” indicate a strong sense of belonging, coinciding with a strong sense of boundaries or delimitation against other research areas:

“In the Humanities, I simply don't see the point. It is easy enough to identify plagiarism because particular scholars have a history of writing about particular issues. Furthermore, because our journal covers a specific set of themes/regions, authors have to write 'for' the journal, meaning it would be unlikely that someone could just submit a plagiarized essay and have it pass our general review.” (ID 179, Humanities)

Research and publishing are said to take place in a tightly knit community that is both social and epistemic. Researchers are typically expected to know and recognize each other’s work, which also indicates a strong identification between people and their research topics. In such a community, automatic screening is seen as superfluous. In general, violations of academic integrity are believed to be very rare:

“I do not believe plagiarism is a widespread practice in our field.” (ID 151, Humanities)

In part, this low level of deviance from accepted research practices is attributed to efficient social control within the community. As described in both quotes above, social control is achieved through the personal and epistemic relationships that connect the members of the community (“our reviewers would have a very good sense of who had previously written on that subject, meaning the ability of someone to pass off someone’s previous work as their own is just highly improbable”). This high level of social control then serves both as a deterrence mechanism as well as a reliable way to identify and exclude those few instances where good research practices might be violated. Additionally, some respondents explicitly mention a high level of trust within the community that makes not only automatic screening but any form of monitoring unnecessary or even inappropriate:

“Also, there is so much trust in the overall system of creating and publishing research findings that again, checking for plagiarism is not warranted.” (ID 77, Social Sciences)

“There is something called "trust" and perhaps we should value the integrity of our colleagues in the academic community. For the very few who might break norms and standards, the authors have a written legal statement with the publisher, and so sanctions can be made in those terms. As editor of the largest and highest ranking journal in my field […], I have not come across the type of issues that would be captured by your software. I personally do not see it as my job to "police" my own academic community.” (ID 287, Social Sciences)

In this imaginary, ensuring academic integrity is seen as the responsibility of the entire disciplinary community. Against such an emphasis on community-based forms of control, the role of the editor then becomes somewhat ambiguous, as can also be glimpsed in the quotation above: On the one hand, editors as gatekeepers do hold considerable authority in academic publishing, an authority that is also invoked with “As editor of the largest and highest ranking journal in my field”. On the other hand, authority is explicitly rescinded when it comes to exercising control in the context of research misconduct: “I personally do not see it as my job to "police" my own academic community.” As such, automatic screening tools might also be seen as problematic because they potentially change the established distribution of authority between editors and their peers. With this community-based mechanism, ensuring research integrity is sometimes implicitly or explicitly likened to peer review as both a community-driven form of quality control and an overarching mechanism of academic self-governance. In this context, some respondents call for open software tools that are developed by and available to the entire academic community, emphasizing the notion of a community that comes together on decidedly non-economic terms to achieve its goals:

“In case we evaluate such tools in the academic/scientific world as useful, they should be available for free to enable equality across journals, the world, scientific areas, etc. Just as scientific reviews are not being paid.” (ID 285, Humanities)

In addition to these critiques targeting the social consequences of automatic screening, respondents raise a number of critical points that focus on the epistemic aspects of screening. Here, respondents cite the tools’ insufficient flexibility, their inability to put results into context, or their inability to deal with the heterogeneity of submitted articles. Particularly respondents from the Humanities (and some from the Social Sciences), oppose screening tools for epistemic reasons. They criticize that most automatic screening tools do not work well with qualitative or hermeneutic ways of doing research:

“Our journal publishes a lot of qualitative work and many programs don't really deal well with issues relating to qualitative research […].” (ID 35, Social Sciences)

“In our humanistic field screening for plagiarism would be useful but is expensive. Screening for statistical and image manipulation is not desirable. The integrity of an article is evident to the human brain on account of the complexity and plausibility of an argument expressed in language on the basis of evidence. Our evidence is not measured in binary fashion as either ‘true’ or ‘false’ but as a convincing critical explanation of human experience and ideals and values worthy of aspiration. No machine can do this job for us.” (ID 152, Humanities)

More than just a technical issue that affects the tools’ usability in certain fields, for many respondents screening tools are related (and contributing to) much deeper issues and developments that affect the academic community and their fields within it at a larger scale. Especially for respondents from the Humanities, there seems to be a worry that academic publishing and research might become more and more dominated by modes of doing research typically found in the Life Sciences, with other epistemologies being crowded out:

“Automatic software heavily favors scientific based research and marginalizes even further Humanities-based research.” (ID 289, Humanities)

“I plead for more consideration of geisteswissenschaftliche practices, goals, values and publications.” (ID 152, Humanities)

Such a crowding out of certain epistemologies is also often linked to changing modes of research assessments, especially the rise of quantitative forms of evaluation. In this regard, screening tools are sometimes described as part of a much larger problematic development towards quantified methods of research assessment and an increased pressure to publish that is taking place in research and the academic community, even by respondents from outside the Social Sciences and Humanities:

“Younger generation is inclined to evaluate the research quality or journal quality merely from the h-index or citation numbers. From this sense, Automatic Screening Tools may become more popular. However, personally (since I am not a computer-age guy) I hate this tendency, which sometimes hides a very creative and new discovery. […]” (ID 187, Engineering)

The imaginary of the safe community, as well as the specific issues raised by editors about the limitations of screening tools appear to be absent from publishers’ accounts. This might seem unsurprising, given that publishers’ accounts are all generally in support of screening tools and thus do not portray them as unwarranted or even harmful. Still, it indicates that the safe community presents somewhat of a contesting view of academic publishing that is held by a group of editors who are typically from the Humanities and parts of the Social Sciences, and who are critical of automatic screening.

Imaginary II: A Dangerous Environment

In contrast to the imaginary of academia as a safe, trustworthy community in which automatic screening is either superfluous or even problematic, respondents who welcome an increased use of screening tools predominantly paint a different picture:

“[I]n the present scenario, with high pressure-to-publish around the world, it is inevitable that malpractice and wrongdoing increase, and I believe that such tools are essential if we are to maintain the integrity of science and scientific publishing.” (ID 122, Natural Sciences)

In this view, scientific misconduct and violations of research integrity are believed to have increased and are described as serious and frequent problems. The tone and reasoning of many of those accounts express a general sense of worry or wariness. With regard to the social dimension of the scientific community, accounts are characterized by a strong distrust against researchers, who are described as “cheater[s]” (ID 38, Engineering), “offenders” (ID 244, Engineering), “copycats” (ID 203, Engineering), or as “lazy authors who will copy paragraphs from their own previous papers without regard to copyright” (ID 205, Engineering). There is a generalized suspicion against authors as strangers that cannot be trusted:

“[V]ery difficult for editors to know everyone and what they submit if it has good integrity.” (ID 222, Social Sciences)

In particular, respondents stress the sheer magnitude of the task of ensuring research integrity in such an environment. Besides (supposedly) rising rates of misconduct, which creates more work for the editors, respondents also mention rising submission rates for their journals, and, most importantly, a rapidly growing body of academic literature overall:

“Too many published papers to compare with manually.” (ID 117, Engineering)

“The publishing space is too large for me to perform these kinds of tasks manually, even with the large editorial team I have for our small journal.” (ID 256, Natural Sciences)

Respondents frequently use expressions such as “too large” or “too many” when describing this imaginary. The environment portrayed here consists of a vast and boundless publishing space that is unknowable to a human editor, something which can also be found in the publishers’ accounts discussed above. Vis-à-vis this unlimited and ever-growing publishing universe, editors appear as small and overwhelmed, with very limited capacity for control. Control in turn is depicted not only as warranted, but as indispensable for ensuring the integrity of the academic literature. Clearly, this imaginary mirrors the narratives (and data) about the exponential growth of the scientific literature found in other discourses. It also echoes well-established concerns about publication pressures or a culture of “publish or perish” that supposedly drives researchers to employ questionable research practices and, in the worst case, commit misconduct (e.g. see Madikizela-Madiya 2022 for a critical account of this aphorism). Even though the relationship between publication pressure and research misconduct is empirically contested (Fanelli et al. 2015) both in the general discourse and in the present accounts it is mostly treated as an undisputed fact. In referencing these issues, respondents situate their accounts within larger (problematic) developments in academia.

On an epistemic level, with the issue of exponential growth and vastness of the academic literature as the core problem, automation in turn can be framed as the optimal solution because it can analyze large data volumes in short amounts of time:

“Machines are faster than humans and can handle bigger volumes.” (ID 255, Social Sciences)

Here, automatic screening is sometimes described as simply “another tool” (ID 50, Humanities) that supplements already existing tools or modes of control. More specifically though, automatic screening tools are lauded as being able to “save editors and reviewers time” (ID 263, Natural Sciences), as a “quick way to check” (ID 261, Social Sciences), and being able to “speed up the review process” (ID 94, Engineering).

Additionally, some respondents see automatic detection methods as more detailed and more thorough than manual checks, and believe them to be more objective, mirroring the claims made by publishers:

“The use of automatic screening tools surely helps avoiding even minor plagiarisms or self-plagiarisms which may hardly be detected by the editors and reviewers.” (ID 154, Life Sciences)

“Facilitate a faster, more informed, objective, and transparent information to aid in the decision process.” (ID 185, Social Sciences)

Here, with both their supposed speed as well as their supposed objectivity, screening tools are similar to other automated systems, which, as Ruha Benjamin puts it, are “alluring because they seem to remove the burden from gatekeepers, who may be too overworked or too biased to make sound judgements.” (Benjamin 2019: 30). For the most optimistic respondents, screening tools possess a wide-ranging efficacy. This becomes most obvious where automatic screening is believed to not only help detect integrity issues, but is seen as actively changing authors’ behavior and decreasing and preventing misconduct:

“It will change the behaviour of the authors and less misconduct will happen.” (ID 25, Life Sciences)

“Automatic screening helps to avoid plagiarism.” (ID 5, Natural Sciences)

In accounts such as these, the exact mechanisms of how social control is supposedly accomplished remain underdefined. Social control as a highly intricate and complex (social) problem is met with the deployment of an automatic tool, which is said to be both simple and highly effective, but the details of which are hardly discussed. These accounts tie back to the idea that screening tools are “easy and quick” (ID 66), or at least that their desired function is to make editors’ work “simpler and easier” (ID 101). They are constructed as quick fixes to highly complex problems that eliminate the need to address the precise (and maybe unpleasant) ways in which solutions are reached. In the context of exercising social control in particular, this then diminishes the accountability for editorial decision-making and exercise of control.

Editors also seem overburdened by the extent of the literature because in this scenario they assume (most of) the responsibility for ensuring research integrity. Here, tools are presented as a solution that helps editors to re-gain control and authority that they lost. To achieve this, editors place a strong emphasis on retaining most of the control and authority vis-à-vis the tools by stressing that even when using screening tools, decision-making power must still rest solely with editors:

“They should be used to provide more information for human decisions, but not to make those decisions automatically.” (ID 220, Social Sciences)

“Especially, I cannot see why automatic SCREENING would be threatening or "bad" in any way (softwares like iThenticate are a great help), but if the result were automatically USED without double-checking, then that is worrisome, as the repercussions on the authors (and even the reputation of the journal) could be severe.” (ID 114, Engineering)

Introducing such a human in the loop thus allows editors to stay on top of a chain of control in academic publishing, and also seems to make automatic screening more acceptable for respondents who are critical towards it, as it is often mentioned as a condition under which respondents would support the use of tools, even if they currently still express reservations. Again, these accounts about the human in the loop then mirror more general views on automation, human oversight and meaningful human control (e.g. Wagner 2019; Vagia et al. 2016). Still, statements such as “the decision is still with the editors” (ID 258, Engineering) provide only little insights into the actual distribution of labor and decision-making between editors and screening tools. It is thus not clear in which way editors actually retain control, or in how far they might simply follow the tools’ recommendations. Moreover, there is also the question of how the distribution of decision-making authority between editors and publishers might be affected by the use of automatic screening tools. Interestingly, the COPE recommendations on the use of AI and automation in publishing prominently address the duties of publishers in the deployment of screening tools, much more so than those of editors. In addition, they explicitly highlight that “[u]ltimately, the publisher remains accountable for editorial decisions, both made by AI as well as human editors.” Publishers are the ones providing the software, often along with information and training materials on how they should be used. As such, software tools might represent a way in which publishers strengthen their authority vis-à-vis the editors. However, in the accounts by editors analyzed here, the relationship between editors and their publishers is hardly addressed, and publishers do not seem to feature prominently in the imaginaries. While it is obvious that the imaginary of the dangerous environment mirrors both more general discourses about automation and the growth of the academic literature as well as publishers’ more specific recommendations, the imaginary of the safe community cannot be found in publishers’ accounts and seems to originate more from the editors themselves. The relationship between editors and publishers with regard to automatic screening tools is thus a question that warrants further research.

This imaginary is most prevalent in Engineering, the Natural Sciences, and the Life Sciences (see Table 1). These disciplines also express the greatest worry about high and/or rising rates of misconduct: Out of the 19 respondents who said misconduct had increased or presented a wide-ranging problem, 16 are from Engineering, Life Sciences or Natural Sciences. Conversely, ten out of the twelve respondents who said misconduct was a negligible issue in their field are from the Social Sciences and Humanities, only two are from Engineering, and none from the other disciplines. However; the theme of tools as a solution is quite a common theme for respondents of all disciplines. Many respondents refer to the potential of automatic tools to save editors time, and/or to prevent misconduct, perhaps because this concept of automated tools and/or “artificial intelligence” as a quick solution to problems of large amounts of data is so prevalent in the general discourse around automation. When looking at the disciplinary characteristics, interestingly, this theme is by far the most prevalent in the Social Sciences, even though other disciplines, i.e. Engineering, Natural Sciences, and Life Sciences, are more likely to employ screening tools in the first place.

Discussion

Through the qualitative accounts of automatic screening in academic publishing, two opposing imaginaries emerge: The first imaginary portrays academic publishing as a mostly small, tightly connected academic community, in which everybody knows and trusts everybody else, informal social control is already very high and scientific misconduct is believed to be very rare. Control is exercised by the community at large, by reviewers, readers, and other researchers.Footnote 2 Even though editors clearly possess a gatekeeping authority in academic publishing, they thus mostly reject the authority and responsibility to control authors for possible misconduct. This imaginary is most prevalent among those who oppose expanding the use of tools and is connected to the notion that automatic screening tools are simply not warranted to exert control. Even more, screening tools are themselves frequently conceptualized as a factor contributing to the further development of a vast, economized, and dangerous publishing space.

The other imaginary consists of academia as a vast and ever-growing anonymous space dominated by a focus on quantitative output and an economic logic, where mistrust against other researchers is high and misconduct presents a serious, almost overwhelming problem. This imaginary then provides the background for welcoming screening tools, and automation in general, as the only solution to grapple with the vastness of literature and increasing violations of research integrity. In its most pronounced form, this approach ascribes an almost unquestionable efficacy in reducing and preventing misconduct to screening tools. In terms of assigning authority for control, in this imaginary, editors are the ones who are assigned primary responsibility to exercise control, and their ability to do so is seen as threatened by rises in misconduct. Tools are then constructed as means for editors to re-gain control. Through retaining final decision-making power as a human in the loop, editors in turn aim to re-assert their control over the tools.

These two imaginaries are unevenly distributed across academic disciplines: While the idea of a vast and endangered publishing space that can only be governed by employing automatic screening tools is most prevalent in the Life Sciences, the Natural Sciences, and Engineering, the concept of a small and friendly research community that would be threatened by employing tools is most prevalent in the Humanities, with the Social Sciences occupying a position somewhat in the middle. These patterns correspond to a number of well-known disciplinary differences: Firstly, publication cultures vary strongly in between fields, especially between Social Sciences and Humanities, on the one hand, and hard sciences and Life Sciences, on the other hand (e.g. Zitt et al. 2019). Here, the patterns of a high emphasis on journal publications, high citation densities and high speed of publication that can be found in the Life Sciences and many of the hard sciences lend some credence to the image of a vast and ever-growing publishing space. Likewise, in the Social Sciences and Humanities, the focus on book publications and edited volumes, the relatively slow pace of publication, and the high intra-disciplinary diversity (Bayer et al. 2019) also seem to correspond to the image of small, tightly connected communities. Secondly, developments such as exponentially rising rates of retracted journal articles especially in the Life Sciences have drawn considerable attention to the issue of research integrity and misconduct in these fields (Hesselmann et al. 2017). Even though this does not necessarily mean that misconduct is actually more frequent here, it does make sense that it is perceived as more of a problem in these fields. Thirdly, epistemic differences between fields also clearly play a role in the perception of usefulness and appropriateness of screening tools. Here, it can be observed that many tools are indeed explicitly or implicitly designed for research and publishing practices in the Life Sciences, particularly for research that uses standardized reporting formats (e.g. STAR methods table), as well as for research with statistical analyses and hypothesis testing. Tools that explicitly address qualitative or hermeneutic approaches in contrast are quite rare. Thirdly, both imaginaries’ problematizations of research integrity might also be seen as symptomatic of disciplinary cultures and predominant disciplinary discourses. In the Humanities and Social Sciences, promoting research integrity is mostly constructed as a social problem or a problem of collective action that is also believed to be best addressed through social, rather than technological means. For Life Sciences, Natural Sciences, and Engineering, on the other hand, technological solutions clearly take center stage. The more positive outlook of on automatic screening tools also seems to be related to these fields’ emphasis on and trust in technological means for problem-solving. The imaginary mirrors discussions around automation, such as the relationship between automation and increased speed, and the questions about humans in the loop and meaningful human control (e.g. Vagia et al. 2016) that are mostly found in more technologically-oriented disciplines.

A very prominent feature in the imaginaries is thus how constructions of problems and constructions of solutions are tailored to each other: Generally, by emphasizing the vastness of the academic publishing space, research integrity is cast mainly as a problem of data volume, making it an ideal target for an automated technological solution. In tailoring problem and solution to each other, respondents invoke a number of themes and motifs from different larger discourses, such as the speed of automation, but also issues such as the exponential growth of the academic literature and the idea of a “publish or perish” culture. More particularly, the tailoring of problems and solutions is strongly related to the prominence of plagiarism screening (i.e. Similarity Check) as “the” flagship tool of automatic screening in publishing, which goes hand in hand with an emphasis on plagiarism as seemingly the most widespread form of publishing misconduct. Plagiarism screening in particular capitalizes on this idea of the vastness and size of the publishing space (and a vast reference corpus to match with), much more so than other types of tools, which might also be distinguished by their speed (e.g. of solving complex equations or calculating statistical models) but which are not so deeply connected to this notion of a potentially endless universe of potential sources to plagiarize from and to compare to. For the imaginaries, size (of the community/publishing space) is one of the, if not the main, characteristic differentiating the two imaginaries, and it lies at the heart of the problematization of exercising control. This strong predominance of plagiarism screening over other types of screening thus exemplifies the mutual constitution of the construction of a specific problem, the depiction of an imaginary where this problem becomes meaningful, and the technological (and commercial) availability of a tool that targets precisely this problem.

Despite their differences, there is a common thread that runs through the imaginaries of automatic screening tools. While respondents differ in how positive or negative they evaluate this development, they mostly seem to agree that the technological change involving automatic screening is inevitable and will continue in the future. Some respondents welcome this development. For respondents who view this development in a negative light, however, there is a certain sense of fatalism: Even if they may not feel as threatened by rising rates of misconduct, they still portray themselves as quite powerless in the face of technological change. Overall, the comments then convey a somewhat gloomy view of the future of academic publishing, or at least a strong sense of worry; be it that editors are worried about losing control in the face of increasing integrity violations, or about losing control in the face of inevitable technological change. However, this depiction of an inevitable loss of control masks that as with all sociotechnical imaginaries (e.g. Jasanoff and Kim 2009; Mansell 2017), there are multiple layers of performativity involved in the use of automatic screening tools and the construction of the respective imaginaries: Firstly, if screening tools are employed as quick checks to save editors time and to speed up the review process, as proclaimed by many respondents as well as in publishers’ recommendations, they themselves contribute to a further speed-up of publishing and thus a further increase in the exponential growth, and the resulting vastness, of academic publishing. They can thus be seen as exacerbating the problems that they are supposed to solve. Secondly, in these accounts, both the employment as well as the refusal to employ screening tools to detect violations of research integrity are justified by a belief that such violations are quite frequent or quite infrequent, respectively. At the same time, these practices contribute to maintaining the respective views they were built on: Employing screening tools on the one hand inevitably increases the number of (potential) violations that are detected, strengthening the belief that issues such as plagiarism run rampant. Refraining from employing screening tools on the other hand also means that less suspicions are raised, thus seemingly validating the view of violations as infrequent and best detected by peer reviewers or editors themselves. Employing screening tools (or refusing to do so) thus can contribute to sustaining the respective imaginaries as plausible accounts of reality of academic publishing.