1 Introduction

How do we assess the positive and negative impacts of research about- or research that employs artificial intelligence (AI)Footnote 1? This is a pressing question, with ambiguities around the role of researchers, governance bodies, and those who will use or/and be impacted by technologies, across both academic and industry contexts. While significant work has been undertaken to describe ethical challenges of AI, and develop guidance and principles to guide practice, there remains concern regarding the governance of AI research, the gap between principles and practice, and the participation of stakeholders in deciding how AI may be used about, with, for, and on them.

This paper engages with this challenge through the analysis of existing research governance material, by investigating the resources that researchers and research ethics committees (RECs) draw upon in articulating and navigating ethical issues arising out of AI-related research. Resources, such as formalised ethical principles and articulated processes, inscribe knowledge. As such, they act as reflections of knowledge and practices, while also shaping that practice through their conceptual and normative (or regulatory) impact on actors. Beyond this, knowing what resources researchers and RECs actually draw upon can provide important insights into existing knowledge and practices, as well as shape future practice. Such knowledge can also help ascertain whether the hype of AI ethics deserves the attention it’s getting, by illuminating whether—and, if so, how—the numerous published AI Ethics principles are actually used, which in turn will help to appraise the utility of such publications. In doing this we aim to contribute to understanding the ethical issues that AI-related research gives rise to and how learning about these might be (or could be) taking place. Through our socio-material analysis of materials relating to AI from a single institution’s ethics committee process, we address this concern, exploring how these materials provide a lens onto and reflection of the ethical concerns of AI research.

2 Literature review

2.1 Ethical principles and practices

To foster ethical action in the developing areas around use of AI and data, a wide range of guidance and sets of principles have been developed. A recent review identified 84 sets of AI ethics guidelines globally with 11 themes among them [45], while another review of 36 principles documents identified that consensus could be seen across eight common themes [32]. A third review of only research studies regarding ethical principles identified 27 such studies with 22 principles [49]. Finally, a fourth review of public, private, and non-governmental organization (NGO) documents providing AI guidance identified 112 such documents [75]. Significantly, this last review identified significant differences in the focus of documents produced by different stakeholders, and their production, with NGOs and public organisations covering more topics, and being more likely to have engaged participatory approaches in their development. Nevertheless, across these reviews, identified principles overlap significantly with the classic Belmont principles.

Moreover, there have been various calls to move from a focus on developing AI ethics principles, to instantiating them in practice and organisational structures to support practical ethics [50, 60, 72, 89, 91]. These calls emphasise the significance of micro-ethics or ‘ethics-in-action’, and a shift from procedural to situational ethics [33, 38, 43, 51, 64]. This shift, particularly as expressed by [38] reflects both that when we apply procedural ethics we are engaged in practices, and that this process of translation is not mechanical and requires interpretation. This is a concern of recent AI work, reflecting that ethics is fundamentally imbued with action, and ongoing interactions in design processes, in ways hard to capture in procedural ethics. In addition, recent calls have highlighted the importance of analysis of ethical issues of technologies in terms of both immediate or direct impacts (hard impacts), and long-term or indirect impacts (soft impacts), that may affect people’s lives [80].

2.2 The role of research ethics committees

Beyond the myriad of AI ethics principles, the role of governance structures in oversight of novel applications of AI has received attention, beyond the governance pages of companies and universities, in popular media coverage (e.g. [13, 42, 52, 53]. These structures—in the form of Research Ethics Committees (RECs) and Institutional Review Boards (IRBs) — play a crucial role in university research internationally, with mounting pressure to create similar bodies in companies, and a recognition of the challenges such bodies face. In this paper, we will use RECs as a general term that includes medical MRECs, human HRECs, and IRBs, except where explicitly stated otherwise.

RECs are typically comprised of multi-disciplinary research experts, alongside non-research members—in some systems, including lay people—who oversee and review research that involves human participants.Footnote 2 Researchers who wish to undertake such work typically submit an application that explains what the research will involve, and how it will address key ethical principles including the Belmont principles of respect for persons (or autonomy), beneficence, and justice [66]. The role of RECs is to assure these principles are instantiated in research that is approved, and to provide feedback to researchers [44]. RECs do this by assessing materials submitted by researchers. However, due to the formalised process of this work, tensions have emerged regarding the bureaucratisation of research and control by RECs with perceptions that RECs are particularly suited to work in a bio-medical model [5], although some of these concerns may relate to local—changeable—practices rather than underlying theoretical issues [40].

2.2.1 International context of RECs

Significantly, the requirements and remit of RECs vary internationally. While research ethics systems share much common history, a number of publications have investigated similarities and variations across international ethics standards and committees [37, 44, 88] and common emerging themes—including that of data and AI [88].

However, there are also more or less nuanced differences in expression and execution of REC processes internationally. A particularly salient example, given that US experiences are often universalised, is found in the United States IRB guidance, which explicitly directs members as follows:

§ 46.111 Criteria for IRB approval of research: (2) Risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result. In evaluating risks and benefits, the IRB should consider only those risks and benefits that may result from the research (as distinguished from risks and benefits of therapies subjects would receive even if not participating in the research). The IRB should not consider possible long-range effects of applying knowledge gained in the research (e.g. the possible effects of the research on public policy) as among those research risks that fall within the purview of its responsibility. [82].

In considering this quote as exemplifying the direction given in the wider document, and in comparison to international documents, we can note that IRBs receive unclear advice regarding their role in assessing merit and integrity (a key principle in the Australian model [70]), and that they must navigate the directions to (1) balance risks and benefits, with those to (2) “not consider” long-range effects [9]. Moreover, common across RECs is that they provide ‘point-in-time’ governance, but monitor and evaluate largely via periodic self-report of applicants or a complaints-based system for participants [20]. This lack of long-view consideration may create an ethical debt, through which technologies are developed without adequate consideration of long-term impacts [68].

This specific statement has thus been highlighted as a key feature in considering the adequacy of the US IRB model for AI research [25]. Importantly, we should be cautious in universalising models of ethics given cultural and contextual variation in values and practices, and in capacity and procedures for ethics governance such that universal expectations of REC review may exclude researchers from countries where no such review is available [55, 90]. Further, complications stem from disciplinary differences, e.g. [18], and conflicts between REC guidelines and ethical norms of communities with whom research is conducted [21]. Crucially, differences in disciplinary, cultural, and other contextual features must be recognised. In places, this recognition and negotiation is a crucial part of ethical practice, because we should expect values to be contextual. In other contexts, there may be variation presently that rests on a lack of clear standards or articulation of possible norms to which we might work; expressing this is important.

2.2.2 AI and data as challenges to research ethics committees

Even outside the context of AI, concerns have been raised about what we know regarding researchers’ level of understanding of research ethics [10], and correspondingly, regarding expectations around interaction between committee members and disciplinary experts and their respective expertise [26]. However, as Hickey et al., [40] highlight, although there are various criticisms of ethics review boards, disentangling issues of practice in particular institutional committees, from more fundamental concerns with their underpinning principles, is challenging. They suggest that the criticisms of RECs can be addressed through fostering positive learning-oriented ethical review processes, adopting practices of open communication, outreach to research communities, and engagement with disciplinary expertise [40]. These suggestions are echoed by Brown et al.’s [17] specific analysis of education researchers’ views in the UK, finding that although there were concerns regarding understanding of the specific methods and issues in educational research, many respondents had positive interactions with their RECs [17]. Nevertheless, a lingering concern is that RECs act as ‘moral bureaucracies’ via a managerial audit approach to ethics, that is likely to incentivise ‘safe’ practice, and reduce productive rich dialogue regarding ethics, particularly in the context of technology [61].

Indeed, this sentiment is echoed in a discussion of the strengths and limitations of RECs’ specifically focused on AI and data research [29,30,31]. Here, two key concerns regard the nature of research conducted entirely outside university contexts (in which REC systems are mandatory and established) or in collaboration with such industry partners; and where the research involves secondary uses of publicly available datasets [35].Footnote 3 Given these concerns, and particularly the potential for unanticipated and unintended consequences, there have been calls to adjust REC’s understanding of research and data [28]; privacy protection in the context of re-identification (e.g. [15]; and moves in some UK RECs to give greater attention to data and AI [39]. An alternative approach has also been piloted in which researchers wishing to access participating funders’ grants undertake a separate review by an ‘Ethics and Society Review (ESR) board’ [11]. This ESR is constituted of interdisciplinary researchers who review author statements regarding possible impacts and risk mitigation, and provide feedback, with positive initial evaluation of the program. Relatedly, specific guidance has been produced to support industry organisations in establishing ethics bodies [74].

While such concerns have been highlighted across a number of studies and media articles, in a seminal study that involved interviewing REC members, some of the interviewed participants rejected the need for yet further guidelines, instead calling for “implementable procedures to assess big data projects effectively” [31], 136). As the authors highlight, differences in requests for specific guidance may lead to differing outcomes from different committees, with potential for negative impact on “researchers’ trust in the oversight system, data sharing practices, and research collaborations” [31], 138). Crucially, a lack of expertise and experience in assessing big data and AI-related projects was also explicitly recognised by the interviewed REC members [31], findings supported by further European and US research [81, 86].Footnote 4

2.3 Learning for AI research ethics

Despite apparent gaps in REC systems, as the Future of Privacy Forum report ‘Designing an Artificial Intelligence Research Review Committee’ [46] sets out, in developing models to adequately address AI research, we can learn from the significant work undertaken in human and animal research, and biosafety committees. In similar work proposing developments in RECs, a 2021 collaboration between the UK’s Ada Lovelace Institute, University of Exeter, and Alan Turing Institute, investigated ‘Supporting AI research ethics committees: Exploring solutions to the unique ethical risks that are emerging in association with data science and AI research’ [1], and the associated ‘Looking before we leap’ project [84]. Their report [2] highlighted six challenges for research ethics committees:

  1. 1.

    lack of resources and training

  2. 2.

    mismatch in principles designed for researcher-subject relationships applied to researcher-data subject relationships

  3. 3.

    lack of established norms regarding principles for use specifically in AI and data research

  4. 4.

    cross-institutional (and sector) research which can lead to research being assessed by multiple committees

  5. 5.

    challenges of assessing unexpected impacts

  6. 6.

    transparency with respect to corporate research ethics groups or involvement of corporate entities in research activities

These challenges, and the context described in the preceding sections, led these researchers to recommend some key foci for RECs in considering AI research (which we revisit in Conclusions), synthesised to indicate their strong parallels in Table 1.

Table 1 Recommended areas of foci for RECs in considering AI research, synthesised from Hine [41], Ada Lovelace Institute [2], and Ferretti [29]
Full size table

2.3.1 The need for learning in RECs

These sets of concerns and recommendations are intertwined, each contributing- and being contributed to by others in the list. A lynchpin of these recommendations is learning. This focus involves understanding how RECs, researchers, and stakeholders learn about AI, its impacts, and the systems into which it is deployed and the ethical concerns of those systems. Research is required to understand the processes of this learning, how learning about AI, ethical development and thinking, and systems thinking come together. RECs should have continuous training for staff regarding ethical review processes and their importance, with ongoing development. This applies in university contexts, but notably: “Many corporate RECs we spoke with also place an emphasis on continued skills and training, including providing basic ‘ethical training’ for staff of all levels.” ([2], 37).

In a powerful move highlighting the significance of learning through editorial policy, the Journal of Empirical Research on Human Ethics includes in its manuscript template an Educational Implications section, intended to discuss the ‘key concepts’ from the article to support teaching to different stakeholders, including research and REC communities, as well as students, and external stakeholders such as participants and the general public [47]. In Ferretti et al.,’s analysis—published in that journal—this ‘Educational Implications’ section notes the significance of: “knowledge exchange and a more productive engagement among the various factors involved in big data research. These include and are not limited to RECs, researchers, research institutions, private enterprises, citizen science groups, and the public” ([31], 139), highlighting that this responsibility involves developing skills around both the technology (AI), and ethical processes and values. As they also highlight, the range of actors for whom there are implications for learning extends to “informing society about issues related to big data and the use of AI in research. Starting with this democratic engagement, the general public can clarify their expectations regarding research with big data and thus inform the decisions of other actors involved.” ([31], 140).

2.3.2 Beyond principles

The importance of learning regarding the application of ethical concepts to AI research has been highlighted. However, as noted in the introduction to this article, while a significant body of work has engaged in developing guidelines and principles for ethical AI with an aim to disseminate and educate a variety of audiences, the operationalisation of these principles into organisational structures, practices, and professional reflection, has received less attention [50, 60, 72, 89, 91]. As Resseguier et al., put it: “this identified gap in AI ethics finds its root in the very nature of the currently dominant approach to AI ethics, i.e. a view on ethics that considers it as a softer version of the law. [They] point to the need to complement this approach […and…] call for a shift of attention in AI ethics: away from high-level abstract principles to concrete practice, context and social, political, environmental materialities.” ([72], 3).

Awareness of these principles and guidelines is important, and has positive impact on intention to consider ethical issues, by providing orienting devices for stakeholders to think with. Specifically, [22] surveyed > 1,000 managers in the US, randomising presentation of different four groupings of AI regulations and asking about ethics in AI and their intent to adopt AI. They found “that information about AI regulation increases manager perception of the importance of safety, privacy, bias/discrimination, and transparency issues related to AI. However, there is a tradeoff; regulation information reduces manager intent to adopt AI technologies.” ([22], 1). Similar reflections were provided by Miller and Coldicutt [59] who polled UK ‘tech workers’ (n = 1010) finding that 81% of those who worked on AI tools (n = 155/192), “would like more opportunities to assess the potential impacts” (p.10). Thus, principles can be useful tools insofar as they offer orienting devices to think with. However, learning to engage in ethical practice goes beyond principles in addressing at least four key concerns:

  1. 1.

    How do we learn to operationalise principles in context: Principles provide useful anchors, but we must learn how to work with them with particular contexts and people, noting that ethical boundaries may change over time and location. As Resseguier et al. put it: “ethics must entail a sharp attention to specific situations and relations, accounting for the different levels of the personal, the interpersonal, the organisational, up to broader social, political, and environmental configurations” ([72], 10).

  2. 2.

    How do we learn to navigate tensions between principles: Classic dilemmas include freedom vs equality, or free speech vs privacy, and there is significant literature on this topic.

  3. 3.

    How do we learn for a substantive ongoing ethics over procedural ethics: There are questions around (1) how we probe why a tool is being implemented, and (2) whether reinforcement of existing systems closes off opportunity for work that develops futures worth wanting? Focussing on ‘doing things ethically’ can lead to abstracted models of action that fail to interrogate underlying aims in, for example, developing particular tools, including how they intersect with existing power relations [6, 72]. As work on Data Feminism highlights [24], current approaches to AI ethics are inadequate to addressing structural entrenched inequality and the material reality of AI development. As [48] note, a focus on ethics in the technical design of systems misses significant concerns (including in their proposal for an ethical ‘Algorithmic System for Turning the Elderly into High-Nutrient Slurry’).

  4. 4.

    How do we assess the indirect, long-range, or soft, impacts of our work: Principles used in research ethics have typically focused on risks to participants, and relatively direct and immediate impacts more broadly (sometimes excluding risks, focussing only on possible benefits). These direct impacts may be relatively predictable, perhaps through a hypothesised pattern of causation and modelling of their likelihood of occurrence. However, many technology systems have broader and long-term impacts. These occur in the ways they may reorganise social relations, and re-shape normative assumptions regarding human value and values.

2.3.3 Resources for learning AI ethics for research

Where, then, should we look for resources to support this learning? In their work surveying 54 and interviewing six AI practitioners, Morley et al., highlight that “the AI ethics community is not yet meeting the needs of AI practitioners” ([62], 6), with more practitioners saying further resources would be useful than those who say that what exists is already adequate, across a range of types (from principles, to design guidelines, and ‘best practice examples’) [62]. Where lessons are being drawn from other parts of the community, the historical parallels—for example, the sharing of security flaws in software as a defensive practice—may not carry over into AI [76]. Various resources exist, including worked cases [3, 67], and one helpful example of how one might complete a REC form [23], and emerging reviews of materials such as those in the ‘Responsible AI Pattern Catalogue’ [54]. Understanding how these resources are being, or could be, mobilised including via the crucial role of RECs is thus important.

Indeed, this is a challenge across fields: understanding how researchers develop and express research ethics, [10]. Based on their review of papers discussing ethical issues in research, Beauchemin et al., [10] highlight a dominance of descriptive ethics, with relatively little use of established definitions or reflection, leading them to call for a greater focus in research outputs on articulating the ethical concepts used [10]. This is particularly salient given RECs may draw directly on literature (or expect applicants to include relevant disciplinary literature) regarding ethical issues. However, if discussion of ethical issues is unusual in academia’s primary mode of communications—research outputs—where should researchers and RECs look in seeking to increase “sensitivity to ethical issues [and consider] how empirical data may be relevant to various ethical principles and problems.” ([27], 16).

In their specific analysis of AI in mental health initiatives, Gooding and Kariotis highlight that ethical and legal issues tend not to appear in the peer-reviewed literature, even if they may have been considered in the REC process [36]. Perhaps as importantly, they also flag that most publications in the space report on pilot work, thus obscuring the potential long-range impacts of the research [36]. Even more concerningly, one analysis of over 227 publications on health technology (from an initial pool of over 14,000 returned) indicated that approximately half made no reference to ethical principles at all [79]. In a review of software engineering journals, Badampudi et al., [8] report that roughly half discuss one of consent, confidentiality, or anonymity, but only 6 of 95 reviewed discuss all three [8].

Here we see how the roles of advisory bodies, formal RECs, publication processes, and the guidelines and principles come together in an ethics ecosystem [73]. In this ecosystem “individuals (researchers), organisations (research institutions and the various committees within) and external bodies (publishing houses, funding bodies, professional associations and the governance policies they produce)” ([73], 317) participate in developing understanding and evaluating of ethical behaviour, through their roles in the research process.Footnote 5 Moreover, we see how different components of a system come together to act on ethical thinking, and provide resources for that thinking. Adopting this view, Chi et al., [19], analysed AI ethics documentation regarding diversity and inclusion, within three large AI infrastructure companies. This expansion of “the range of documents past high-level corporate principles sheds light on how firms translate principles into action and provides greater clarity about the problems and solutions they hope to address through AI ethics work.” [19]. Through this analysis, they highlight that diversity and inclusion initiatives within these companies is configured to an “engineering logic”, thus while they claim AI ethics expertise, they act as “ethics allocators” pushing decisions regarding impacts of tools downstream to customers [19].

Importantly for this paper, they highlight a key claim: That on the one hand, the various sorts of documents or material resources organisations produce and draw on are reflections of (or, ‘containers for’) value statements, while (on the other hand) they also shaping this discourse (reflection on) through the resources they provide and the particular kinds of narrative they encourage and recognize as genuinely ethical. In this way “they are a kind of agent, educating clients, the public, and the broader field, articulating and defending values, developing scripts for ethical action that allocate work and responsibility to internal and external actors, and constructing the knowledge and expertise AI ethics work requires.” ([19], 2).

3 Method

3.1 The materiality of research ethics

Despite the significant body of work drawing attention to the ethical impacts of AI, alongside corresponding guidelines and principles, relatively little is known regarding the resources drawn on and produced through the workings of actors within the research ethics ecosystem (including researchers, those impacted by the research including participants, and ethics committee members and secretariat).

These diverse components of ethics ecosystems, including the ethics process itself, are forms of knowledge which, as Freeman and Sturdy [34],Footnote 6Footnote 7 put it, are inscribed, embodied, and enacted. Inscribed in different kinds of artefacts that encode knowledge, including ethical principles and templates, that are made available for use across contexts. Embodied within individuals who bring this knowledge to bear in their actions, often in implicit ways. And enacted, in the sense that new knowledge emerges from interactions, and is available for use in, particular contexts. An example offered by Freeman and Sturdy is helpful: “When a committee convenes, embodied and inscribed knowledge is brought into the room in the form of what each of its individual members knows [embodied], whether through education or experience, and in what has been recorded in the minutes of previous meetings and in the documents prescribing the committee’s remit and procedural rules. But the committee’s knowledge is not limited to what is brought into the meeting. In the course of discussion, the committee may generate new knowledge: new ideas and insights, new aims, and new rules for how to fulfil them [enacted]” ([34], 12). How the learning transfers beyond the members of the committee is an important question to ask.

Because many resources that inscribe knowledge in an ethics process are used explicitly as objects by multiple agents—researchers, REC members, stakeholders, compliance organisations, to name a few—these resources act as boundary objects [4, 16, 78]. In this way, resources such as formal policies, standardised forms, principles, and learning resources, are materials that inscribe knowledge in order to span across context and actors. Simultaneously, they are interpreted and reinterpreted in context, thus their meaning is not only held in the resource itself, but in the way its knowledge is mobilised and negotiated (or, enacted). Resources such as REC application forms and the materials to which they refer thus provide textual lenses reflecting both stances in their own right, and instruments that shape discourse [63].

3.2 Mapping research ethics

In Australia, the primary research ethics document—with legal standing in national research governance systems—is the National Statement on Ethical Conduct in Human Research [65] (henceforth “National Statement” or just “Statement”). While research ethics naturally extends beyond this document, and the document is grounded in histories of practice, culture, and artefacts, here we will treat it as the first order ethical document. From this document, we can see second order materials arising through the ethics process, at varying steps removed from the National Statement:

  1. 1.

    Institutions develop ethics forms, intended to support researchers in instantiating responses to the Statement’s key principles.

  2. 2.

    Researchers then complete these forms, with reference to the Statement, and other soft (e.g. disciplinary guidance documents) and hard (e.g. privacy legislation) policy.

  3. 3.

    Completed forms are evaluated by RECs, and their evaluations are articulated (using National Statement concepts) with the intent that researchers will respond to them.

  4. 4.

    Researchers then conduct research, within the scope of their REC approval, and ongoing commitment to ethical practice (which may be further informed by disciplinary and cultural norms).

  5. 5.

    Journal editors and reviewers, funders, and RECs, will then review submissions at different levels of granularity regarding the completed work, for internal or external reporting/dissemination. These should include reference to the REC process (at minimum reporting that one was completed), and any issues arising.

Therefore, to map the material ethics ecosystem we conducted a review of:

  1. 1.

    Resources available institutionally, to be drawn on by the REC process.

  2. 2.

    Where these resources are taken up in research practice, through a systematic search of our internal REC application database, and external publication databases.

  3. 3.

    And within these materials, an analysis of the REC application detail, supplemented by semi-structured interviews of the respective researchers, and further review of published outputs.

While previous research [19] has mapped documents from multiple organisations to analyse expressions of values and ethics, we instead focus on a single organisation. In that prior work documents were coded as representing different functions regarding ethics: (1) pedagogic tools; (2) product documentation; (3) legal/policy; (4) general communications. As our focus is on understanding networks of resources linked to specific projects internally, and how this analysis can help us understand the socio-material context of the work reflected, we develop an approach informed by Chi et al. [19]. Specifically, our analysis maps out the following materials:

  1. 1.

    Pedagogical tools—specifically guidelines, courses, and scholarly outputs such as reviews of ethics strategies, of participant preferences, etc.

  2. 2.

    Process resources—these include materials such as ethics proformas in document or web-form format, for example focussing on data protection, and REC application details

  3. 3.

    Legal and policy instruments—these include statements of principles, the Australian National Statement on Research Ethics [65], and legal instruments such as relevant privacy legislation

  4. 4.

    General communications—these include any available communications from the institution referring to relevant issues, made available through general (rather than ethics targeting) channels

  5. 5.

    Discursive resources including REC consultation, and stakeholder consultation (or other forms of input, such as codesign)

  6. 6.

    Reflection on practice, including any expression regarding previous experience (e.g. provided in REC applications), or experiences of relevance within the project (e.g. in publications, or public reflections).

As described above, analysis of these material resources frames these resources as providing an expression of, or lens onto, the conceptual space that shapes and is shaped by ethical discourse.

3.3 Interviews

Interviews were conducted based on invitations to researchers from a higher educational institution who had been identified as submitting relevant applications in our search process (described below).

A semi-structured interview schedule was developed, to understand perspectives of the researcher stakeholders regarding their use (or otherwise) of ethical frameworks in their research on data and Artificial Intelligence (AI). Participants were invited to speak about their organisational contexts—which, for some, crossed university and industry settings—and any practical challenges in use of AI and advanced technologies and approaches to address these ethically. Interview questions (Table 2) were developed to probe the dimensions described above, regarding:

  1. 1.

    Developing approaches to ethical concepts and principles

  2. 2.

    Learning to navigate tensions and challenges

  3. 3.

    Procedural and substantive ethics: Process (and adequacy) of REC in mediation

  4. 4.

    Challenges in AI research and soft impacts

Table 2 Semi-structured interview protocol setting out thematic area of probe, specific questions, and the issues we anticipated eliciting
Full size table

The questions were designed to leave open the discussion of principles used and any ways these were identified and navigated by participants, and to allow for discussion of the range of pedagogic, discursive, reflective, legal, and other resources used alongside the formal REC process and any others followed.

Interviews were scheduled for 30–60 min duration, conducted via online video conferencing. They were conducted following a consent process in which we requested access to key REC materials related to projects of relevance (described below), these thus act as an anchor for the interviews, acting as a preliminary stimulus and material artefact. We also provided reference to other principles in advance via the introduction to the interview, including the National Statement, Australia’s Artificial Intelligence Ethics Framework [7], and the Human Rights and Technology Issues Paper: UTS Submission [85]. As a semi-structured protocol, not all questions were asked of all participants, although all themes were introduced in all interviews. The initial questions often naturally led to further discussion of ethical issues and the role of the REC, and the interview protocol served as a guide to steer these conversations (even where the questions were not explicitly used).

The interviews were conducted by a single researcher, who also implemented the first analysis of the interview and REC material data. The interviews were professionally transcribed, and these transcriptions were selectively coded alongside other research texts (submitted REC applications and files) drawing on approaches to discourse and document analysis [14, 71].

In reporting, the transcription convention used is that […] indicates words were removed (where these are not relevant to the key issue), and [unclear] indicates words that were inaudible or unclear. A non-verbatim transcription is used, with non-linguistic features (gesture, and fillers such as um, er, etc.) not transcribed.

3.4 Ethics

The work was internally funded through a faculty seed grant. REC material may fall under the intellectual property of the institution or be considered internal material for the purposes of evaluation and quality improvement. However, because of the research intent of the work, and the inclusion of semi-structured interviews, a REC application was submitted (ETH216658) and data sharing agreement put in place, building on an earlier application (ETH205567) regarding use of ethical frameworks based on responses to a public consultation on AI ethics. This updated application provided approval for:

  1. 1.

    a search to be conducted on the REC database for keywords across titles and summaries, with results provided to the authors;

  2. 2.

    the authors screening these as described above;

  3. 3.

    the authors contacting lead researchers on relevant projects, to seek their consent to access their full REC materials, and invite them to interview (these were treated as separate consents);

  4. 4.

    the authors liaising with the REC secretariat to provide consents (where given) for sharing the REC material for the stated purposes; and

  5. 5.

    using the REC materials to inform the interview discussions, where those occurred.

Separately, we also sought references to REC approval in published works (as above). The reporting here is not intended to identify specific authors or their work, and we have sought in our aggregation and excerpts of interviews and other material to maintain confidence and reduce risk of re-identification.

The reporting here is also not an evaluative reflection of any work noted, at an institutional or individual level. Our analysis is limited to the data available to us, selected through a particular search strategy at a single institution. We have no reason to suppose that this data is particularly unusual, but nor do we make claims about generalisability either at our institution or more broadly. Rather, our interest is in how the process of conducting such analysis may inform understanding of ethics processes, and how our specific study may provide broader insights.

4 Data—instantiations of ai ethics resources in use

4.1 Mapping ethics resources

In our first step, we sought to map the institutional ethics ecosystem, using the model described above, and drawing on the visual representation in Chi et al., (see, [19], 4). That foundational work analysed multiple technology companies and their expression of ethics and values with respect to diversity and inclusion. A helpful step in their representation was to (1) colour code documents according to department or product space within the organisations, and (2) draw connections to explicitly highlight how documents referred to each other. Neither is appropriate in our case. That is because (1) the documents we are drawing on are all within the research governance space, with the exception of “general communications”, and (2) the documents are highly interconnected in their present form (again, with the exception of general communications). Figure 1 below indicates the set of resources returned through searches of both internal and external sites. In addition to these resources many other materials may be drawn on by individuals, groups to which we do not have access, and from external sources. Our intent here is not to suggest this resource set is exhaustive either of the set of resources within the institutional ecosystem, or—clearly—of the set of relevant resources in the wider ethics ecosystem.

Fig. 1
figure 1

Mapping the institutional ethics ecosystem

Full size image

4.2 Search strategy and output

A term-based search was conducted on all REC applications, using the centralised system through which all such applications are submitted. This system allows for searching over the text-field submissions, which comprise most of the application, barring attachments which typically consist of items such as: consent forms; participant information sheets; budgets; organisational approval letters; data collection materials of various sorts, such as survey instruments and interview protocols; elaborated answers to text fields, such as rationales for particular approaches, study design diagrams, etc. At the point of the initial search, the RECs received 6–700 applications per year across the full REC panels and faculty level delegation.

The initial search was conducted in October 2021, for applications dating from 2015 (when the system was launched). A follow-up search was conducted in September 2022. Applications on which any of the co-authors were an investigator, or involved in the research in a non-investigator role (e.g. advisor, participant, student-of), were excluded. In some cases, no researcher was still at the institution, and these applications were excluded. Some researchers had multiple studies identified, in one case two submissions were discussed in interview; in others, the researchers either declined or did not reply to an earlier invitation, and thus any later applications were also excluded. Results of this search and screening are summarised in Fig. 2.

Fig. 2
figure 2

Search strategy for AI Studies via REC

Full size image

4.3 Research outputs

To complement our search of REC applications, we conducted a bibliographic search of the Web of Science (WoS) core collection (Fig. 3), which provides comparable coverage to Scopus as an indexed article collection [12, 77]. This approach was intended to (1) act as a check on further applications that may have been missed in the internal-system search; and (2) provide further insight regarding the expression of ethics by researchers, through analysis of reflections of ethics in their published works.Footnote 8

Fig. 3
figure 3

Search strategy for AI Studies via publications

Full size image

We also conducted a search for obtained REC numbers (e.g. searching for “REC-15000”) in Google Scholar, to supplement the materials in the REC process, though this did not identify any further material.

Finally, we also conducted preliminary searches of the institutional repositories (an Open Access self-hosted repository, and via the Dimensions database, with which we have an institutional arrangement), using a full-text search for the same terms. These searches were not systematically reviewed due to significant overlap with the WoS search which yielded data saturation.

5 Results and analysis of ethics materials and interviews

The materials retrieved were analysed and drawn upon to identify and invite interviewees. From the n = 11 applications shared, the set of resources drawn on explicitly within the application, or via the interview data, were mapped using the framework in 1. As indicated in Fig. 4 there is significant overlap between the resources available in the ethics ecosystem (Fig. 1), and those drawn on (Fig. 4) in practice, notably:

  1. 1.

    The National Statement featured as a central principles document

  2. 2.

    The REC process itself was explicitly noted as drawn on in ethical consideration

  3. 3.

    The Australian Privacy Principles (APPs) and generic ‘university policy’ provided some policy context

  4. 4.

    Discursive resource via colleagues (peers, supervisors or other senior colleagues), and other stakeholders were mentioned as a key resource

Fig. 4
figure 4

Mapping Aspects of the institutional ethics ecosystem drawn on in practice. *The interviews of course provide a clear indication of reflection on practice. Here we are specifically interested in examples of resources that are designed to promote reflection, or/and instances where materials (including the interview data) refer explicitly to a prior occurrence of reflection, such as learning from previous experience on a similar project

Full size image

However, as our interview data indicates, the depth of use of these resources is unclear in places. The pedagogical tools and general communications referred to were targeted at the specifics of the projects, and thus differed significantly from those available via the institutional ecosystem. However, although some resources in this internal ecosystem were relevant to AI, with the AI ethics principles being clearly highly relevant, the former were not drawn upon at all, and the latter were only mentioned once.

Table 3 sets out key responses from the pool of interviewees, mapped against the four key concerns in learning to engage in ethical practice (see p.8). The five researchers interviewed are identified (R1-5), and the topics of their research projects were:

  1. 1.

    Transcript 1: Understanding (through self-report methods) organisational practices for data projects, using both qualitative and quantitative methodologies (such as path analysis).

  2. 2.

    Transcript 2: Interviewing developers of an AI system to understand how their design practice avoided bias.

  3. 3.

    Transcript 3: Developing and deploying a system at a field site, including secondary analysis of data captured on site (with removal of any data that could de-identify people on site, prior to receipt by the researchers).

  4. 4.

    Transcript 4: Effective delivery of data science initiatives in a specific sector.

  5. 5.

    Transcript 5: How organisations manage and use their AI technologies.

Table 3 Summary of Key issues and resources identified in interviews
Full size table

Interviewees 3 and 4 were building AI tools via their research (others may have been in other capacities). This may suggest that in the process of REC submission, while information regarding methods is elicited, this elicitation does not capture the range of relevant approaches adopted.

Developing approaches to ethical concepts and principles: The participants were invited to consider nationally relevant ethics principles, alongside which they noted national privacy principles, and the European General Data Protection Regulation (GDPR) in passing. Participants referred to self-reflection in contrasting ways, with regard to a trigger for seeking out an ethics framework (R3), and the idea that “My ethical framework is myself, and that’s good enough, I think.” (R1).

Learning to navigate tensions and challenges: Participants referred to challenges in operationalisation of principles not only into practice but into other material forms, for example saying “so this is what the documents say, and how are we going to transfer it into our ethics applications” (R3). This went alongside a sense that outside of the university context, ethics is not a consideration in research and development, with one researcher (R3) who contacted external researchers reporting that “they don’t have ethics or they don’t really care about the ethics around this.” (R3).

In discussing the published outputs of their research, R3 notes that a core concern of their methodology was to ensure that the site of their research could not be re-identified from images contained in these outputs. For instance, R3 described their response to a person who, at a conference presentation, asked about their collection process and ethics: “I told them, if you want more details, this is the ethics number. […] So, you can contact us. Because [they were] very interested. [They] wanted to do something similar. And [they were] interested about the ethics and the data collection process around it.” (R3). Another (R2) notes their research was informed by a well-known case of “AI failure”, with part of the work investigating how designers seek to avoid these kinds of biases: “my idea was, are there processes that we can put in place to prevent that from happening” (2–1). In both cases, we see how knowledge is inscribed in resources made externally available for shared learning.

Tensions between data quality and ethical considerations were a recurring theme, as were reflections on whether standardised processes could help navigate such tensions. For instance, R3, who required images of a physical space, but not the people in it, noted that “one of the main questions that was raised was do we have the consent of those [people] to be appearing in the video. […] if I had set up the cameras by myself, then I would have [inadvertently] captured the [people]”. To navigate this challenge, secondary data (alternative images) were provided and filtered to ensure people were not visible, but this meant that the images obtained were not captured from a position the researcher would have chosen. Two researchers (R3 and R4) noted the challenge that high-quality imaging increasingly makes it harder to de-identify subjects by filming from a distance, because by-standers may still be recognisable even if they were not the intended target of analysis. R4 commented: “Then you say, hang on, I got 25 projects trying to do the same. What can we standardise? What’s the guiding principles? What are the governance frameworks?”. However, as R1 observed, a challenge for such standardisation, and broader concerns about consent, is that particular research methods used may require bespoke (i.e. not standardised) approaches: “An interesting experience I made recently is that people don’t understand my analysis, not even academics, and that might make it a bit complicated in terms of, maybe, ethics as well.” (R1).

Procedural and substantive ethics: Process (and adequacy) of REC in mediation: Participants reflected on tensions between procedural and wider ethical considerations, including features of the REC process and requirements around such things as data privacy. R5 observed: “it’s stipulated by the university what you need to do and how you need to keep your data […] So, it’s not really an ethical decision. It’s more like there’s rules to follow. So, I don’t need to make any ethical decision.” Later also noting that “there’s a difference between following the law and an ethical decision” (R5).

Nevertheless, participants recognised that established REC processes supported ethical reflection by encouraging them to “think about things a bit differently” (R1) and “stop and think a moment” (R2), and even suggested: “I think it would be wise if more organisations would have an AI ethics committee to stop and think before they build the AI because there are so many problems around this area. And many organisations don’t stop and think. They just do, and as a result we have a lot of biased AI and a lot of problems. So, I think the concept of having an AI ethics committee can be very, very valuable and we should actually move that from the university to the more corporate world as well.” (R2).

The importance of fine-grained contextual factors, and not merely relying on generic procedural ethics, was also noted. For example, that consent practices must be adapted to specific contexts, moving beyond basic procedural requirements. R3 notes: “We have to establish that dialogue with them. They’re not into reading consent forms, user agreements. So, we have to do the face to face dialogue and to explain to them. And some of them didn’t even realise what machine learning, AI, deep learning means. For them, it’s like they think we’re doing something robotics when we talk about AI. So, it’s very understanding. Different people have different perceptions. […] So, it goes beyond documented consent forms and user agreements. You have to have these dialogues, verbal communication. I think that’s very, very important in AI research.”.

Perhaps unsurprisingly, a technical framing of ethical concerns was another theme among our participants’ responses—for instance, seeking to employ technical approaches to explainability or bias to proceduralise ethics—and using terms like “explainable”, “ethical”, and “responsible” interchangeably.

Challenges in AI research and soft impacts: Although the REC ethics process was generally seen as rigorous, and our participants viewed their own research as posing relatively low risks, concurrently they also observed that AI more generally might raise more- and different kinds of issues that the REC process is “not reflective of, how should I say, the ethical implications for artificial intelligence for the whole of society.” (R1).

Two examples of the relatively-uncontroversial nature of many current uses of AI were that human-in-the-loop systems are often used to mediate AI’s decisions, and that the purpose for which AI is used is often relatively tame. As R5 put it: “not to be dramatic or controversial. So, I guess that’s an ethics thing that they are tapering their AI. They’re not making the extreme” (R5). At the same time, though, they also recognised broader ethical concerns regarding responsibility to the conduct of science and the public’s trust in science: “I guess that people feel aggrieved or unfairly dealt with. So, I guess if that feeling swelled, there would be less people who’d want to take part in my research if there was that feeling that it was unsafe, unsecure. And then I wouldn’t be able to conduct my research. Or if it grew wider, then no one would conduct any research sort of thing if there’s such mistrust there.'” (R5).

Participants also commented on issues in AI research around soft impacts and commercialisation, including that the ethical use of data and ethical use of AI raise different issues (R4), and their sense that there is a gap in ethical research and development outside of universities: “outside, people are doing whatever they want” (R1), and “AI is not necessarily localised and AI is borderless, and organisations would need to apply to all these different regulations when building the AI, which doesn’t really help in the process.” (R3).

6 Discussion

Demands are emerging to put into place governance structures for AI research across sectors, inspired by existing research ethics governance models. In light of the findings of this research, we point to key issues and reflections in Table 4. As the table indicates, findings are largely consistent with prior work. The researchers were generally positive about the REC process as a means to support their reflection and provide oversight, however noting concerns regarding oversight of cross-sector work and long-range impacts. The implication, then, is that in considering the ethics ecosystem (Fig. 5), and how it draws on resources (Fig. 4), attention should be paid to how ethics governance and reflection can be inscribed so as to cross-institutional and temporal boundaries, in order to foster ethical reflection and action across all research (and in this context, all research involving AI) (Table 4).

Fig. 5
figure 5

Elaborated research ethics ecosystem

Full size image
Table 4 Reflections on recommended areas of foci for RECs in considering AI research
Full size table

Grounded in the findings reflected in Table 4, we propose a broader updated ethics ecosystem (Fig. 5) that builds on the governance recommendations reviewed (Table 1), Samuel et al.’s [73] ethics ecosystem model, and the features of it described in Sect. 3.2, highlighting the kinds of resources, and their role in learning, borne out in this research.

7 Conclusion

Rising awareness of AI has prompted increasing demands for its ethical governance and a plethora of ethical AI guidelines. RECs have a well-established history in ethics governance, but there have been concerns about their capacity to adequately govern AI research. However, no study to date has examined the ways that AI-related projects engage with the ethics ecosystem, or its adequacy for this context. This project is based on a single institution, of projects identified via the particular search strategy, and notably only of those that undertook a REC application. These contingencies present limitations, although we have no particular reason to believe that the results are particular to our institution (where AI is a strategic focus). Moreover, the model developed for analysing these applications presents a novel approach to understanding and assessing an ethics ecosystem, a contribution with broad application across both university and industry RECs.

Our results suggest that, despite calls for new structures, existing REC models can effectively support consideration of ethical issues in AI research. REC principles and processes were drawn on and referred to by our participants, and — in the Australian REC context at least—are embedded in a lineage of work on research ethics that is continuing to develop. Thus, where new materials are required, we propose that they should be embedded in this existing well-established ecosystem, rather than creating novel governance mechanisms tailored specifically to AI.

Gaps were identified in the resources drawn on, and by participants in the interviews. Participants expressed uncertainty about some practices, and noted that long-range impacts and issues such as secondary use of data may not be effectively addressed in existing guidance. However, it is not clear these issues are addressed in AI ethics guidelines, and indeed only one participant referred to use of AI ethics principles specifically, with multiple participants raising concerns that outside the research ethics context —a context from which these new guidelines have largely emerged — practices were more varied, and less rigorous.

One upshot of our study’s findings is that the development of new AI principles may not be an optimal strategy for addressing ethical issues related to AI. Indeed, it is far from clear that the proliferation of AI-targeted principles has helped in practice. The results indicate that shared artefacts of practice, such as ethics applications and published articles referencing ethics, provide one lens (socio-material) into the practical usage of principles in context. These resources may be used to support learning by individual and organisational stakeholders. In tandem, organisations seeking to engage with ethics and AI should look to the well-established structures of RECs to build on this lineage. RECs themselves may develop further and support uptake in new contexts by evaluating how their communities—REC members, researchers, the public, etc.—learn regarding ethical issues, and where within institutional governance structures the kinds of issues specific to emerging technologies are addressed, and updated in an ongoing way.