1 Introduction

There is growing recognition of the need to embed consideration of the social and ethical implications of emerging applications of AI in their design and deployment. There is a heightened awareness of the risks of AI, including concerns about control and management of AI systems, fairness, bias and transparency and increasingly, questions of sustainability, environmental health, and the longer-term impacts of AI [1,2,3].

The development and deployment of AI has rapidly outpaced regulation and legislation. Thus, the onus has fallen on those developing and deploying AI to ensure that it is done in alignment with principles of responsible innovation, and with sufficient consideration of the ethical, legal, and social issues (ELSI) which emerging technologies can entail. Though a broad range of ethical principles for AI have been proposed, formulated, and synthesized within the public and private sectors (e.g., [4,5,6]), many organizations opt to develop their own principles, to signal the values which are of importance to them with the aim to build trust in the systems they deploy [7, 8]. However, there has been relatively little research to date on the processes through which these principles are developed, particularly from diverse settings.

In this paper, we report on a novel, collaborative initiative in Japan between researchers in the humanities and social sciences, and industry actors to develop AI ethics principles and an accompanying checklist. This collaboration is being undertaken between the Osaka University Research Center on Ethical, Legal, and Social Issues and Mercari R4D, the research and development arm of Japan’s top consumer-to-consumer retail platform.

In 2017, Barocas and boyd observed that, in considerations of the ethics of their technologies, data scientists often ‘speak of trade-offs,’ ‘draw[ing] on a wide range of values to work through difficult tensions.’ Through a qualitative analysis of minutes collated from meetings between researchers from the ELSI Center and relevant members of Mercari Group teams, we find evidence of consideration of multiple trade-offs in this Japanese context. In this regard, we find that, as raised by Barocas and boyd, researchers involved in the development of organizational AI principles at this R&D organization were indeed keenly aware of the inherent trade-offs–and at times, contradictions–which result through the development and application of AI ethics principles. Moreover, the findings of this study suggest that the researchers involved at times saw these trade-offs as not only between ethical concerns and organizational benefits or technological development–as is often presented in the literature [9]—but also between competing, ethically-oriented positive outcomes.

Thus, three main contributions are made through this paper. First, the identification of these trade-offs offers evidence for Barocas and boyd’s [10] observations, and shows that, contrary to some claims made in the literature (e.g. [11],), there is indeed evidence of a desire to do good for society. Considering the way in which ethics frameworks are designed and implemented, Saetra [12] has argued that ‘ethicists end up enabling tech companies to cherry-pick some framework that is easily implemented without the need for cumbersome concessions and changes.’ Yet, by focusing on the discourse which occurred in the discussions about AI ethics and at the sites at which these ‘paper tigers’ [7] are developed, the findings of this study suggest a more nuanced picture, highlighting a willingness to engage with complex ethical issues, and the resulting tensions experienced.

Second, this evidence is particularly significant as it emerges from an under-studied Japanese context. Ethics is a ‘fundamentally sociocultural concept’ [6], and there is a need for the consideration of culturally-grounded values in AI ethics. There has been acknowledgment of the need for contributions to the discourse on AI ethics from Asia broadly [13], and from Japan in particular [14]. Despite this, there is a paucity of work considering cases from Japan–a gap we contribute to filling.

And third, in the absence of sufficient social scientific work on AI [15] we showcase a novel initiative based on the ‘constructive collaboration’ called for by Zook et al. [16] and Barocas and boyd [7, 10] as we report findings from interdisciplinary collaborative research between experts in the humanities, social sciences and data scientists. We show that such collaborations work effectively in Japan and are meaningful as they ensure that considerations of the ethics of AI are ‘grounded in practitioners’ needs’ [6] and in the practical use of the technologies [7, 10].

Below, we briefly review recent literature on the role of AI ethics principles in organizations. We then introduce the methods employed in this study and the resulting trade-offs identified, as well as the ways in which participants sought to resolve these trade-offs. We follow this with reflection on the implications of these trade-offs, situating them in recent literature. Finally, we consider the limitations of this work and the need for further research in this field. Throughout, we argue for a more nuanced perspective on the development of organizational ethics principles that highlights awareness of the trade-offs inherent in the development of ethical AI systems, and a desire to reconcile these trade-offs for social good.

1.1 Considering the role of AI ethics principles

There has been a shift towards a focus on ‘Ethical’ or ‘Responsible’ AI, defined as ‘the practice of using AI with good intention to empower employees and businesses, and fairly impact customers and society’ [17]. This wave has arisen in recognition of the harm that can occur from the implementation of AI (e.g. [2, 18,19,20],). As described above, there has thus been an imperative for those developing AI to show that they are pre-emptively considering the potential ethical issues which may result from the implementation of their systems. Particularly, in the absence of definitive ‘hard ethics’ or regulation to set appropriate bounds and standards, this ‘soft ethics’ [7] approach is relied on as a way to attempt to minimize the harms of AI systems.

This emphasis on the ethics of AI—still relatively recent in history— draws on longer-standing tendencies in the consideration of emergent technologies [21]. As documented by Casiraghi [21,22,23], the field of ‘traditional’ bioethics, which began in the 1960s, and emerged distinctly in the United States and Europe, has had a particular influence on AI ethics today. This influence was in part through bioethics’ focus on principlism, which was valued for its ‘appealing basic structure ensuring simplification for decision making as opposed to disorderly or somehow arbitrary systems used before’ [21]. Principlism in this sense refers to a principle-based approach, and represents a ‘middle ground between the two competing moral theories of deontology and consequentialism and, in line with utilitarian considerations (i.e. balancing risks and benefits), it fits with a pluralist society and market-oriented framing of regulatory issues’ [21, 24]. Indeed, Seger [22] traces this influence back to Hippocrates, and the principle of nonmaleficence behind medical practice itself. Similarly, principlism has been used in AI ethics to provide a starting point for consideration of the issues posed by emergent applications of AI [22].

However, there is a robust body of literature raising concerns about the way in which practical problems arising from the development and use of AI are abstracted into vague terms in ethical principles, as well as challenges in operationalizing AI ethics principles in practice (e.g. [25].). Indeed, many question the efficacy of such principles and the fundamental logics behind them, (e.g., [26, 27]), critiquing this situation in which the fox is trusted to guard the henhouse [28]. This ‘soft ethics’ approach can at the same time be seen to distract attention from the need for hard and enforceable regulations,Footnote 1 and thus risks doing away with opportunities for democratic consultation on AI [31]. This may be achieved through ethics-washing or ethics-shopping, so that companies can ‘wash away the concerns raised by a company’s behaviour or a techno-political crisis’ or ‘strategically ‘shop’ for the principles that limit one’s action as little as possible while simultaneously presenting oneself as contributing towards the common good’ [7, 32,33,34]. It is noteworthy, however, that there is also critique of this tendency towards ‘ethics bashing’—’a tendency, common amongst social scientists and non-philosophers, to trivialize ‘ethics’ and ‘moral philosophy’ by reducing more capacious forms of moral inquiry to the narrow conventional heuristics or misused corporate language they seek to criticize’ [32].

Furthermore, when operationalizing AI ethics principles is facilitated by the use of checklists, binary questions risk ‘framing complex decisions with many competing factors as a deceptively simple compliance process,’ or as issues which can be ‘addressed via purely technical solutions, implemented by individual practitioners’ [6]. Indeed, many of the issues arising from the development of organizational checklists and principles for AI ethics are grounded in their ‘abstract nature,’ which can make them unsuited to operationalization [6]. They additionally tend to focus primarily on how to tweak AI systems, rather than questioning ‘whether AI systems should be built in the first place’ [21].

Despite these drawbacks, organizational AI ethics principles remain influential, and continue to be pursued, particularly as a tool to support developers in considering ethical issues in the development and design of AI technologies [35]. However, relatively little research has as yet examined the processes through which individual organizations develop these ethics principles [6], as these processes often occur behind closed doors. Furthermore, there has been a lack of research on the extent to which members of such organizations are already aware of the trade-offs and issues which are inherent in the creation of AI ethics principles.

This research is especially lacking from East Asian settings, where the development of AI is being rapidly pursued. Japan in particular is a significant case as its government has actively promoted the development and implementation of AI in society, through its vision for ‘Society 5.0’ [36] a society which utilizes AI and big data to solve a broad range of societal issues. Moreover, Japan has been recently spotlighted as a key site for the ethics of AI through the initiation of the G7 Hiroshima Process on Generative Artificial Intelligence (AI) [37]. Japan’s major investments into AI thus make it a frontrunner and also a learning site for other settings in which the development and deployment of AI is being pursued.

There have been initiatives by multiple Japanese companies to create principles for AI ethics, including by major international players such as Fujitsu, Hitachi, Mitsubishi Electric, Panasonic, Sony and Toshiba [38]. Yet, there has to date been little reported collaborative research and development in the Japanese context on organizational AI ethics principles between experts in the humanities and social sciences, and those working on the development of AI. This is problematic given that for ethics principles and their associated checklists to be effectively applied, they must be ‘aligned with teams’ existing workflows and supported by organizational culture’ [6]. Moreover, the inclusion of ‘external perspectives’ in the consideration of strategies for responsible innovation are meaningful [39]. Thus, this work aims to address a gap in the literature by shedding light on the discussions through which a set of organizational AI ethics principles are being developed in Japan.

2 Methods

The process to create AI ethics principles and the associated checklist was started in 2021. This work was conducted collaboratively online between the Osaka University ELSI Center and participants from Mercari Group with researchers from each organization meeting regularly to determine the content of the principles and the checklist. There were three researchers involved from the ELSI Center, with expertise in the humanities and social sciences, including ethics and philosophy, and three researchers from Mercari Group, with expertise in AI engineering. Other experts were invited to participate as needed. YN was engaged with the process from its inception, while AK was involved following its close as an observer.

We analysed the minutes from 20 meetings from the formative stage of this process. Each meeting was around an hour in length, and participants consented to recording and analysis of the data from the meeting. The process was ended in May 2022, based on participants’ perception that they had reached saturation in discussions over the principles, and although subsequent meetings were held to further evaluate the principles and checklist once following internal piloting, these meetings they were deemed to be outside of the scope of this analysis, which is focused on the initial development process. It is noteworthy that, at present, the principles continue to undergo internal piloting, with release intended for an as yet undetermined date, and as such have not been directly referenced in this paper.

Researchers on both sides collaboratively took down minutes from the meetings, noting key ideas. These minutes make up the data source drawn on for this analysis. The minutes were comprehensive, detailing the views put forward by each participant, generally using words close–if not identical–to the original speakers’. This approach was selected as the minutes reflected what participants in the discussions found to be most important from among the topics of discussion in each meeting. Thus, the use of this data privileges the perspectives of those involved and provides additional insights into the decision-making process Footnote 2.

We approached the data with the following research question: what are the issues of importance to the researchers in their discussions of AI ethics within the co-design process for AI ethics principles? To answer this question, the minutes were analysed by the two authors using an inductive process drawing on thematic analysis [40], consisting of open, line-by-line coding of the minutes, followed by the synthesis of codes across multiple meetings. These codes were then collapsed into overarching themes. The creation of these themes led to the data-driven observation that multiple themes from across the selection of meetings analysed were in opposition to each other, suggesting emergent trade-offs, aligned with Barocas and boyd’s [10] observations detailed above. At times, some of these trade-offs were explicitly articulated by participants themselves. Thus, the analysis reported below is focused on these key trade-offs.

3 Results

Through this analysis, we identified four core trade-offs as themes across the range of meetings held (Table 1). We then clustered the considerations within each trade-off into two categories: themes which deal with technology-oriented approaches and issues, and those which reflect human-oriented approaches and issues. Each will be addressed in turn below.

Table 1 Emergent trade-offs and competing considerations
Full size table

3.1 Trade-off 1: relying on data-driven insights vs. preventing the reproduction of discrimination

The first trade-off identified was between the benefits of data-driven insights from algorithmic systems to expand inclusion, and the possibility that they could lead to the reproduction of discrimination and the need to prevent this. Inclusion and fairness were distinct but recurrent themes throughout the discussions. Inclusion was centred around the need to see the individuals represented by data points, to respect the diversity of their values, and to treat them fairly. This included preventing the embedding of prejudicial or arbitrary viewpoints in automated systems and avoiding discriminatory outcomes.

This discussion of the benefits and issues of AI systems came up in consideration of Mercari’s credit-allocation Merpay systemFootnote 3 and how the principles under consideration could be applied to the potential use of AI to facilitate allocation. Drawing on an awareness of the issues arising from credit allocation and from AI, participants expressed concerns that the use of AI in credit allocation systems may lead to exclusionary outcomes towards particular individuals or groups. Yet, this was countered with the proposition that the use of AI in credit allocation could help to expand the availability of credit and could in fact contribute to financial inclusion. For example, whereas an individual with an existing loan may be unable to apply for further credit under typical credit allocation systems in Japan, using data-driven insights rather than relying on what were perceived to be stereotypical judgments based on personal attributes could allow for an expansion in access to credit. This trade-off tied in to a further consideration of how to ensure fairness overall–whether proactive initiatives taking an ‘affirmative action’ style approach, or whether non-intervention would be preferable.

Ultimately, this trade-off was addressed through a consensus among participants that the focus should be on ensuring fair outcomes, such as by fine-tuning algorithms to avoid discriminatory outcomes. To this end, the law was also raised as a key reference point for decision-making and ensuring appropriate outcomes.

3.2 Trade-off 2: advancing automation through the use of AI systems vs. ensuring human-centricity

The second trade-off arose in the balance between imperatives to advance automation and to introduce AI-driven decision-making, while maintaining human-centric systems through a ‘human-in-the-loop’ approach. As participants noted, AI systems had advanced to the point that activities currently handled by humans could in many cases be entirely automated. This then raised the question of how to ensure the ‘human-centric’ approach called for by international principles such as the European Commission’s Assessment List for Trustworthy AI [41]. This led to consideration of how human-centricity could be defined more broadly than just keeping a ‘human-in-the-loop’.

This expanded definition included a perception of human-centric systems as systems that were lawful, ethical and robust. To this end, these systems would reflect an orientation to accountability, human rights, and privacy.

In this way, the trade-off between advancing automation while maintaining human-centricity was addressed by allowing for automation in alignment with organizational aims and technological capabilities, while also adopting a broader, rights-oriented definition of human centricity. Practically, there was consensus around requiring that logs be kept to allow for human reviews of system outputs.

3.3 Trade-off 3: protecting corporate proprietary information and the personal information of users vs. facilitating transparency and accountability in AI system development and use

The next trade-off was concerned with the balance between transparency and accountability on the one hand, and the protection and non-disclosure of proprietary information on the other.

Participants sought to identify ‘core AI values’ based on Mercari's existing Market Principles: safe, trustworthy, and humane.Footnote 4 Translating these Principles into core values would require transparency and accountability as one part of safety and trustworthiness. Yet, there was also awareness that transparency and accountability could conflict with the need to protect proprietary, corporate information or the personal information of users, which could be vulnerable if systems were overly transparent. A further conflict could then arise between transparency and the creation of robust and secure systems. Given that the systems could draw on attributional data, including potentially sensitive data, these were seen to be key concerns.

This ultimately led to the conception of a spectrum of openness and transparency, with disclosure of the use of AI as the minimum permissible standard at one end. Then, further openness beyond this minimum would, for example, involve disclosure of information about the algorithms in use. Thus, this trade-off was resolved by focusing on the informational needs of stakeholders. This meant an orientation towards providing sufficient information about algorithmic functions and their implications to ensure that users were empowered in decision-making, while still ensuring the robustness and security of the systems.

3.4 Trade-off 4: using personalized automated recommendations to facilitate transactions vs. allowing for user choice and serendipity

The final trade-off of relevance here was one which reflects Mercari’s status as Japan’s largest consumer-to-consumer retail and resale platform. This trade-off was centred around finding the balance between, on the one hand, using AI to provide recommendations and facilitate meaningful transactions by enabling users of the platform to discover goods which could suit their needs and interests, while on the other hand maintaining an open market on the basis of free participation. This tied into the open question of how and from whose perspective to define ‘value.’ For example, the use of recommendations could lead to ‘filter bubbles’ through which users would be directed to recommended goods and thus encounter a limited number of potential goods. This was seen to be in contradiction with the open market principles espoused by Mercari. Moreover, recommendations could lead to an ‘uncanny valley’—if they were perceived by users to be overly ‘on the nose,’ they could create misconceptions among users about (mis)use of their data. Specifically, there were concerns that a sophisticated recommendation system could lead users to believe that the system had picked up on sensitive information about users such as sexual orientation or pregnancy that they had not explicitly disclosed, creating privacy issues.

Thus, although there was acknowledgment that recommendations could facilitate transactions of value to users, opportunities for serendipity and user-driven discovery were seen to be of value. For this reason, then, this trade-off was resolved through a focus on the diversity among individual users. Any implementation of automated recommendations would need to be in alignment with this, and avoid being overly prescriptive.

4 Discussion

Overall, the co-design process for these ethical principles was found to be meaningful for researchers both from academia and from industry, as the process allowed academic and research-driven insights to be coupled with on-the-ground realities. As discussed here, the collaboration brought attention to the trade-offs that arise from ethical complexity of AI. Yet, these trade-offs were in many cases made up of competing considerations that were on both sides socially-conscious and oriented to positive outcomes. Thus, the findings of this study offer a more nuanced perspective on the development of ethical principles than what has been reported in prior research. Below, the four trade-offs will be situated in the broader academic literature, showing their relevance to settings beyond the one examined here.

The first trade-off was around the balance between the benefits of data-driven insights from algorithmic systems and the risk that they lead to the reproduction of discrimination. This trade-off points to the tensions inherent in algorithmic systems, which extend beyond this particular case study. Algorithmic systems are generally reliant on past data. Though the use of such data appears to offer relative objectivity and a way to avoid the biases and faulty heuristics in decision-making by individuals, it also risks not only reproducing but amplifying entrenched social inequalities by aggregating the results of flawed human decision-making [20]. Indeed, a growing body of literature has explored the ways in which algorithmic systems draw on prior data to reinforce social biases around gender [19], race [18], and disability [42], for example. Yet, participants’ desire to utilize objective data presents an alternative to Morley et al.’s [11] findings that there is a lack of attention to social good among those creating ethical principles for AI. In this case, there was a perception that algorithmic systems drawing on relatively objective data would be preferable to existing approaches which were seen to rely on problematic biases and reinforce inequality.

Thus, this trade-off draws attention to the benefits of considerations of AI grounded in the social sciences and humanities, as called for by Joyce et al. [15]. Social scientific, and particularly sociological, perspectives help to shed light on the ways in which seemingly objective data hold embedded inequalities. The resolution of this trade-off by participants through their focus on ensuring fair and appropriate outcomes suggests an approach aligned with the principles of ‘design justice’ proposed by Costanza-Chock [43], which call for the prioritization of outcomes over intentions.

It is also noteworthy that discussions around this trade-off brought attention to the need to look beyond data points and consider the diverse individuals they represent. Indeed, Cheney-Lippold [44] has argued, drawing on Deleuze, that there is a need to distinguish between the individual and the dividual–the collections of data points which often stand in for the individual, thus effacing the individual from consideration. Here, the participants’ interest in returning to a focus on respect for individuals suggests a meaningful reversal of this tendency.

The second trade-off revolved around the balance between advancing automation and ensuring human-centricity. Automation is seen to increase profitability and efficiency–principles aligned with Taylorism, scientific management, and the broader imperatives of a capitalist system. A focus on human-centricity can be seen to be an alternative to this, though the extent to which humans should be involved throughout ‘the loop’ remains an open question [35].

Yet, discourses of automation and increasing technological capability tend to obscure the continuing roles of humans in facilitating the development of AI. These are tensions drawn out by Gray and Suri [45] in their discussion of the ‘paradox of automation’s last mile,’ through which human labour is essential at the frontier of automation–as automation advances, certain roles previously occupied by humans are automated and the frontier is pushed further out; yet new needs for human involvement appear at the new frontier. Therefore, Gray and Suri argue that even as automation advances, there will always be a need for human involvement to cover this ‘last mile.’ Moreover, this idea connects to the notion that human involvement and oversight itself has important implications, even if, as some have tried to argue, areas such as employment, medical diagnosis, and criminal justice would be more accurate if completely automated [46]. Thus, participants sought to centre the roles of humans, even while recognizing the importance of automation. This included an expanded definition of human-centricity, which encompasses not only oversight, but also broader human rights and privacy-oriented protections for all humans in the loop.

One important consequence of the expanded definition of human-centeredness in Trade-off 2 is that technical due process, which aims to ensure adequate opportunity to challenge algorithmic decisions, was the focus of the participants. Thus, they were aware of issues in navigating the choice between human oversight and full automation. This also reflects an awareness of recent stipulations within the European General Data Protection Regulation (GDPR; [47]), which call for freedom from automated decision-making, and which have set a de facto international standard.

Trade-off 3 was centred around the trade-off between transparency and accountability on the one hand, and the need to protect proprietary and personal information. This trade-off is linked to a common critique of proprietary algorithmic systems whose workings remain opaque, while this opacity generates problematic power imbalances that disadvantage the most vulnerable [20, 48]. The imperative within organizations to prioritize ‘confidentiality of the most innovative aspects of their R&D activities’ [39] impacts the ability of stakeholders to be meaningfully engaged on these technologies. Thus, participants in this study sought to centre the informational needs of users, to ensure their ability to make informed decisions about their interactions with the systems.

The fourth trade-off dealt with the balance between AI-driven recommendations and maintaining an open market on the basis of free participation. This is a key issue for Mercari–since its inception in 2013, Mercari [49] estimates that there have been 3 billion items put up for sale through its platform. This raises the question of how to ensure that people are able to find what they are looking for given the abundance of options, and thus creating a role for recommendation systems [50]. However, there is concern both within the discussions analysed here, and in the broader literature, about filter bubbles which arise as a result of the widespread use of recommendation systems, through which opportunities for serendipity and novel encounters are lost, leading to increased homogenization in user choice [51]. This trade-off also highlights questions about the role of AI in society more broadly: Zuboff [52] has argued for the ‘right to the future tense,’ critiquing the ways in which the use of algorithmic systems can foreclose possibilities and limit the ability to choose. Therefore, participants chose to centre users, ensuring that recommendations would be deployed in such a way as to facilitate, rather than restrict, user choice.

5 Conclusion

Thus, this analysis highlighted multiple trade-offs which arose in discussions about the ethics of AI at this organization, as participants in the discussions sought to reconcile multi-faceted and at times competing aspects of AI ethics. The presence of the four trade-offs raised here in the discussions about AI ethics principles shows that organizational actors are themselves aware of these trade-offs. Moreover, it indicates that these actors are not just concerned with offsetting potential harms but are oriented towards doing what they perceive to be good for direct stakeholders and for society more broadly.

Next steps towards the implementation of the principles discussed here were beyond the scope of this research. However, there is a need for further research to better understand the long-term impact on organizations and on society of these soft, organization-led approaches to AI ethics, including the relationship between such approaches and the necessity of governmental oversight.

Furthermore, in light of the recent abundance of organizational ethics principles, this exploratory work highlights a need for greater research on how these principles are developed, as well as critical reflections on their limitations and alternatives. Particularly, there is a need for reports on these trends in ethical AI to emerge from underrepresented settings, including the Japanese context taken up in the case study presented here. This study drew on meeting minutes collated by participants to capture the perspectives and priorities of participants themselves. However, further qualitative research could more directly address these themes through ethnographic approaches or interview studies with those involved in the development of AI ethics principles. Here, we show that collaborations between academic and industry stakeholders, based on principles of co-design, are a useful approach to considering these complex issues. Such collaborative approaches provide access to insights into decision-making processes which may be otherwise inaccessible to stakeholders and draw attention to the nuance in on-the-ground considerations of AI ethics.