1 Introduction

By claiming that AI research and development has to become an interdisciplinary field that should include the humanities and ethics in particular, the notion that AI ethicists are pivotal in the quest for “trustworthy”, “value-based”, “human-centered”, “beneficial” AI has become mainstream, along with the demand to hire professional AI ethicists in corporations [1], using designations like chief AI ethics officer, chief trust officer, ethical AI lead, AI ethics and governance, or trust and safety policy advisor [2]. Such a hiring decision can have undeniable advantages for AI organizations. These range from proactively detecting noncompliance issues, unwanted technological consequences, fairness biases, accountability gaps, privacy violations, transparency issues, security vulnerabilities, strategic decisions concerning AI products, and the like. But apart from these advantages, embedding ethicists or AI ethics principles in AI organizations bears a range of specific risks that are hitherto disregarded in the field of AI ethics itself. Hence, this comment discusses and lists risks that either originate from AI ethicists themselves or from the consequences their embedding in AI organizations has. The compilation of risks comprises psychological considerations of thinking about ethics and acting (un)ethically, subject-specific and knowledge constraints AI ethicists often succumb to, the complicated professional role of AI ethicists, the missing impact of AI ethics guidelines, as well as negative side effects of ethics audits for AI products.

2 Risks

In the following, several types of risks are discussed that are tied specifically to the inclusion of professional AI ethicists, ethical considerations, or ethical audits in organizations researching and developing AI systems and applications. Itemizing these risks is rather unusual for the field of AI ethics and seems to contradict the field’s very purpose. Accordingly, the AI ethics literature is nearly completely focused on stressing the positive effects and importance of ethics considerations, conceptualizing ethics as a service, and eventually putting ethical principles into practice. On the other hand, self-critical approaches are rather rare [3]. This comment does not aim to diminish the importance and necessity of ethical reflections in the AI field. Rather, it stresses that implementing AI ethics is not as straightforward and self-explanatory as it seems to be. In fact, well-meant ethical intentions can have inadvertent, hidden side effects that, if evaluated on their own, would be deemed unethical. This comment wants to list these side effects. They are not necessary but, in many cases, likely concomitant with AI ethics. This means that whenever claims on AI ethics or AI ethicists are made in this paper, this does at no time comprise AI ethics in general or all AI ethicists. This comment cannot make any quantitative statements. However, what can be claimed is that instances of the risks that are mentioned in the following sections do exist to an extent that they are worth discussing.

2.1 Role-model risks

In general, one can uphold a “boost hypothesis” that assumes that ethical considerations have a positive effect on moral behavior and attitude–behavior consistency. However, this hypothesis does not bear closer examination. Empirical research has shown that professional ethicists—which include AI ethicists—do not appear to behave morally better than non-ethicists [4,5,6,7,8,9,10]. Studies on the moral behavior of professional ethicists in the context of theft, voting participation, membership in the Nazi party, littering, honesty, caring behavior, etc. found that, on average, ethicists do not behave differently compared to non-ethicists of a comparable socio-economic background. A possible explanation for this phenomenon is that professional ethicists mainly focus on very abstract, obscure thought experiments that are often detached from issues of everyday life. Another simple but plausible explanation is that mere knowledge about ethical issues and theories does not affect behavior. Indeed, research in moral psychology shows that ethical reasoning is mainly a post hoc rationalization of already entrenched opinions and behavioral routines, that it is the result of automatic mental processes that are relatively detached from conscious reflection on moral principles, and that factors other than moral reasoning significantly influence moral judgments and moral behavior [11,12,13,14,15,16,17,18,19]. Moreover, empirical research has shown that endorsing moral principles and possessing explicit philosophical expertise about ethics does not enhance the consistency of the application of these principles or the stability of moral judgments against the influence of biases [20]. Professional ethicists succumb to biases, inconsistencies, post hoc rationalizations, superficial differences, irrationality, or other psychological factors that they would not endorse upon reflection to the same degree as non-ethicists. This happens despite high levels of academic expertise and training and even though one would assume that expertise in ethics serves as some kind of protection against these factors [21, 22]. In addition to that, research shows that people typically have a biased perception of their own ethicality, deeming that they behave more ethically than they actually do [23]. Similar insights hold true for another meta-bias where people are able to identify others’ susceptibility to biases but they see themselves as immune to these biases [24].

One must suppose that all the mentioned factors do hold true for AI ethicists as well. Hence, they are no different than non-ethicists, likely to have a flawed self-perception, deeming themselves more ethical and bias-sensitive than they actually are. In addition, they may very well not live up to the expectations regarding their own moral integrity when it comes to their actual behavior. Hence, while one may already be skeptical about ethicists’ participation in AI research and development contexts due to their limited technical expertise (see Sect. 2.3), one may be also skeptical when AI ethicists are depicted as individuals with particularly wise conduct, as role models, or even as personifications of morality and virtues. However, it is obvious that one should not exclude AI ethicists from organizational structures in research and academia. Rather, AI ethicists have to become knowledgeable in moral psychology and able to reflect on the psychological forces that impinge on them. One cannot separate personal belief systems, biases, or behavioral traits from one’s professional work for instance on AI policies or assessments. The former always influences the latter [25, 26]. AI ethicists, due to the nature of their profession, must be particularly self-aware of this correlation. If they do not keep it in mind, non-ethicists like AI practitioners are in no way less competent or versed in reaching morally right and ethically reflective decisions in their field compared to professional AI ethicists. This is, as research on moral psychology suggests, due to the fact that other factors than ethical expertise and knowledge predominantly determine moral behavior and decision-making. First and foremost, these factors comprise peer influences, situational forces, as well as intrapersonal factors of bounded ethicality [27]. AI ethicists should learn to deal with these factors that act upon their very own considerations and decision-making.

2.2 Bounded ethicality risks

When using a naïve ethical action theory, one can assume that individuals go through three phases. First, they perceive a confrontation with a moral decision they have to make. Secondly, they reflect on ethical principles and come up with a moral judgment. And finally, they act accordingly to these judgments—and, therefore, morally right. However, such a naïve action theory is flawed in several regards. As mentioned before, moral judgments are, in most cases, not influenced by moral reasoning [11]. Moral judgments are made intuitively, and moral reasoning is used in hindsight to justify one’s initial reaction. In short, typically, moral action precedes moral judgment [28]. This leads to specific consequences for AI ethics. Typically, AI ethicists may be tempted to mainly convey mere knowledge (instead of practical skills or motivations) about ethical issues that are related to AI. However, up to now, they do not consider the many hidden psychological forces like powerful unconscious biases, blind spots, situational forces, and the like that are likely to thwart ethical decision-making not just in themselves, but also in AI practitioners [27, 29]. In research on moral psychology, those factors are subsumed under the umbrella term “bounded ethicality” [27, 29]. Factors of bounded ethicality can, as long as AI ethicists do not acknowledge and address them in AI practitioners, increase the likelihood of unethical decision-making, rationalize it, or reframe it in a misleading way. Hence, to effectively improve ethical decision-making in AI practitioners, AI ethicists must go beyond simply conveying knowledge about ethical issues. Instead, they should engage in initiating measures for ethics training that foster virtues, motivations, or character dispositions that are potent to protect against factors of bounded ethicality and help “debiasing” ethical decision-making [30]. In practice, AI ethicists who are embedded in academia or industry settings are mostly supposed to ensure legislation requirements, to evaluate AI systems along ethical criteria, to embed ethical principles into the business use-case, AI design, and deployment, or to facilitate public debates [2]. Hence, providing behavioral training is not a common practice for AI ethicists, besides its importance in increasing the likelihood of ethical decision-making. In this context, especially business ethics resources describe plenty of measures for ethics training in organizations [31,32,33,34,35]. This training, as soon as it is specialized on AI-related topics, clearly falls into the range of responsibilities of AI ethicists. If they only engage in composing and discussing high-level frameworks of principles and refrain from also putting a focus on cognitive biases, value–action gaps, overconfidence, implicit biases, in-group favoritism, moral disengagement, self-serving biases as well as many others, they will not contribute to the full extend to the quest for reducing the likelihood of unethical behavior in AI organizations [36,37,38,39,40,41,42,43]. However, up to know, AI ethics neither discusses bounded ethicality in AI ethicists themselves (Sect. 2.1) nor how to address it in AI practitioners despite the decisive role methods to circumvent effects of bounded ethicality would have.

2.3 Non-expert risks

Whereas AI research, or the computer sciences in general, is driven by numbers and data by nature, ethicists sometimes claim to have special access to non-quantifiable areas of human life, to areas that cannot be “translated” into numbers [44]. Hence, ethics seems to have an epistemological advantage compared to the former since it can, so the argument goes, grasp emotions, beliefs, desires, or values that are deemed unquantifiable. But as long as ethics fosters such self-concepts, its discourse lacks a decisive enrichment, namely empirical, and with that, quantitative underpinnings. This does not mean that quantitative data should bear more weight than value-centered, qualitative arguments. However, to produce sound scientific insights, both should be combined, instead of just relying on the latter. However, philosophy, and hence philosophical, non-interdisciplinary ethics, has a long tradition of embracing the naturalistic fallacy or the is-ought problem [45], meaning the tenet that one cannot base normative statements on observations about what is. This tenet led to a severe neglect of empiricism in practical philosophy and an amnesia of the value and importance of empirical results for ethical reasoning [46, 47]. Similarly, AI ethics research works stemming from philosophy often put together arguments without delving into empirical evidence, meaning technical details, about particular AI systems. This has in many cases led to redundant, abstract discussions on topics like trust, autonomy, existential risks, fairness, explanations, moral robots, etc. However, if practical philosophy intervening in AI-related ethical questions would also possess a certain understanding of engineering or respective areas of computer science and be able to empirically investigate AI systems first-hand, this could be a significant enrichment for AI ethics.

In theory, the requirements for a professional AI ethicist who does not only focus on traditions of practical philosophy must include an overview of computer science, psychology, sociology, law, and many others. AI ethicists cannot be pigeonholed into one of these disciplines [2]. Traditionally, applied ethics is a strict subfield of practical philosophy, though. Philosophy, in turn, has a reputation for being a rather conservative subject that engages in research mainly by exegesis of classical philosophical texts but does very seldom yield knowledge that is not reliably determinable by other sciences. Philosophy intellectually grasps current societal and technological phenomena, but when it does, it often focuses on speculative scenarios that, by its very nature, do not require detailed empirical knowledge. Therefore, one can assume that philosophers typically rely too much on introspection and thought associations in their arguments, rather than on empirically observable facts. In the AI field, this circumstance manifests in the popularity of fictitious topics like machine consciousness, artificial moral agents, trolley problems, robot rights, AI “narratives”, superintelligence, or other futuristic or speculative scenarios. For interdisciplinary AI research settings that possess tangible real-world implications, these are rather bad premises, though. To the best of my knowledge, hitherto there has not been a study that gathers data on the actual level of technical knowledge about machine learning among AI ethicists. Hence, only anecdotal evidence can support the tentative claim that, in many cases, AI experts with a background in computer science are better ethicists than genuine AI ethicists themselves. This is because the most pressing problems that are discussed in AI ethics, like fairness, robustness, privacy, explainability, etc., can be ameliorated by technical means. In the end, putting ethics into practice happens to be an issue in AI developers’ everyday world. Despite that, AI ethicists often work with assessment lists, principled frameworks, and other deontologically contextualized codes. However, linear auditing mechanisms or abstract principles are not compatible with agile technology development [48]. In fact, motivational settings or character dispositions regarding particular “AI virtues” [30] as well as detailed how-to knowledge on bias mitigation, robustness gains, differential privacy, explainable AI, etc. are decisive in achieving “ethical AI”. However, both aspects only make sense when internalized by AI practitioners, whereas AI ethicists are mostly bystanders.

2.4 Effectiveness risks

One of the main results of the field of AI ethics is the surge in publications of AI ethics guidelines [49, 50]. Lists of abstract principles that are at the core of many guidelines are not the right ingredient to bring about change, though. This argument can be further underpinned by empirical research. In a controlled study by McNamara [51], researchers critically reviewed the idea that ethical guidelines serve as a basis for ethical decision-making for software engineers. In the survey, test subjects were 63 software engineering students and 105 professional software developers. They were presented with eleven software-related ethical decision scenarios, testing whether the influence of the ethics guideline of the Association for Computing Machinery (ACM) [52] influences ethical decision-making in six vignettes, ranging from responsibility to report, user data collection, intellectual property, code quality, honesty to the customer to time and personnel management. The results are disillusioning. The main finding was that the effectiveness of codes of AI ethics is almost zero and that they do not change the behavior of practitioners. No statistically significant differences in the responses were found across test subjects who did and did not read through the ethics guideline [51]. All in all, the study shows that knowledge about ethical issues does not affect behavior. It is a lesson on the ineffectiveness of applied AI ethics, or, to be more precise, the current methods ethicists are using to propel their field.

This ineffectiveness of the intended effects of codes of ethics may be accompanied by another effectiveness risk resulting from moral communication, or, in other words, attempts to persuade others of the importance of ethical principles. In theory, AI ethicists should possess significant influence in all departments of organizations researching and developing AI applications. However, the decisive question is how this influence takes place. By training, ethicists do not acquire persuasion techniques or knowledge of methods for strategic communication [53, 54]. However, the only means ethicists have to decrease the distance between actual and target states towards the latter are not money, power, or other “symbolically generalized communication media” [55], but persuasion techniques. That means that ethicists provide normative orientations, engage people’s motivation to head towards normative goals, and shape the path towards them [56]. The respective persuasion attempts can vary in their nature, though. The most obvious way of making persuasion attempts is through the use of moral language and vocabulary, meaning to stress good what is good and bad. However, this will often trigger confrontational situations, especially when contradictions between ethical and business requirements arise. And these situations, then, are likely to trigger reactance phenomena, backfire, or boomerang effects [57,58,59,60]. Reactance occurs when pressure is applied to individuals regarding a desired change in their normative attitudes or when individuals feel that their behavioral freedoms are to be diminished. In addition to that, reactance implies a reinforcement of attitudes that are contrary to what is actually desired by ethicists [61]. Hence, well-intended ethical interventions can, in cases where strong and confronting moral language or other suboptimal persuasion techniques are used, result in the exact opposite of what was intended. In such a case, having AI ethicists in place bears the risk of actually strengthening unethical tendencies in organizations. To avoid that, AI ethicists should be trained in the psychology of ethically motivated persuasion and strategic communication. However, under normal circumstances, such training is neither part of ethicists’ academic education nor their daily work.

2.5 Ethics washing risks

Since involving ethical assessments in AI research and development is not mandatory, it is likely that AI organizations will only accept overhead expenses and install ethicists in cases where they already expect their AI products to be well rated. Organizations, however, that fear significant reputational losses as a result of serious ethical scrutiny are unlikely to initiate it in the first place. If that hypothesis is correct, AI ethicists are most needed where they are least likely to be called upon. Hence, ethical oversight and associated consequential costs are likely to disadvantage those organizations that are already economically disadvantaged due to their more ruthless competitors. A simple test to see whether ethical considerations are taken seriously in corporations is to look for cases where they refrain from seizing business opportunities even though they would be completely legal. Relinquishing monetary advantages, however, goes against the fundamental system conditions of the company in question, which leads to ethics becoming a “natural” opponent of economic success, at least as long as it is deemed to be some kind of supplementary external auditing entity. However, since incorporating ethics into corporations can at least boost their reputation, a twisted version of ethical assessments can be embraced.

In this regard, AI ethicists and ethics discourses can be deployed at the forefront of companies and AI organizations, while business as usual continues behind the scenes [62]. In an empirical study by Vakkuri et al. [63] about applied AI ethics in industrial settings, the researchers conducted semi-structured qualitative interviews in various companies. When asked whether their AI development practices take into account ethical considerations or ethics tools, all respondents answered “No” [63]. However, according to the study, AI developers consider the physical harm potential of AI systems, and responsibilities are approached pragmatically from a financial, customer relations, or legislative stance. What is salient here, though, is that these opinions correspond to established moral attitudes that most people have internalized anyhow, but that are detached from what is actually discussed in AI ethics, like, for instance, the evaluation and governance of structural, social, or emotional effects of AI systems. In general, papers on AI meta-ethics regularly ascertain that AI ethics is “toothless” [64], has a fig leaf function [65], is a mere invention of the AI industry [66], a puppet of industry stakeholders [67], used for whitewashing purposes [68], an escape from binding legal norms and regulation [69], and the like. All in all, AI ethicists may be willing tools for interests that are actually situated outside of ethical settings. The existence of AI ethicists may allow the successful and persuasive conduct of actual malpractices like choosing ethical principles that retrofit pre-existing behaviors, making misleading or unsubstantial claims about ethical measures, exploiting ethics to delay or avoid necessary regulation, etc. [62] in the first place. Without AI ethicists or the embedding of ethics discourses into AI organizations, such malpractices and the associated abuse of trust would not play out since referring to “ethics” would not work. Notwithstanding, one may be tempted to accuse companies of ethics washing when they hire ethicists and to accuse them of ignorance if they do not hire ethicists. Such contradictoriness is obviously not productive. Ultimately, professional AI ethicists are always dependent on their employers, and that raises the likelihood of them not objecting to participating in ethics washing practices. However, it is true that such a restriction is not a necessity that a priori allegations without any underpinning through actual cases are counterproductive, and that AI ethicists can bring about real and authentic changes towards beneficial and ethical AI applications.

2.6 AI authority risks

A common tenet when discussing the use of AI systems in high-stake areas is to deem these systems as mere decision support tools, but not as decision makers themselves. However, this is too short-sighted since AI systems have a particular kind of authority, a semblance of an objectively functioning, mathematically precise, and neutral tool—despite the fact that they will under any circumstances contain errors and unfairness [70]. Especially in situations with high stress or time pressure, human decision makers who are supposed to merely consult AI recommendations may be prone to suspend this tenet and simply take the AI decision for granted, showing trust in the “truth-telling” power of technology. Algorithmic decision-making promotes defensive human decision-making [71], meaning that humans tend to offload the strain of decision-making onto computers. Medical, military, or other practitioners in high-stake areas must be aware of AI authority.

Ethics-based audits, AI impact assessments, ethics consulting, AI fairness tools, ethics-as-a-service offers, ethics labels for AI systems, and the like [48, 72,73,74] are supposed to make AI systems trustworthy, but in this way, they can actually exacerbate the problem of defensive decision-making. The more trustworthy an AI system is supposed to be, the more following the respective algorithmic decisions becomes the most defensible decision one can make, though perhaps not the best. And the more AI systems are implemented in high-stakes areas, the more comprehensive the auditing requirements become. And with that, the trust in the system and thus the defensive decision-making increases, although making decisions that one believes are the best for clients or organizations, even when it entails deviating from second-best algorithmic decisions, becomes all the more important. In fact, defensive decision-making is mainly a problem in the medical field, where practitioners tend to show “assurance behavior,” meaning ordering unnecessary tests, medical imaging procedures, drugs, or surgery since this decreases the likelihood of malpractice litigation [75]. But, on the other hand, the respective decisions can very often deviate from what is best for patients. AI systems exacerbate this situation in two ways. First, when fed with training data on past medical decision-making routines, they perpetuate patterns of these routines and therefore mechanize suboptimal decision-making procedures. And second, as mentioned above, especially medical AI systems are subject to ethical scrutiny [76], hence trust in the outputs of these systems is likely to be increased, too. And with that, users are even more prone to AI-induced defensive decision-making.

Obviously, dynamics leading to defensive decision-making cannot and should not be avoided by refraining from auditing AI systems, by not increasing their trustworthiness via impact assessments, labels, etc. However, AI ethics should at least account for the potential that well-meant or mandatory audits of AI systems can exacerbate the problem of defensive decision-making. The constitutive dynamic described here is neither caused nor a necessary accompanying effect of ethics-based influences on AI research and development. However, it is a potential, undesired side effect that in and off itself should be considered in a self-reflective impact assessment of AI ethics itself.

2.7 AI avoidance risks

AI authority is one side of the coin. The other side is AI skepticism [77]. While trusting AI recommendations too much, being skeptical of them can also lead to suboptimal outcomes. Skepticism towards AI systems can, among other things, result from a lack of value alignment. This lack can also be fostered by AI ethicists. Usually, and as described in the last chapter, it is argued that trust in AI applications and hence their adaption can be fostered when ethicists play a part in auditing them. Otherwise, severe social opportunity costs associated with AI system underuse may occur [48]. However, one can also argue that it is exactly the other way around. Following Bonnefon et al. [78, 79], one could use the example of self-driving cars. Ethicists would very likely opt for a utilitarian regulation that states that autonomous vehicles should strive to minimize casualties, even if this means sacrificing their own passengers for the greater good. Alternatively, they opt for a complete prohibition of determining different values of human life [80]. However, such an implementation of ethical principles or prohibitions in AI systems bears the chance of people opting out of using the very systems, which nullifies all their potential benefits. When AI ethicists demand a limit on the overall number of accidents and hence the greatest safety benefits for all, meaning passengers and pedestrians, this could paradoxically result in thwarting these very goals since consumers would likely be inclined to ditch such cars and hence the safety benefits that are connected to them [79]. This scenario could also hold true for other applications. Bonnefon et al. [78] mention sentencing [81] and organ allocation algorithms [82]. Here, making these systems more “ethical” in the sense of minimizing rearrests or giving younger people priority for organ transplants could again cause paradoxical effects that induce people to ditch these systems despite their likely advantages. Other examples may involve all AI systems that promise certain benefits but offer easy ways to opt out in case of a disagreement with amendments encouraged by AI ethicists. In the end, AI ethics audits should always be two steps ahead, meaning that ethicists do not just consider immediate, but also secondary technical consequences. This, in turn, may mean discarding obvious ethical decisions in favor of non-obvious, seemingly unethical ones. And furthermore, this means that AI ethics has to question its fundamental negativity bias. This bias, which causes the field to concentrate on negative issues and events rather than positive ones (which also holds true for this paper), goes hand in hand with or even triggers the negativity bias of media reports on AI topics. This way and by not stressing the positive sides of AI technologies, AI ethics assists in diminishing the reputation these technologies possess for the wider public. However, this can again lead to severe opportunity costs that occur due to the underuse of potentially life-saving or life-improving AI technologies. Hence, AI ethics—besides its strong preference to shed light on fairness issues, black box problems, privacy breaches, etc.—should learn to also have an eye on the positive events and positive sides of AI to prevent a situation where benefits are given away due to (unsubstantiated) fears regarding the likelihood of machine biases, accidents, or other harms.

3 Conclusion

Intuitively, it seems that AI ethics is a discipline that necessarily reduces the likelihood of unethical outcomes in the AI field. But this intuition does not hold true under scrutiny. In fact, one can identify several risks that either originate from ethicists themselves or from the consequences their embedding in AI organizations has. These risks relate to psychological considerations about bounded ethicality in ethicists themselves, to considerations regarding the individuals who react to (or ignore) ethical principles and advice, to the complicated professional role of AI ethicists, the lack of impact of AI ethics guidelines, or the potential negative side effects of ethics audits for AI products. With that, this comment does not aim to diminish the importance of AI ethics. Quite the contrary, its objective is to increase self-reflection and thus the effectiveness of the very discipline. However, the comment also stresses that implementing AI ethics is not as straightforward and self-explanatory as it seems to be. Well-intentioned ethical considerations can have unintended, hidden side effects that, if evaluated on their own, would be deemed unethical. These side effects should be avoided to turn AI ethics into a discipline that can live up to its own standards.