The Development of Ethical and Trustworthy AI Systems Requires Appropriate Human-Systems Integration

Abstract

The development of Artificial Intelligence (AI) technologies experiences worldwide an ongoing challenge to become trustworthy and ethical for users and the general public. This challenge currently stands between the promise of AI to create immense societal and individual impact and its realization. Because of this, possible large marketplaces still remain hesitant or closed. We have investigated this problem and identified potential solutions in the InSecTT project, a large international EU research and development project that investigates ethical, smart technologies. Thereby, working with industrial and research partners, we assert that developing trustworthy AI technologies is not foremost a technical challenge, but increasingly an organizational and process challenge that results from applying traditional ways of conceiving, designing, and selling technologies to technologies with very new types of user and societal implications. Because AI technologies can shift the role that humans and society see as acceptable, traditional development processes that rely on the strict separation of specialties are overburdened. In our view, a research and development approach for trustworthy AI systems should put human concerns and needs at the start of the development process to effectively integrate humans and systems and thereby have a chance to meet EU guidelines for developing ethical AI. Our proposed approach to develop such ethical, trustworthy AI systems centers around the assessment of trustworthiness risks through intensive user involvement prior to the elicitation of system requirements that are then managed throughout the system’s life cycle. Also, the approach includes concrete recommendations to establish the organizational prerequisites for producing ethical, trustworthy AI systems. In this chapter, we describe the Human-Systems Integration (HSI) approach and motivate the underlying principles in some detail. The chapter intends to inform managers of technical organizations and product managers, as well as principal investigators, to set up the prerequisites for trustworthy AI, while also soliciting inputs for further refinement and discussion.

1 Is Trustworthiness of AI a Problem?

The market for smart technologies that utilize Artificial Intelligence (AI) is growing at high rates,¹ but uptake of these technologies is lagging behind due to increasing lack of user trust and acceptance of these technologies. Among many examples, in 2020, Amazon had to stop selling its face recognition software due to racially biased categorizations and Microsoft soon followed suit.² Similarly, the German Bundesnetzagentur warned against the use of certain intelligent toys because of potential spying on customers.³ There are many examples that have triggered debates on how to create ethical AI systems that are consistent with the interests and rights of their users. This emphasizes the drawbacks of smart technologies that result from the often unintended consequences of users interacting with them in the real world.

Smart technologies often shift the roles and responsibilities of those surrounding them and these impacts are often difficult to see beforehand. A single police officer with biased racial views is one thing but an automated facial recognition system with such biased views shifts the problem to another level as hundreds of thousands of biased categorizations could occur.

Thereby we define a trustworthy smart system as one that facilitates its user’s trust as “… the attitude that an agent [smart system] will help achieve an individual’s goals in a situation characterized by uncertainty and vulnerability” [1]. Trustworthy systems must be lawful, ethical, and robust [2], which makes ethical compliance a necessary but not sufficient precondition for trustworthiness.

Based on the recently proposed EU AI act [3], we consider an ethical system as one that honors the rights of the human user and thereby conforms to moral principles, specifically ensuring the autonomy of humans to decide for themselves, not exploit their data and rights for other purposes, and ultimately serve the human, rather than purely those selling products.⁴

“Smart” functions increasingly take over the cognitive and manual tasks that previously only humans could perform. Smart systems recognize preferences, understand voice commands, keep the distance to the vehicle driving ahead, or provide diagnostic information to medical doctors. Thereby in most cases beyond the most repetitive and simple environments, smart systems do not usually replace the human operator but assist them in their work and daily life. Full automation is often too expensive and not realistic, given liability concerns. A survey among 500 decision-makers from German companies shows that smart AI systems are mostly intended to assist human operators but not to replace them [5]. Whereas fully automated systems are often implied by public debates about the capabilities of AI, the limits are often not clearly stated and fiction and reality are blurred. Thereby, AI-developing organizations are motivated to overstate the capabilities of their smart technologies. This can result in misunderstandings about their actual ability and produce societal backlashes against the technology without understanding the real use cases. Because smart systems cannot yet decide about their contextual enabling and disabling, solve untrained situations, or take responsibility for failures, realistic use cases usually involve the human operator to decide in these situations. This is the case in many different operational domains such as medicine or security: a medical diagnostic system can help the medical doctor during the diagnosis process but not take responsibility for medical decision making; an effective security system can smartly detect anomalies within large data sets and then inform the operator with the needed information to resolve a security breach. These represent new roles for the human and the smart technology needs to be designed for such teamwork.

The main challenge of teaming AI with humans is that humans who are for some periods of time out-of-the-loop, are difficult to bring back-into-the-loop. The user of a highly automated vehicle who is watching a movie and is not aware of an imminent dangerous situation needs some time and active effort to understand the critical situation in order to take back control and safely maneuver the vehicle. This “bringing-back-in-the-loop” can be unsafe and cause risks of accidents or biased decisions. To get back into the loop requires the human to re-establish an understanding of the situation, either through establishing situation awareness themselves (a driver) or receiving explanatory information from the automation (e.g., a medical doctor receiving diagnosis suggestions). Out-of-the-loop problems are not new to human-system integration with complex technologies: nuclear power plants, modern aircraft, and many military applications exhibit high degrees of automation but still require humans for specific decision making. For such large systems, standard development processes have been developed to early on integrate the human role into the system design. Accidents happen if such processes are not followed such as the Chernobyl nuclear disaster in 1986 [6] and the Boeing 737 Max accidents in 2019 [7]. Also, users of end-user-devices have been observing these problems; for example, vehicle navigation systems are known to “occasionally” lure truck drivers to tiny mountain passes due to incorrect map data: it is difficult for even professional drivers to calibrate their trust for a system that seems to work right most of the time but then occasionally fails. As automation becomes pervasive, such experiences will multiply if not appropriately designed for calibrated trust.

The described challenges have been well recognized and are starting to be addressed worldwide. We start with an overview of available guidelines and approaches toward trustworthy AI in the next section. Then we identify remaining gaps that we address by introducing our approach in the subsequent section.

2 Current Initiatives to Address Trustworthiness of AI

2.1 Guidelines and Regulations

Governments and private companies address the challenges to develop ethical and trustworthy AI by proposing guidelines. A review shows that 84 guidelines on ethical AI were published worldwide until 2019 [8]. Analysing the authors of these guidelines reveals that private companies and governments seem to have a common interest in guiding ethical AI development. Private companies (22.6%) provided the highest number of guidelines, closely followed by governmental agencies (21.4%). Furthermore, most guidelines include similar aspects. The requirements transparency, justice & fairness, non-maleficence, and responsibility are represented in a minimum of 71.4% of guidelines. The most frequently stated requirement of transparency (in 86.9%) focusses on explainability, interpretability, data use and human-AI interaction. Figure 1 provides an overview of the principles that are suggested by the AI guidelines.

Fig. 1

Numbers of most-frequently addressed ethical principles

An AI ethics guideline that got high attention and serve as basis for further considerations of AI ethics (e.g., in [9, 10]) was provided by the European Commission’s High-Level Expert Group (HLEG) on AI. The AI ethics guidelines focus on following seven dimensions:

• Human agency and oversight: AI applications should support the user’s agency, autonomy, and decision-making.

• Technical robustness and safety: technical robustness is central for preventing harm. AI applications should be developed to prevent risks and to ensure reliable functioning.

• Privacy and data governance: to prevent harm, the privacy and date need to be protected and used data sets need to be of high quality.

• Transparency: an AI application needs to be traceable and explainable. It should be transparent that an AI is in operation.

• Diversity, non-discrimination, and fairness: diversity and inclusion needs to be ensured throughout the AI application’s lifecycle.

• Societal and environmental well-being: to ensure fairness and prevent harm, the environment should be considered as stakeholder.

• Accountability: auditability and reporting of negative impacts are important.

Building upon the EU ethics guidelines, the European Commission proposed regulations for high-risk AI. The proposed act includes ethics aspects and is currently in the European legislative process [3]. The proposed legislation includes similar aspects of trustworthy and ethical AI and will make them mandatory for systems with high safety risks such as applications for biometric identification, critical infrastructure, or employment access. Such mandatory requirements increase the urgency of the AI industry to adopt ethics assessments in the development processes.

The presented guidelines have a large area of applicability but lack details to support implementation for which several implementation support options have been proposed that will be described next.

2.2 Implementation Support

One type of implementation support consists of checklists that are intended to allow AI developers to quickly estimate how well AI ethics are considered in their AI application. The AI ethics guidelines directly come with a checklist for the ethics criteria (ALTAI). The checklist allows developers to self-assess their AI application regarding ethics requirements. Based on the developers’ responses, the assessment list provides explanations on which ethical aspects are missing. Similarly, checklists exist for software development applications like regression test selection [11].

While checklists are applied towards the end of development, the Eccola approach aims at starting ethical discussions throughout the development process. The approach summarizes different ethics guidelines to provide cards with easy understandable explanations and questions on ethical AI topics [10]. Eccola invites developers to discuss relevant topics throughout their development sprints. Concrete discussions outputs are documented, and the process aims at making developers aware of AI ethics.

The Z-Inspection approach takes the responsibility off the developers and involves interdisciplinary experts [9]. The expert team follows a process to identify AI risks and to provide development with concrete recommendations. Within the Z-Inspection process, socio-technical scenarios are used to identify ethical issues and tensions. Furthermore, the results are mapped with the AI ethics guidelines.

2.3 Observed Gaps in Current Initiatives

The guidelines toward developing ethical and trustworthy AI are comprehensive and go far beyond the amount of available implementation support that is rather limited in detail and depth. For example, while simple checklists are easy to apply by developers, they do not capture critical contextual information outside the developer’s expertise. Also, the checklists are not part of a comprehensive development approach that clarifies responsibilities and tasks for the ethical outcomes of the developed AI system.

Similarly, the Eccola approach raises important ethics related questions but does not per-se prioritize the questions and does not provide success criteria. Also, other domain experts are not foreseen to participate which the limits the outcomes purely to the perspective of the developers.

Z-Inspection provides a process of how interdisciplinary experts develop recommendations for specific AI applications, but again, no responsibilities and tasks are assigned to actually implement them. We expect that considering the ethics recommendations from the Z-Inspection will add development costs. In consequence, an organizational process to ensure the consideration of AI ethics in actual development is required.

The needs for an ethical AI development process that includes an organizational framework becomes apparent when analyzing the proposed EU AI act. Based on a review of the EU AI act, we categorized a selection of regulations (as shown in Fig. 2) and complemented them by aspects out of the EU ethics guidelines that are not included in the AI act.

Fig. 2

Impact areas of EU AI act and EU ethics guidelines

The algorithm requirements bring the need for continuous engineering processes that do not end with fielding an AI system [12]. In detail, algorithm requirements specify how AI models are developed, how AI is documented, how data is handled, and how the system’s activities shall be recorded. In traditional production, the manufacturers’ active role ends with selling a product or, at the latest, when the product warranty ends. However, such active role is already extended by the security requirements for software products that require continuous updates. The proposed AI act brings a further extension and makes risk and quality management throughout the system’s life cycle mandatory for high-risk AI systems.

Functionality requirements bring the need to involve users in developing AI systems from early on. The upcoming regulations require high-risk AI to be designed transparently, so that users can “interpret the system’s output and use it appropriately.” Furthermore, they shall be designed and developed so that they “can be effectively overseen by natural persons” (Human Oversight) [3]. As a consequence, functionality requirements enable the user to interpret the system’s output and oversee its activities. We argue that approaches of conducting user testing on prototypes and final systems are not sufficient per-se anymore because they often come too late to have a high impact on the product’s concept. Therefore, intensive involvement of stakeholders from the very beginning of AI development is necessary.

3 From Technology-Centered to Human-Centered Development of Smart Technologies

As outlined in Sects. 2 and 3, the development of trustworthy smart technologies requires shaping the development process toward the specific user and user context situations out of which ethical and trustworthiness issues only emerge. This can be difficult in traditional development environments where technology-centered development processes shape the role of users and their tasks quasi as byproducts of the technology (as shown in the block (A) in Fig. 3). This can result in “unbalanced” systems where the tasks may not be acceptable and trustworthy by users. Instead, what is needed for trustworthy development is shown to the right of Fig. 3, block (B), where the joined considerations of user, tasks, and technology, as well as task environment lead to a balanced systems where trustworthiness and acceptability are part of the whole development process. The second process seems necessary to conform with the EU AI act.

Fig. 3

Technology (a) versus orchestrated (b) processes

To exemplify the differences between approach (a) and (b), let’s imagine a developer implementing a facial recognition system that recognizes criminals and handing it to the users, without knowing or clearly considering the possible consequences of algorithmic problems. Therefore, the technology is then transitioned to an operational use (a). As a result, the users (and the larger public) find out that the system incorrectly categorizes minority members more likely as criminals than majority members. This is unintended and results in loss of trustworthiness on the societal level. In contrast, following the approach (b), the developing organization undergoes a detailed analysis of the use-situation and extracts such risks of biases and addresses them as part of the technology development. This is, in a nutshell, what the EU AI act attempts to do.

Therefore, a key aspect to develop trustworthy systems is to sufficiently analyze and address the specific context of use and involved users and tasks early on and allow this to shape the product design and development. Specifically, the analysis of the user and use-situation should result in a set of trustworthiness risks, i.e., risks that, if not addressed, may lead to loss of trustworthiness. Those trustworthiness risks are managed in a life-long risk management process and thereby guide the product development and life-long operational process.

Moving from technology-centered to human-centered development methods requires an orchestration framework and process, as detailed in the following.

3.1 Orchestrating the Development of Ethical and Trustworthy AI

In order to move from traditional technology-centered approaches to human-centered approaches, we came to realize that is necessary not only to first establish appropriate organizational structures, but then also to establish the necessary processes for organizations to interact and achieve trustworthy outcomes (see e.g., [13]). The situation of developing AI resembles the development of safety and security critical systems, where many requirements originate from the use of the system in its real application context, that are often not available at design time until explicitly brought into the process through specific analytic and research activities. The safety challenges of an airplane originate in the real world of flight operations. Similarly, security breaches in the real world bring knowledge to prioritize security requirements. A similar situation results out of the use of AI systems during actual operations that make apparent the ethical and trustworthiness considerations that should have been considered during design time.

3.1.1 The Human Systems Integration Framework

Realizing trustworthy systems means therefore to “break down the silos of excellence” within which most normal technological developments currently occur. Traditional development organizations often focus their expertise on innovations at the technical level that knows little or nothing about the actual objective or mission: the experts of Machine Learning algorithms usually have no idea about the requirements of an end-user to understanding the outputs of the algorithm for his/her work environment. This is however critical for acceptance. Technological competences are necessary but not sufficient prerequisites for ethical and trustworthy products.

One approach to break down the silos of excellence is through applying the Human Systems Integration (HSI) framework that postulates three interconnecting cornerstones: (A) an organization that is able to conceptualize and investigate the use of technology within a sociotechnical context, (B) a holistic development organization that is able to identify solutions in a strong multi-disciplinary effort, and (C) a life cycle long learning and maintenance operations that addresses continuously changing aspects of the system. These cornerstones are linked via tools, processes, and standardized certification schemes that help to bound the solution space and aid collaboration and teamwork, as shown in Fig. 4.

Fig. 4

HSI framework to facilitate trustworthy system’s development processes

HSI Cornerstone (A)—“Assess, Understand, Conceive” is to provide the necessary information about the intended use-situation for the development of smart systems that are to achieve sustainable acceptance and use. Such information includes the context of use, the goals to be achieved, as well as the user needs and important limitations of use and their situation, and forms an essential starting point for the system design. Technical feasibility and cost-effectiveness are thereby concept-forming factors equal to the usage situation information; this is a novelty here. Such information about the usage situation is only available to the developers of today's systems to a limited extent. Usage situation information also includes characteristics of the user population and the tasks to be performed, including criticality, responsibilities, and influences of the organizational environment, as well as the work environment. In particular, organizational context and processes within which the system is used are important for design decisions, for example, to select appropriate methods of explaining smart technologies to the user. Data collections include observations, interviews, surveys, analyses, and especially virtual methods that allow users to make contextualized assessments (e.g., driving or flight simulators), as well as physical methods (e.g., Wizard-of-Oz studies). User and context are captured and translated into a high-level vision for how the system consisting of human, technology, and task & situation constraints could work.

HSI Cornerstone (B)—“Design Holistically” translates the vision from cornerstone (A) into a holistic design of the system. The word “holistic” means that orchestrated teams of multidisciplinary specialists work together to develop solutions across the various discipline-overarching dimensions. In the so-called “living labs” products are co-designed to achieve a trustworthy, acceptable, and safe usage of the systems. This serves as a point of convergence across the disciplines and teams. Technical and cost factors act as limiting modulators. The challenge consists of making these larger contextual perspectives visible and overcome traditionally isolated disciplinary hierarchies so that experts from different disciplines can effectively work toward such convergence. This requires sufficiently large, multidisciplinary research environments in a climate of positive holistic goal orientation and go beyond use and stakeholder abstractions as “matchstick men” (see the block (A) in Fig. 3). Especially virtual simulation and modeling tools can support this process to combine the expertise of human factors, science, and the various technical engineering disciplines.

HSI Cornerstone (C)—“Continuously Adapt” consists of continuous adaptation and updating of products, as well as the education of users during the lifecycles of smart technologies are expected. System adaptations require detailed information about user and usage conditions. This requires a certain level of trust so that the user does not feel exploited or observed but sees himself as part of an improvement cycle. This also includes the possibility of user feedback which can not only promote user trust but also requires it. In addition to product adaptations, it is also important to promote the standardization of user knowledge and digital competencies in the form of standardized competence modules that enable users to find their way over time in what is otherwise perceived as a digitalization jungle. The creation of European training curricula for end-users, employees, and employers is a goal that must be initiated by technology developers, as this is where the critical information in the HSI process is available. The implementation of the digital competence modules in training curricula will then take place at the European level.

3.1.2 The HSI Process Model

Whereas the HSI framework postulates the organizational prerequisites for trustworthy and ethical smart technologies, how are these cornerstones stitched together into a working whole?

In Fig. 5, the HSI process model shows the orchestration of the three cornerstones. Boxes 1–4 (green) indicate the processes in cornerstone (A), including the risk management process. Boxes 5–9 indicate the activities of cornerstone (B). Box 10 indicates the activities of cornerstone (C) [14].

Fig. 5

HSI process model for the orchestration of trustworthy systems

Critical in the HSI process model are the interactions between the three cornerstones to maintain the focus on the overall user experience concerning trustworthiness and ethical acceptability.

3.1.3 Extraction of Trustworthiness Risks Using Scenario-Based Methods

Scenario-based methods are commonly used in user-centered development efforts (e.g., [15]). Such methods can be used to identify risks by imagining the user within realistic and concrete environments risks [4]. As outlined above, the principles of ethical and trustworthy AI need to be contextualized in specific use conditions to become meaningful (see box 2 in Fig. 5). Otherwise, they are too abstract to be able to derive specific requirements for implementation.

With a scenario we mean here a description of how an intended function can be accomplished under a realistic set of use conditions and stakeholder characteristics. A scenario thereby makes constraints visible that remain otherwise invisible. A risk consists of the description of a situation that, if it became real, would expose an undesired danger. We suggest that risks are at its core, formulated in simple sentences containing a precondition and a consequence, for example: “If a driver is not informed about his/her responsibilities, the risk for accidents is increased.”

The scenario description is the results of an analysis in which the ethic trustworthiness criteria are asked as questions: “What could that criteria mean in a specific condition for a specific user?”. In Table 1 we give examples of how this can be done around partial driving automation (SAE Level 2 [16]) that supports driving with longitudinal and lateral control, but leaves the driver fully responsible for monitoring the assistance and stepping to manual driving anytime when needed.

Table 1 Building blocks mapping to components and scenarios

How can scenarios help to identify risks? Figure 6 shows a scenario that brings out EU ethics criteria in a concrete context and a concrete user. The scenario description focuses on a specific stakeholder. To ensure diversity, different scenarios representing different drivers (young/older, gender, etc.) are required in praxis.

Fig. 6

Example scenario to show the link between scenario and trustworthiness criteria

4 Conclusions

The key to creating trustworthy and ethical smart technologies consists of integrating humans and the system from the beginning of the system design and having the processes and organizational structures in place that allow to do so. Whereas current AI guidelines already prepare the principles for ethical and trustworthy AI, they are lacking the implementational considerations that paramount for realizing such systems. In this chapter we described a Human-Systems Integration approach with the organizational preconditions consisting of three functional cornerstones to (A) systematically assess the context of use and insert this into the (B) holistic designs, and to (C) continuously adapt the system while keeping the human in the loop. These organizational cornerstones are interveaved using HSI processes centered around continued risk-management of trustworthiness and ethical risks. We have introduced several methods to facilitate the assessment of trustworthiness and ethical risks using scenario-based methods.

Standard engineering processes in companies of today are often rather isolated in their specialties and separate from user and use contexts. This hinders the development of trustworthy, ethical systems as foreseen by the EU rule on AI and many guidelines that have emerged worldwide. Instead of seeing such guidelines as burden and impediment, we propose that unique selling propositions can be created through offering products that are trustworthy and ethically aligned. In the end, such products are closer aligned to the needs and constraints of end users than systems that are produced without detailed knowledge of the context of use and therefore are more prone to sink within the storms of user-indignation and public outcries. We had started this chapter with a reminder of some of these and hope to have contributed methods to avoid such commercial and human mishaps while still taking advantage of the immense capabilities of AI for systems from which everybody profits.