1 Introduction

The industrial sector has evolved from mechanization and steam power to mass production and automation, known as Industry 4.0. This digital revolution incorporates technologies such as Additive Manufacturing, Internet of Things (IoT), Edge and Cloud computing, Simulation, Cyber-Security, Horizontal and Vertical Integration, and Big Data analysis for efficient task execution [1].

To prioritize resilience, societal well-being, and economic growth, the European Commission (EC) introduced Industry 5.0, aiming to transform traditional factories into resilient providers of prosperity [2]. Artificial Intelligence (AI) plays a crucial role in this transition, helping in automating and getting insight in large-scale production and customization.

Building user confidence in AI products is essential, achieved through risk mitigation and facilitating AI technology adoption. Trustworthy AI (TAI) and human-centric AI integration are key challenges that need to be addressed [3]. Human-centric AI focuses on optimizing performance while prioritizing human needs, improving working conditions, and fostering a sustainable and socially responsible industrial ecosystem.

We provide an overview of the transition from Industry 4.0 to Industry 5.0, highlighting the use of AI and advanced technologies to create sustainable, human-centric, and resilient industries. It explores the challenges and opportunities of adopting TAI, offering strategies to mitigate risks and promote effective adoption.

The presented strategies for risk mitigation in TAI assets align with established approaches in the industry, particularly the Process of Risk Management (PRM) based on the ISO 31000 standard. Combining these approaches, we propose the TAI-PRM framework. The framework is designed to meet current and future regulatory requirements for AI, with specific goals that include:

  • To support management units and developers in incorporating trustworthy requirements within the AI life cycle process.

  • To secure the use of AI artefacts independently of legal and technical changes. The legislation heterogeneity applied in different countries on the use of AI can be varied; therefore, flexibility is key.

  • To ease the combination of the frameworks that handle ethical-driven risks with other approaches commonly used for PRM. It needs to be designed as a complementary asset to these used by the industry and not a replacement to facilitate its adoption.

  • To facilitate an iterative process to handle risks on AI artefacts within the framework. Many processes in software do not follow sequential development but spiral/iterative development processes. Therefore, the framework must flexibly adapt to any development cycle applied by developers for its incorporation.

  • To ensure that Key Performance Indicators (KPIs) can be tracked through well-defined metrics that register the progress on the identified risks handling. Tracking KPIs is essential for their daily operations and business units. In addition, managerial levels can use these indicators to understand the impact of incorporating ethical aspects.

  • To construct an architecture that addresses personal responsibilities and channels of communication. With this aim, the framework must foster communication between technical and non-technical stakeholders.

  • To foster the reuse of outcomes in other research areas and market segments to avoid duplication of effort. This is translated as savings in revenue and research time on future developments. In addition, well-structured risk identification can avoid the repetition of failing conditions on AI components with similar target objectives.

  • To enable a seamless path for the transition to Industry 5.0. By linking ethical considerations and risks, companies can handle TAI requirements as a PRM.

  • To provide users with a tool for performing the TAI-PRM and evaluate approaches already developed for TAI.

1.1 Understanding the Transition to Industry 5.0

Industry 4.0 incorporates cyber-physical systems and digital platforms into factories to improve production processes and decision-making [4]. Different technologies have helped shape this industrial revolution. The impact of these technologies extends beyond industry, including home products, business models, clean energy, and sustainability, surpassing those of previous industrial revolutions. Industry is recognized as a driving force for sustainable transformation, necessitating consideration of societal and environmental aspects [5].

Previously, technologies focused on economic optimization, neglecting sustainable development. To address this, there is a growing demand for a shift to circular economies, prioritizing well-being, social governance, environmental efficiency, and clean energy. In 2020, the EC organized a workshop that gave rise to the concept of Industry 5.0. This vision, incorporated in industry’s future roadmap, combines AI and the societal dimension as enablers [6].

Industry 5.0 entails integrating technologies to optimize workplaces, processes, and worker performance, emphasizing collaboration between humans and machines instead of replacing one another. This human-centric approach promotes the development of technologies that enhance human capabilities. To promote the adoption of Industry 5.0, the EC has established initiatives like the Skills Agenda and the Digital Education Action Plan, aiming to enhance the digital skills of European workers [7, 8]. The Commission also seeks to boost industry competitiveness by accelerating investments in research and innovation, as outlined in the Industrial Strategy [9].

Environmental sustainability is a key consideration for the EC, which promotes the use of resources efficiently and the transition to a circular economy through supporting initiatives such as the Green Deal [10]. Additionally, to promote a human-centric approach, initiatives and regulations have been developed that include the Artificial Intelligence Act, the White Paper on Artificial Intelligence, and the Trustworthy requirements [11,12,13].

Industry 5.0 introduces new challenges and technological enablers. These include real-time decision-making, human-centric solutions, edge computing, and transparency. Managing approaches and AI techniques can support and drive these advancements. Meanwhile, challenges from Industry 4.0, such as security, financing, talent, data analytics, integration, and procurement limitations, remain relevant [14, 15].

In terms of technological enablers, different approaches have been recognized, which includes Individualized Human–Machine Interaction, Bio-Inspired Technologies and Smart Materials, Digital Twins and simulation, Data Transmission, storage, and analysis technologies, AI, and Technologies for Energy Efficiency, Renewable, Storage, and Trustworthy Autonomy [16, 17].

1.2 AI, Trustworthy AI, and its Link to Industry 5.0

Industry 5.0 builds upon Industry 4.0, integrating ethical considerations for AI assets while prioritizing humans. It aims to promote societal well-being through the collaboration of humans and machines in smart working practices. Ethical concerns in AI vary across domains and application types, with unique considerations.

Trust development emerges as a common challenge, crucial for instilling confidence among users. The EC emphasizes TAI as a foundational ambition, acknowledging the significance of trust in advancing AI and establishing a robust framework [18]. Ensuring trustworthiness becomes imperative in AI technology, as reliable interactions between AI agents and humans are essential [19]. AI technology providers must address risks related to performance and user impact, emphasizing collaboration between AI-driven systems and humans, considering transparency, reliability, safety, and human needs, to foster user acceptance [20].

The interaction between humans and AI in manufacturing can be categorized into three types (human-in-the-loop HITL, human-on-the-loop HOTL, and human-in-command HIC), depending on the level of human involvement. Importantly, the decisions made by these systems can be influenced by humans and data, thus impacted by societal, legal, and physical considerations.

According to the High-Level Expert Group set up by the EC [21], TAI has three main pillars, which should be met throughout the system’s entire life cycle: it should be lawful, ethical, and robust. To deem an AI Trustworthy, different requirements must be addressed, depending on the intrinsic risk of the AI assets. These requirements are related to Human Agency and Oversight, Technical Robustness and Safety, Privacy and Data Governance, Transparency, Diversity, Non-discrimination, and Fairness (DnDF), as well as Environmental and Societal Well-being, and Accountability. The Trustworthy requirements [13] aim to minimize risks associated with AI and its potential adverse outcomes throughout the AI life cycle. These requirements align with five key principles derived from 84 ethics guidelines: transparency, justice and fairness, non-maleficence, responsibility, and privacy. They ensure that AI systems respect fundamental rights, are secure and reliable, protect privacy and data, are transparent and explainable, avoid bias, promote stakeholder participation, and are subject to accountability mechanisms.

1.3 PRM and Considerations for AI Implementation

In a 2020 study, Hagendorff [22] provides guidelines and strategies for addressing critical AI issues, focusing on areas of greatest attention. However, the study lacks specific definitions for handling AI assets throughout their life cycle [23]. Moreover, the implementation of these approaches in different domains and environments can present specific challenges that need to be understood [20].

Different organizations have developed diverse methods based on multiple ethical principles to facilitate practitioners in developing AI components. These organizations include academia, trade union, business, government, and NGOs.

To ensure a trustworthy component, the actions of any agent over humans must be reliable [19]. This definition depicts the linkage between trust and risk management, in agreement with the definitions in the EC AI Act [11]. By defining a proper methodology that tracks and manages risk components derived from trustworthy requirements, the probability of producing adverse outcomes is minimized.

Developing a framework that ensures the ethical use of AI while minimizing potential harms and maximizing benefits requires a comprehensive approach. How can such a framework be implemented in the manufacturing sector, considering existing approaches?

PRM involves identifying, assessing, and controlling risks that can impact systems and organizations. Integrating ethical considerations into PRM processes requires defining the concept of ethical risks (E-risks). We propose the following definition: “E-risks are conditions and processes that can disrupt the expected behaviour of an AI asset due to the lack of consideration of TAI requirements, including values, social, legal, environmental, and other constraints.”

The PRM involves intervention, communication, and involvement of various areas within the managing companies. A risk management framework consists of three key components: Risk Architecture (RA), Strategy (S), and Protocols (P), forming the RASP strategy. RA provides a formal structure for communication and reporting, S defines implementation strategies, and P includes guidelines and procedures for managing risks [24].

The risk management policy statement plays a crucial role in the framework. It outlines the organization’s strategy and approach to PRM, aligned with its objectives and tailored to the specific environment. However, when establishing enterprise policies for AI management, minimal considerations may exist due to regulatory requirements, such as the Artificial Intelligence Act [11], which may influence the policy’s content.

ISO 31000 is widely used as a standard for PRM in industry. It provides procedures for assessing, treating, monitoring, reviewing, recording, and reporting risks. ISO 31000 is often combined with other specific standards, such as ISO 9001 for supply chain and product quality improvement [25] and ISO/AWI23247 for the digital twin manufacturing framework [26]. However, these standards need to be reviewed and updated to align with the requirements of smart manufacturing and Industry 5.0 [27].

Although ISO 31000 does not explicitly address application security risks or provide guidance on implementing RASP controls, organizations adopting the RASP framework can benefit from considering ISO 31000 principles and guidelines. ISO 31000 promotes a systematic and proactive approach to PRM, which complements the objectives of the RASP framework.

The TAI-PRM adheres to ISO 31000 and adopts a RASP approach [28, 29]. TAI-PRM provides a supporting structure for communication and reporting of failures, implementation strategies, and guidelines for managing risks. In this chapter, we focus on developing and testing the protocols for the PRM, based on ISO 31000 with some considerations. The communication and consultation activities are part of the RASP architecture, not the PRM itself, and the Ethical PRM includes the Monitoring and Review process as a primary component after Risk Evaluation and Risk Treatment.

The mentioned works [28, 29] provide a scope contextualization for understanding the PRM and evaluation, as well as a groundwork for AI development and management, establishing connections between requirements and risk components.

2 Failure Mode and Effects Analysis

The Failure Mode and Effects Analysis (FMEA) method is commonly used for risk assessment in various industries, including the technology sector. When it comes to assessing risks derived from TAI considerations, the FMEA method can be an effective tool for several reasons:

  • Holistic approach: The method can be applied on the entire system, rather than just focusing on individual components. This aligns with the principles of TAI, which emphasizes a comprehensive approach to ethical and technical considerations.

  • Identifying potential hazards: TAI requires identifying potential hazards, assessing their likelihood and impact, and taking steps to mitigate them. The FMEA method provides a structured framework for this, helping also to uncover hidden hazards.

  • Systematic approach: The method involves a systematic approach to risk assessment, helping to ensure that risks are addressed in a consistent and comprehensive manner, which is critical for TAI.

  • Continuous improvement: The FMEA method is designed to be an iterative process. This aligns with the principles of TAI, which emphasize the need for ongoing monitoring and continuous improvement to ensure that AI systems remain ethical, transparent, and accountable over time.

The key steps of the FMEA are outlined next. These steps should be performed consecutively with the exception of the Identify Failure, Detection Methods, and Existing Risk Control; Analyse Effects; and Identify Corrective Actions:

Define the Analysis

Developing TAI systems involves defining constraints, failures, and objectives while considering the system’s context. Trustworthy requirements, including explainability, are outlined in the AI Act [11]. Adapting these requirements to the industrial context and AI asset goals is crucial. Methods for achieving TAI depend on functionalities, data usage, user understanding, and objectives [11].

Development of System Functional Block Diagrams

Various supporting documents, including block diagrams, analyse failure modes, and their impact. Detail level should correspond to the AI asset’s risk level, with worksheets and matrices as valuable tools. Additional information such as system and AI boundary descriptions, design specifications, safety measures, safeguards, and control system details are necessary for risk analysis.

Identify Failure Modes

Identifying failure modes is crucial in understanding system failures. Certain failure modes align with the needs of TAI-PRM have already been identified. For instance, [30] define failure modes in IT safety. FMEA has been applied to TAI with limited success, focusing mainly on fairness [31]. To expand the scope of failure modes, we propose incorporating eleven ethical-based failure families derived from system norms and TAI requirements. These encompass failures related to robustness, safety, transparency, accountability, societal well-being, environmental well-being, human agency and oversight, privacy, data governance, bias (DnDF), and users’ values. This approach facilitates detection and metric definition.

Identifying failure modes involves measuring observable conditions using supporting protocols. We propose to group the conditions based on drivers such as physical, social, data, user/system interface, and algorithms. Physical drivers include power supply, communication/data link cables, robot parts, wearables, lenses, and sensors. Internal social drivers relate to stakeholders’ values and biases, including social responsibilities and ethics-in-design. Data drivers encompass sources affecting AI trustworthiness, such as biases, quality, quantity, and security. User and system interface drivers are linked to inadequate usage, absence of information display, tutorials, guidelines, and user’s ill-intentioned usage. Algorithm drivers include problem-solving processes and expected functionality during AI code execution.

Identify Failure, Detection Methods, and Existing Risk Control

Detecting and managing failure modes using metrics and methods to reduce risk conditions. Adequate intervention procedures are needed for high-risk AI components. Linking further actions, such as control devices and circuit breakers, is crucial to enhance system understanding and reduce risk. A lack of detection methods could affect system robustness, security, and transparency. The importance of these actions should be linked to the intrinsic risk level of the AI assets involved.

Analyse Effects

This step analyses the consequences of failure modes, including the end effect and its impact on social-driven components and each HSE (Health, Safety, and Environment) element.

Identify Corrective Actions

Involves identifying actions that can prevent or reduce the likelihood of failure, comply with legal requirements, ensure safe operation of the system, recover from failures, and incorporate norms based on user values and AI trustworthiness requirements.

Ranking

Assigns values to each failure mode’s likelihood, severity, and detectability, which can be used as KPIs for estimating the TAI state.Footnote 1 The Risk Priority Number index (RPN) with respect to a concrete failure mode enables the normalization of the AI artefacts risks. The RPN mathematical expression is \(RPN_{item}=S \cdot O \cdot D\). Here S is the severity (scale of 1–10, with 10 being the most severe), O the occurrence or likelihood (scale of 1–10, with 10 being the most likely), and D corresponds to the detection ranking (scale of 1–10, with 10 being the least capable). The RPN indicates the risk level of the failure mode, with a higher RPN indicating, indirectly, a higher risk for the AI component.

To evaluate the overall risk of an AI component on different failure modes, the Global Risk Priority Number index (\(GRPN_{item}\)) can be calculated by summing the RPN of each item i to its corresponding failure mode ratio; \(GRPN_{item}=\sum _{i=1}^{n} RPN_i\alpha _i\). In this equation, i represents the different Failure Modes linked to the same source; n is the number of failure modes of a specific component; and \(\alpha _i\) is the failure mode ratio and represents the part attributed to a concrete failure mode if the failure materializes. This means the percentage of the AI asset to fail with the specific Failure Mode.

The Failure Mode Ratio can be estimated as \(\alpha _{i}=\frac {e_{i}}{e_{\mathrm {tot}}}\). \(e_{i}\) is the amount of failing conditions of a specific failure mode, and \({e_{\mathrm {tot}}}\) is the total amount of all failing conditions in the system. The failure mode ratio can be used to group risks with the same trustworthy requirement, by a proper weighting, thus identifying the most significant TAI risk consideration for the AI asset.

Tabulate and Report

The report includes documentation and a repository for users to understand failure modes, risks, control measures, safeguards, and recommendations.

3 TAI–PRM Protocol

An overview of the TAI-RPM is shown in Fig. 1—more details at [29]. The figure shows an abstraction of the PRM with activities related to AI artefacts and TAI considerations.

Fig. 1
A flowchart has the following flow, A I confirmation, decision box of is A I in any component, e-Risk identification and classification with A I modification, decision boxes of is the e-Risk acceptable and can be modified to be acceptable, A I scope definition, analysis of values, and e-Risk management process.

Benchmark e-risk management process

Full size image

The diagrams show the starting point (black circle), activities (boxes), decision points (diamonds), and endpoints (crossed circle). Some activities contain sub-flow charts, denoted by a [+] symbol, that follow the same approach explained here.

As seen in Fig. 1, the first activity is the AI confirmation, which involves identifying AI elements by defining and categorizing them.Footnote 2 Next, the e-Risk identification and classification activity focuses on identifying the AI elements’ intrinsic level of risk under regulatory conditions (i.e., given the current regulatory approach in the EU, we focused on the risk levels defined by the AI Act [11]). This activity has previously been thoroughly discussed [28, 29], so it will not be covered here.

If AI classification is deemed not acceptable in the mentioned stage, the user will proceed to either the AI Scope Definition activity or the <can be modified to acceptable?> node—Fig. 1. This node assesses whether the AI’s approach, data characteristics, and functionalities can be modified to meet TAI considerations for high-risk assets. If modification is feasible, the AI modification activity is executed to redefine the AI, followed by another round of <e-risk identification and classification>. If it is not possible to secure an AI asset’s acceptable risk level, the design, deployment, or use process should be halted or decommissioned.

In Analysis of Values (see [28, 29] for more information), the potential for biases and use of personal information that could impact the requirements of DnDF are evaluated. It also considers the level of automation that is left to the AI asset and how this affects the agency of humans in decision-making processes. This activity also allows integrating users’ values into the system. If there are conflicts, well-documented approaches for decision-making (e.g., ANP or AHP) can be used to resolve it. Finally, e-Risk Management Process encapsulates the risk assessment, treatment, monitoring, and review—described in Sect. 3.1.

3.1 e-Risk Management Process

Figure 2 shows a high-level diagram of the ethical-driven PRM.Footnote 3 Each of the activities in the figure is discussed next.

Fig. 2
A block diagram has the following flow, establishing context, merging with other risk management processes, risk analysis and evaluation, risk treatment of transfer, tolerate, and terminate, estimate K P Is with risk register and monitor, update interactions and requirements, and review, update, and implement.

High-level overview of the e-Risk Management Pipeline

Full size image

Establishing Context

In this activity, a gathering and run of information is performed. This is done by: (1) Define how the AI asset interacts with other components and subsystems (i.e., software architecture documentation), (2) Define hierarchical extension of the components. Users must track cascade effects on risk analysis, for example, using root cause analysis. (3) Analyse the interactions between AI elements actions, UIs, and humans, defined in the activity AI scope (Fig. 1 in Sect. 3). (4) Analyse constraints set up as requirements to control functionalities, input behaviours, system outcomes, and component values. If relevant, these constraints were established with the physical context, enhancing the system security, especially in AI–user interaction. (5) Analyse diagrams that include Connectivity with other components and subsystems, Dependencies, and Human–AI and Human–UI interactions. (6) Analyse collected information from Requirements and values.

Merging with Other Risk Management Process

When several PRM instances are identified and similar approaches are used (i.e., FMEA), the analyses can be merged (e.g., merging with DFMEA and PFMEA processes; the DFMEA involves a comprehensive analysis of systems, while the PFMEA identifies and evaluates failures in processes).

Risk Analysis and Evaluation

The FMEA (or Criticality Analysis (CA) if information exists—FMECA) is used as core component for risk analysis. Given the depth of the content, this activity is described in detail in Sect. 4 as it contains a detailed sub-flow—core in the management of e-risks.

Risk Treatment, Transfer, Termination, or Tolerate

Depending on the risk appetite and the risk levels, this activity evaluates if an AI asset should be: Treated: Failing conditions can be modified, upgraded, or safeguarded. Transferred: External safeguards will allocate the responsibility of the failure events if materialized—Conditions can limit the option to transfer e-risks for TAI requirements. Tolerated: No need of AI asset modification. Periodic updates on the status must be continued according to the frequency established in the risk management policy. Terminated: The AI asset should not be used or developed. These activities are known as the 4T’s of risk management. This process and its activities are further extended in Sect. 5.

Estimate KPIs, Risk Register, and Monitor

The KPIs should be linked to each trustworthy requirement or value defined for AI and their risk level measured. The monitoring part of the activity involves analysing and evaluating stakeholder outcomes to trace contingency actions, and the implementation of PRMs. The risk register involves the process of Tabulate and Report of the FMEA.

Update Interactions and Update Requirements

The outcomes of the previous process involve modifications related to AI assets and their interactions with other components, data structures managed by other AI assets, and the incorporation of additional AI assets or new functionalities that can impact the system’s trustworthiness. Therefore, it is necessary to analyse these new interactions to identify potential risks. The iterative process of running the PRM must be tracked for accountability.Footnote 4

Review, Update, and Implementation

Once there are no outstanding updates, the user must implement the risk treatment strategy, which depends on the interactions between the different personas involved in the PRM as described in the Risk Architecture [29]. In addition, the status of the failure modes has to be revised, along with the mechanisms and processes to implement the 4T’s—secure protocols, strategies, and control mechanisms for implementing the AI asset. Internal or external auditing processes can be used to evaluate the system status and the PRM. Finally, the risk register must be updated for accountability.

4 Risk Analysis and Evaluation Activity

Figure 3 extends the component described in the previous section. The figure depicts a flowchart that defines the instruments between the approaches FMEA, FMECA, and RCA (Root Cause Analyses) used for the risk assessment.

Fig. 3
A risk analysis and evaluation flowchart defines protocols, identifies failures, limits top events, checks whether A I updates are needed, determines the stage of A I system, modifies the system, examines the robustness, and quantifies the risk evaluation, among others to use F M E A and R C A for various analysis.

Risk analysis and evaluation flowchart overview

Full size image

The first decision node, <Protocol defined?>, analyses if the user has defined an FMEA or alternatives (i.e., RCA—The RCA is a well-documented approach beyond the scope of this chapter, and therefore, it is not further explained.) for risk assessment. If no, the set of decision nodes drives the user to the most convenient approach to follow. The set of questions are:

  • <All failures identified?>: Whether it is aimed to identify all possible failing conditions. This means that the user is interested in detecting every situation that might trigger risk outcomes.

  • <Top events limitations?>: This question focuses on the circumstance that the number of failure events is large or could be unexpected.

  • <AI updates needed?>: When an AI asset requires human intervention or software updates, FMEA has extensive applicability and efficiency.

  • <AI system early stage:?> When the system is in the design or definition phase—An FMEA approach can better help to detect conditions that could lead to system failures.

  • <System modified?>: This decision node analyses when the system will be modified considerably in future stages.

  • <Robust examination?>: If AI assets are used in critical systems, or AI failures can have severe impacts on users and the environment, this decision node will allow the user to decide if a robust examination is required.

  • <Quantity risk evaluation?>: Since FMEA allows classification and grouping into failure modes in terms of their occurrence, severity, and detection, this decision node facilitates the definition.

  • <Human error failure?>: This decision node addresses the systems that require considerable human intervention.

  • <Profile of dependencies?>: If failures can trigger cascade events when they materialize, FMEA can foster a proper identification of system interdependence.

  • <Concern with events?>: If explanation of the relations between failures that can lead to severe consequences and impacts is required.

In case the user is led to Use FMEA activity—detailed in Sect. 4.1—information regarding qualitative and quantitative evaluation must be gathered. Two decision nodes are used to define these: <Failure probabilities?> and <Expert judgement?>. These led to the two activities in Figure Qualitative criticality analysis? and Quantitative criticality analysis?.

Within them, a CA could be performed. In order to be performed, additional information, such as: (1) the Failure Mode Ratio (\(\alpha \)), (2) the conditional probability \(\beta \), which represents the probability that the failure effect will result in the identified severity classification, and (3) the \(\lambda \), which represents the overall system failure rate due to different causes over the operating time in units of time or cycles per time, are needed. Then, the Criticality Number (\(C_M\)) provides a metric to classify a specific failure mode of an AI asset as follows \(C_{m}=\beta \alpha \lambda \). The Overall Criticality Number (\(C_r\)) estimates how critical an AI asset is with respect to a complete system; \(C_r=\sum _{i=1}^{n} (C_m)_i\).

Furthermore, a heat map and a risk matrix must be constructed to keep the quantitative information tracked and to incorporate the numerical analysis based on the probabilistic information collected about the failure modes (as described in Sect. 4.2).

4.1 Use FMEA Activity

Figure 4 shows the FMEA pipeline. The process starts with a decision node, <design risk analysis defined?>, where the user can check if PFMEA or DFMEA protocols are run in parallel with TAI-PRM. When this occurs, the next decision node, <scope enable ethics?>, checks if the scope of the framework and the ongoing approaches can be merged or extended. This means to define components, items, dependencies, and to establish similitude between policies. An activity named Define and Merge DFMEA/PFMEA defines the strategy to extend the functional blocks—if the process is running.

Fig. 4
An F M E A flowchart defines or merges D F M E A or P F M E A, analyzes each stage, identifies design failure modes, identifies ethical failure modes, analyzes model or root cause occurrence, analyzes severity, detects failings, causes of potential failure, and provides recommendations to fill F M E A forms via various steps.

FMEA pipeline

Full size image

The decision node <AI lifecycle considered?> checks whether the user has considered analysing the complete AI asset life cycle. If that is the case, an FMEA approach must be considered for each stage of the AI life cycle and analysing the risks involved during each phase: design, development, use, and decommissioning.

The following decision node, <All failures identified?>, is an internal check that evaluates that all relevant failing conditions, and their drivers, have been considered for the AI asset. After this, it is required to check if the AI asset is used within maintenance or operational processes. If it is, a PFMEA process must be set for the combination of both processes. This must be executed by the user within <PFMEA in parallel?>.

Independently of the path that the user takes for the FMEA flow, TAI-PRM provides two activities key to identifying failures. The first, named Identify Ethical failure modes, defines the scope of the analyses of ethical, trustworthy, and values to be considered by the user. The second activity is Identifying and/or design failure modes, where, based on the design of the AI asset, the user must analyse possible system’s general failure modes and based on the analysis as overall system including DFMEA and PFMEA considerations (i.e., not ethical base only).

A list of over 130 Failure Modes related to trustworthy considerations have been already identified.Footnote 5 Users can access this list in the repository mentioned or through the tool described at the end of this chapter. Importantly, the possible Failure Modes are not limited to those presented, so users can expand the knowledge base for Failure Modes based on e-risks. This list is based on a literature and industrial feedback from case studies. Furthermore, the tool constructed to perform TAI-PRM (described in Sect. 6 of the document) further facilitates the identification of failure modes and adds the capability to incorporate and share Failure Modes between tool users.

After identifying the failure modes, a ranking activity of these must be executed.Footnote 6 As observed in the figure, for each failure condition—Model/Root—on each component, the likelihood/occurrence rank (O), Severity (S), and Detection (D) must be considered. It should be considered that if the detection (D) is performed by an entity external to the system, for example, throughout IoT devices, constraints of the communication channels should be implemented. This could drive the risk appetite to be set in a more stringent condition.

The final three steps of the FMEA process focus on documentation for improving the detection and accountability of future failure modes. The first, Causes of potential failure, focuses on the keeping control on the failure mode causes for future corrections.

In the case of Recommendations and estimate the Risk Priority Number—RPN activity, failure compensating previsions, functionality extensions or restrictions, or AI asset modifications should be documented to prevent/reduce the likelihood and severity, or improving the detection of the failure modes. Finally, in Fill FMEA forms, the user must fill the Risk Register.

4.2 Heat Map Construction

The heat map—also referred to as risk matrix—provides a mechanism to identify and compare AI artefacts and failure modes associated to the risks. This component is directly linked to a process of Fig. 3 with the same name.

Authors in [32] detail how to construct a risk matrix. Nevertheless, an extension can be made depending on the intrinsic risk level of the AI asset, as defined in the AI Act.

The basis for the risk matrix is the risk definition, which is the combination of the severity of a risk when it materializes and its likelihood. To describe the risk, a classification can be used for severity and likelihood following qualitative descriptions and scales.Footnote 7 As a result, the failure modes can be allocated on a matrix constructed based on the likelihood scale, the severity scale, and the risk score, by using a direct translation of the risk appetite.Footnote 8

4.3 Perform Analysis

To link the heat map with the AI Act, different risk appetites should be considered, depending on the intrinsic risk of the AI asset—i.e., the higher the risk of the AI asset, more stringent must be the risk appetite.

Table 1 imposes the contingency actions for the risks associated to the 4T’s, described in detail in Sect. 3.1.

Table 1 RPN ranges in function of the intrinsic risk level
Full size table

Further refinement is required as further knowledge is obtained from implementing the PRM in AI assets and our use cases. Importantly, users can modify the proposed ranges based on the risk policies established in their own enterprises, always considering the constraints of the AI Act.

5 Risk Treatment Transfer Terminate or Tolerate Activity

The process associated with the 4T’s of risk management is part of the larger process depicted in Fig. 2. The process should consider if the PRM process is repeated or new. If it is repeated, the user should determine if new failure modes have been identified for the AI asset. If new failure modes are identified, a new evaluation of the failure modes is necessary using the 4T’s analysis.

If there are no new failure modes, a confirmation over the possible modifications of the risk appetite should be performed.

The process should be run accordance to the specific needs (requirements and intrinsic risk condition) of the AI asset and thus specifying the actions of Treat, Transfer, Terminate, or Tolerate over their explainability, transparency, accountability, human agency and oversight, privacy, and robustness, among other conditions. Each of these analysis leads to specific activities that provide recommendations and options to address the corresponding risks.

A flowchart is provided in a repositoryFootnote 9 that also includes a decision node for other requirements and an activity called “Framework Construction” to handle new approaches and requirements. Finally, the “Risk Processing” activity summarizes the previous analyses and guides the user to determine whether to terminate, tolerate, or treat the AI assets based on the risk appetite and recommendations.

Overall, the flowchart provides a structured approach to assess and manage risks associated with AI assets, considering various factors such as failure modes, risk appetite, explainability, transparency, accountability, privacy, and robustness.

6 Validation and Real Case Scenario

The TAI-PRM validation process was executed on multiple AI assets as a cooperation between different manufacturing companies and research institutions involved in the ASSISTANT project [28, 29, 33]. The project involves five academic and seven industrial partners. The real case scenario has been used to develop and refine the framework with their inputs over multiple iterations. It is currently in use for its improvement, given the iterative process defined in its implementation.

Furthermore, a tool was developed to facilitate the implementation of the framework and evaluate the ALTAI tool and its applicability in the manufacturing sector.

The rest of this section describes the ASSISTANT scenario as a case study. The case study description starts from Sect. 3 of the TAI-PRM.

7 TAI-PRM Tool

A tool based on TAI-PRM has been developed for the ASSISTANT use case. The tool is accessible via the https://assistant.insight-centre.org/ page and offers comprehensive information on the PRM. It includes two main sections with user feedback. Users can access information on how to use the ALTAI tool or the TAI-PRM process. The tool is not an alternative to the ALTAI tool but rather helps to record the linkage and usefulness of Trustworthy requirements, component assessment, and AI implementation functionality. The records will be compared to provide users with trends based on their domain. Users only interested in the TAI-PRM process need not complete the ALTAI component.

To use the tool, select the My TAI-PRM tab and create as many PRMs as needed. Complete the pipeline before generating a downloadable report. The tool also allows users to create and share failure modes to extend knowledge based on the presented strategies. Users can provide feedback at the end of each TAI-PRM to improve the tool.

A total of eight steps are needed to perform the TAI-PRM including: (1) Initiation of PRM, (2) E-risk identification and classification, (3) AI Scope Definition and Analysis of Values, (4) Establishing Context, Merging with other PRM, and defining criticality analysis or FMEA, (5) FMEA or CA, (6) Ranking, (7) Risk Register, and (8) Treat, Terminate, Tolerate, and Transfer.

The tools follow a distribution similar to the ALTAI tool to familiarize users with both elements. Facilitating the acceptance of user’s that have already driven ALTAI assessments.

Finally, information from the analogue of the ALTAI tool can be linked to a specific TAI-PRM, so information generated from it is also used in the final report of the TAI-PRM.

8 Conclusions

In this chapter, we explored the current paradigm in which Industry 4.0 is evolving towards Industry 5.0, where AI and other advanced technologies are being used to build services from a sustainable, human-centric, and resilient perspective.

Furthermore, we present a framework for developing and designing responsible AI artefacts in the manufacturing sector. The TAI-PRM approach combines PRM and TAI, utilizing the failure modes and effects analysis (FMEA) method. The methodology includes defining risk appetites and strategies in the industrial sector based on the AI Act. It also incorporates the 4T’s of risk management.

TAI-PRM can be used to implement trustworthy considerations and manage risks in compliance with current regulations. It can also serve as a guide for developing standards in AI risk management. Feedback from various domains emphasizes the importance of considering the human factor in risk management processes. However, the lack of regulatory conditions, certification, and standards for managing AI poses obstacles to the adoption and implementation of transparent and accountable AI systems. Addressing this challenge requires substantial training and effort to transform business and development team operations.