1 Introduction

Contention and turbulence continue to surround the landscape of artificial intelligence (AI)-based systems. On one hand, advances in machine learning are creating technological miracles in countless domains such as healthcare [1], astronomy [2], and education [3]. On the other hand, a plethora of reports and studies highlight examples of AI systems having negative effects on well-being [4], showing discriminatory and biased behaviours [5], and producing in-explainable results [6]. AI technologies continue to balance on a tight wire, with the potential of tipping one way towards further unchecked harms and the marginalisation of several communities, or the other way towards being an inclusive, human-centred technology capable of actualizing benefits across various spheres of influence [7]. One of the biggest barriers to achieving the latter is the lack of human-centred priorities in the field when it comes to initial considerations [8], design [9] and evaluation practices [10], and the lack of external participation in creation processes [11]. These considerations and activities should be outlined in the design processes used to create AI-based systems and not be reliant on creators’ interpretations and left to their discretion. Currently, several design processes exist for AI-based systems. While some only consider technical aspects, others focus on the socio-technical considerations and activities needed to ensure the resulting system is more human-centred, responsible, or ethical. Despite the benefits of an increased socio-technical perspective and increased participation offered by the latter, few explicitly account for fostering value-sensitivity or offer support for the methods outlined by value-sensitive design (VSD).

VSD is a methodology capable of tipping AI technologies in the right direction. By focussing on the values that matter to those who will be using and are affected by the proposed systems, its creators can ensure that the results encapsulate the voices of relevant stakeholders, are meaningfully beneficial to them, and are in line with their needs and expectations. Such a focus takes a solid step towards more inclusive and human-centred AI. Value-sensitivity can also promote reflexivity amongst AI practitioners [23], helping to address some deeper issues such as the homogeneity of the AI workforce or the reliance on largely biased data [5]. In fact, when collaboratively designing AI-based systems, such as conversational AI, with stakeholders and focussing on their values, the outcomes produced can exceed the state-of-the art [96]. This state of value-sensitivity can only be achieved through the meaningful participation and collaboration of a diverse set of voices across a variety of disciplines, which the field of AI stands to benefit greatly from [12].

Accordingly, this paper establishes a set of criteria for evaluating socio-technical design processes enabling the creation of value-sensitive AI (VSAI). Socio-technical processes are considered to be ones aiming to create responsible, user/human-centred, or ethical AI. The dimensions of comprehensiveness, level of guidance offered, and methodological value-sensitivity are selected because of several concerns in the literature regarding how existing interventions fare with regard to these dimensions, thus the review aims to verify the legitimacy of those concerns through its assessment of the design processes. In doing so, the paper also provides a novel review of existing socio-technical design processes for AI system design. After assessing each process and synthesising findings into broader trends, resulting recommendations are provided. These recommendations are derived from the findings of the review in terms of analysing the different design processes to understand the extent with which they address each concern mentioned. The recommendations target areas where concerns were not sufficiently mitigated by existing processes.

The dimensions and recommendations provided are important to the AI community as they ensure that those following design processes or creating design processes for AI-based systems have clear and actionable best-practice-based steps to follow. They also ensure that these steps will help them actively consider and respect the values of those using the systems and being affected by them, and that the resulting outcomes will be more standardised and predictable in terms of their value sensitivity. The review of socio-technical design processes for AI-based system and their assessment also helps creators and followers of design processes to understand what already exists and how each process can be used, updated, or adapted for different needs and purposes.

Section 1 presents important background information on topics related to the review, particularly VSD and VSAI, and socio-technical design processes for AI-based systems. Each subsection also introduces the challenges and limitations of the respective space as these form the basis of the assessment criteria used to evaluate design processes (based on the extent to which they help overcome existing challenges). Section 2 outlines the methodology of the study in terms of the review process followed, the design processes reviewed, the criticisms used to derive the assessment criteria, and the criteria themselves. Section 3 provides the results of the assessment of the mentioned design processes using the derived criteria and discusses their synthesis into broader trends. The section concludes with a set of resulting recommendations for design processes enabling VSAI.

Scoping and positionality It is worth noting that the authors of this work come from predominantly design-based backgrounds and therefore adopt a designer perspective throughout the paper. This perspective is clear in the advocation for more socio-technical and design-based considerations for AI-based systems, as well as the body of works consulted throughout the paper. It is also reflected in our focus on AI designers and AI design interventions, as opposed to more technical practitioners such as engineers or data scientists. Nevertheless, the creation of AI-based systems is very much an interdisciplinary endeavour, and we recognise that our perspective is one of many involved. The goal of this work is to specifically shed light on how design processes for ethical and responsible AI address stakeholder values and incorporate them into the AI creation process. Such a focus is rooted in design-based disciplines and does not diminish the overall significance or value of reviewed design processes. Instead, the aim is to shed light on the overall lack of focus on value-related aspects within existing processes, and to encourage users of these processes to consider such a focus when adopting or adapting these processes, or even creating new ones.

The intended audience for this paper is therefore a broad audience engaged in the socio-technical design of AI-based systems, starting from a project brief, problem statement, research question, or user need and aiming at producing a functional AI-based system. This includes academic researchers and industry practitioners, who are either adopting, adapting existing, or formulating new design processes.

2 Background

2.1 Value sensitive design and value-sensitive AI

Value-sensitivity originates in the branch of design known as value-sensitive design (VSD). VSD advocates designing for values over designing for functions [13] under the mindset that the computational systems we build are a reflection of our value systems [14]. The overarching aim of VSD is to make design decisions that proactively account for and embody stakeholders’ values and preferences early on and then comprehensively throughout the process [15, 16]. It builds on the idea that these values and ideals can be embedded and embodied in the technology itself [16, 17]. In the context of VSD, values refer to what people consider important, how they wish to behave, and their desired end-states of different processes [18, 19]. They differ from norms, as norms are seen as something obligatory to follow in society and are culture-specific, whereas values are what people consider to be good and right, and can be emergent, subjective, and change according to context or the passage of time [18, 20]. They also differ from needs which tend to lead to functional thinking and overlook several consequences and value breaches [21]. To elucidate this further, an example of a value might be ‘privacy’ or ‘autonomy’, which reflects how someone might want to be treated, might want to live, or might want systems around them to behave. Meanwhile, an example of a norm might be that people should ask others how they are doing before engaging in further conversation, this reflects how people think and act in a society or culture. Finally, needs are more functional aspects that need to be fulfilled, such as the need to check a bank account balance or the need to access health information. While there is much focus on uncovering and understanding needs when creating technology, and norms are more widespread and arguably easier to surface, it is crucial to discover and understand individual stakeholder values for a project, instead of making assumptions [22].

The tripartite methodology of value-sensitive design consists of three layers of investigation: conceptual, empirical, and technical. On a conceptual level, stakeholder analysis considers how ideas might impact direct and indirect stakeholders. On the empirical level, behavioural, contextual, experiential and cognitive data are collected to identify and prioritise stakeholders’ values and opinions. Finally, on a technical level, reflections on how the technology will help or hinder stakeholders’ values and how those values can be embedded in the technology take place. The field of VSD offers several tools and techniques to capture and embed stakeholder values throughout the design process [16]. It is also particularly valuable when applied to the design of systems that have serious ethical implications, such as AI systems [23]. There are a variety of different methods, toolkits, and activities developed to elicit values from stakeholders and ground projects in value-sensitivity [49,50,51,52,53], even within the context of artificially intelligent systems [24,25,26]. Few socio-technical design processes for AI-based systems have begun explicitly focussing on value-sensitive design [23, 27] and are therefore expected to offer support for the tripartite investigations that need to take place.

The space of value-sensitive AI is an emerging one, as opposed to the also new, but more developed, space of Human-Centred AI (HCAI) [28, 29]. [30]‘s recent survey of how different branches of human-centred design research contribute to the field of AI demonstrates the novelty of Value-Sensitive AI (VSAI) as it was not included in the survey. Nevertheless, the survey also highlights the value of VSD in AI as it combines the benefits of several of the other movements mentioned in the survey. While HCAI might not necessarily focus on stakeholder values and might instead focus on aspects such as the participatory design of AI [31, 32], VSAI is a subset of HCAI that foregrounds the values and principles of the stakeholders both involved in the creation of AI-based systems and affected by their outcomes [33]. This review focuses on VSAI specifically because of VSD’s ability to promote self-reflexivity [23], potentially addressing the root causes of deeper issues in the design of AI-based systems that might lie within practitioners’ mindsets and cultures. It also calls for the engagement of direct and indirect stakeholders [23], bringing in the benefits of HCAI design in a more structured and meaningful way.

Using some of the design processes reviewed in this study, VSD has already been applied to weapons both autonomous [34, 38] and not [35], AI-based manufacturing assistance systems [36] and industry 5.0 factories [37], smart home systems [39], nanotechnology [40], energy transitions [41], for aiding collaborations and coordinating between stakeholders when creating AI-based systems [42], and care robots in the form of Care-Centred Value-Sensitive Design [43]. It is, therefore, crucial to assess design processes for VSAI given the criticality of the use-cases they are applied to. This assessment and its results can help ensure that future design processes used will support different aspects of VSD and that their outcomes will be consistent in terms of value sensitivity.

2.1.1 Limitations of value-sensitive design

A lack of value elicitation VSD can be applied to AI-based systems by using pre-defined values or principles from published works (e.g. by the United Nations or the European Union), this is referred to as a top-down VSD approach that begins with existing values and embeds those in the technology [24]. [44] criticises VSD’s use of a list of ‘universal values’ “often impacted upon by technology” [45]. They state that these values are difficult to define in different contexts and are not certainly universal. Instead, they call for “value discovery” [44, p. 1145] or value elicitation, which they define as “inquiring about the values present in a given context and responding to those values—being sensitive to those values—through design.” [p.1143], instead of "favouring known values" [44] or "frontloading" [46]. This is known as a bottom-up approach that learns from human actions and experiences and elicits underlying values from those to work with [24]. Value elicitation is preferred [44, 47] as it provides situated, context-specific values beyond simply providing lists of “abstract words” [48]. Acquiring this type of values can also help establish a form of ``cultural relativism" which has been called for in the space of AI-based systems [8] and is especially important as values prioritised by the general public were found to change in different contexts and situations [49]. Case studies that have included these stages, such as by using the Value Deliberation Process [50], have been found to help stakeholders reach value-based consensus in the case of autonomous technologies [51]. Showkat & Baumer (2022) [95] offer one example of such a case study, where values were effectively elicited from users of Natural Language Processing (NLP) technology through the use of design activities and made clear to the creators of NLP technologies to make use of.

A lack of value embodiment [52] criticize VSD’s “implicit assumption in the methodology of VSD that one will know what to do in a normative sense, once these values are known.” This criticism relates to how elicited values are used. Indeed, VSD is less descriptive with regards to embed chosen values in technology, and with how to evaluate the effect of conducted activities [53]. [54] distinguishes between the intended values of the creators, the embodied values of intermediate prototypes and artefacts, and the realised values of the final outcome. While intended values might not make it to the final outcome, and realised values can only be assessed when the system is deployed and used for a significant amount of time, embodied values are the most apt to consider [54]. Additionally, both intended and realised values can fall prey to the “wrong reasons problem” [55]. Motivation for intended values lie in practitioners’ intentions and motivation for realised values might be to prevent negative outcomes or misuses—in both cases, the reason for a positive attitude towards the values might lie in reasons outside the technology itself [54]. Embodied values assess whether prototypes or artefacts created during the design process have successfully considered and respected stakeholder values and creators’ intended values [57]. They allow for a timely assessment of the value-sensitivity of both the process followed and its intermediate outcomes [58,59,60]. Other works have also suggested that values arise from actuating prototypes [28].

2.2 Socio-technical design processes for AI-based systems

An AI-based system design process has the final goal of creating an AI-based system. This system can include a number of other elements, such as a form of user interface, a database, and so on, but must always include an AI element. Definitions of what an AI system is and what it should be able to do have varied largely across different fields and different times, but there are common factors across all the definitions [61]. The definition adopted by this research comes from the synthesis of various statements provided by scholars in the field of AI, namely: [61,62,63,64], based on the recurring elements between them:

An artificial intelligence system is one where an agent (i.e. an entity that takes action) can sense or receive external data, learn from it, interpret it based on its circumstances/environments, its goals, and its experience; and act intelligently (i.e. both appropriately and flexibly).

The technical design process for creating AI-based systems can be considered standardised with several sources describing similar or equivalent activities. These are typically: data collection, analysis, and exploration/understanding (of data sources, quality, environments, etc.); data preparation (cleaning, labelling, etc.); trade-off analysis and algorithm selection; building the pipeline; testing, validation and tuning; deployment and monitoring (operationalization) [65,66,67,68,69,70]. [31] describes this process as “technically-focussed, representationally imbalanced, and non-participatory.”

Besides the technical design process for AI-based systems that is largely followed within a variety of industry-based and scholarly settings, several other processes have been developed in recent years to go beyond simply creating ‘correctly functioning’ and high-performing AI-based systems, but also those which are more human-centred, responsible, value-sensitive, and ethical. Design processes that list the creation of these kinds of AI-based systems as their objective are the ones termed ‘socio-technical design processes for AI-based systems’ by this study. The term socio-technical refers to the “interaction of social and technical factors [to create] the conditions for successful (or unsuccessful) system performance” [71] and encompasses any system or intervention that takes both social and technical requirements and constraints into consideration. Therefore, when we refer to ‘design processes,’ we are encompassing a comprehensive approach that involves a series of socio-technical steps required for developing an AI-based system. These steps encompass both technical activities (such as model training), and non-technical activities (such as collaborative ideation with end-users). Our usage of ‘design processes’ extends beyond just the activities specific to the design or engineering disciplines, and includes the broader range of tasks involved in creating AI-based systems.

It is worth noting that whilst the field of AI-based systems has seen several frameworks [72], recommendations, and guidelines [73], as well as a smaller number of toolkits or interventions to actualize design processes,Footnote 1 there are even fewer socio-technical processes that can be interpreted and followed as a series of steps, activities, or phases to create AI-based systems. These processes are the focus of this review. Other styles of interventions often only provide a list of recommendations, values, questions, or directives, with little guidance as to what they mean exactly, why they are important, which stage they should be applied in, and how they can be adapted for on-the-ground scenarios and varying contexts. There also exists several frameworks for creating responsible and equitable technologies and outcomes, without focussing specifically on AI as a design material, such as the Design 4 Diversity Framework [74], the Technology-Driven Design Process [75, 76]’s value-driven design approach to envisioning speculative futures, and the responsible design framework [77,78,79], which lie outside the scope of consideration due to their lack of focus on AI-based systems.

2.2.1 Challenges and limitations of current socio-technical interventions for AI-based systems

Guidelines and recommendations: abstract with little guidance Guidelines and recommendations are a form of addressing values through the top-down approach discussed earlier, which does not account for stakeholders’ context-specific values for AI-based systems. Another major concern is that existing guidelines and recommendations are too abstract and are therefore difficult to practically implement in real-world settings [65, 80, 81]—this is termed as “the ambiguity challenge” by [82]. These styles of interventions are thought to be too vague or general to guide concrete actions [21, 65, 83], cannot be meaningfully measured [83], and are not found to be as useful as more practical resources by AI practitioners [84]. Furthermore, these guidelines fall short in terms of resolving conflicts between non-hierarchal values (i.e. “the ranking challenge” [82]) and failing to include values which might be important to specific contexts or domains (i.e. “the tacit exclusion challenge”) (ibid.).

Design activities: isolated with limited coverage Looking towards various design fields and approaches is one technique that is proving to be effective in tackling some of these limitations within the space of AI-based systems [85]. Several design techniques have been employed within the field already, such as the use of iconography [86], design fictions [24, 86], visualisations [9, 87], reflective checklists [89], thought experiments [10], decks of cards [25], usability heuristics [90], and by creating design documentation [88] for AI models. While certainly human-centred, these efforts tend to be composed of isolated co-design activities or custom-made workshops conducted by specific groups of people for limited outcomes. Instead, standardised design processes that guide the use of these approaches and methods systematically throughout the entire design process of AI systems, coupled with multi-faceted evaluations and stakeholder involvement during the different phases involved, is needed [11]. Another problematic aspect of design activities is that they do not offer coverage of the entire life cycle of an AI-based systems. Instead, they are limited to specific objectives or outcomes within specific phases. These activities are typically conducted during design and evaluation stages, with very little coverage of ideation, prototyping and implementation stages as found by a review of ethics-focussed design methods and a review of culturally informed design practices for conversational agents [91]. Recent studies have also highlighted that “more support [is] needed for early phase ideation and problem formulation” when it comes to developing human-centred AI [92].

Design Processes: Imperfect Middle Ground Design processes offer a middle ground between theoretical and practical approaches, offering both standardisation and implementability. [93] recommends that such design processes need to be bottom-up, procedural, participatory, and deliberative to overcome the limitations of guidelines and recommendations mentioned earlier [94]. Similar to design activities, socio-technical design processes have also been criticised for their incomplete design phase coverage [11, 65, 74]. Similarly to other design interventions in the space of ethical AI and human-centred AI, they also tend to overly focus on initial planning and ideation phases and later testing and evaluation phases and overlook mid-process phases such as design and implementation [92]. This creates a gap that makes it unclear how to carry over the values and insights gained during initial phases into mid-process phases, shielding those following the process from unexpected outcomes during evaluation, and ensuring that those values are adequately embodied and respected consistently until and after deployment.

3 Methodology

Firstly, socio-technical processes for designing AI were identified and selected, then assessment criteria were established, and the selected publications were assessed. An overview of the study is presented in Fig. 1.

Fig. 1
figure 1

Study flow diagram

Full size image

3.1 Selecting studies

Publications and practical tools related to designing AI were reviewed in academic databases and search engines. The search included materials available in English in the following databases without any date or geographic restrictions:

  • Science Direct

  • Institute of Electrical and Electronics Engineers (IEEE) Xplore

  • The Digital Library of the Association for Computing Machinery (ACM)

We also used Google search engine to identify practical tools developed by industrial stakeholders which would not necessarily be represented in academic publications. We searched for materials using keywords relating to both ‘design processes’ (see Keywords 1 in Table 1) and to ‘AI’ (see Keywords 2 in Table 1).

Table 1 Keywords used for the identification of studies
Full size table

Feedback on the selection process and findings was also gathered from two experts in the field of ‘Responsible AI’ (UK-based academics, external to the authors team). The results of the review and selection process are shown below in Table 2.

Table 2 A description of the design processes included in the review
Full size table

Out of 593 identified papers which were retrieved using the keyword combinations shown in Table 1, only six were found to introduce design processes based on the definition used by this paper (in terms of providing concrete and actionable steps, activities, or phases to follow). One additional design process was obtained using the Google search engine.

3.2 Establishing assessment criteria

Given the limitations, challenges, and concerns in the literature regarding socio-technical AI design processes, VSD, and VSAI interventions, the following dimensions were derived to evaluate the extent to which different design processes address these issues:

  1. a.

    Comprehensiveness

  • Assessment method the comprehensiveness of value-sensitivity support is noted based on whether the mentioned activities span all major phases of a design process (i.e. ideation, design, implementation, evaluation, monitoring & deployment).

  • Ideal outcome a design process should outline activities and steps that should be taken during all the major phases of design in detail. This is to ensure values and needs elicited during earlier phases are effectively carried forward and embodied in the final outcomes of the process.

  1. b.

    Level of guidance offered

  • Assessment method this is assessed by whether the design process offers detailed instructions on how to practically carry out the steps and activities listed, or whether they offer a corresponding toolkit to help practically conduct activities.

  • Ideal outcome the design process should describe how steps can be carried out in practice or include tools to help achieve this, in order to effectively operationalize the theoretical recommendations and instructions included.

  1. c.

    Methodological value sensitivity

  2. i.

    Supporting tripartite investigations of VSD

  • Assessment method this is assessed through a process’ inclusion of activities that enable the three investigations of VSD to take place. The conceptual level is assessed based on whether the design process calls for any stakeholder identification activity. The empirical level is evaluated based on the mention of any stakeholder engagement activities, such as user studies, co-design sessions, value elicitation activities, and so on. Finally, the technical level is assessed based on any reflection activities with regards to stakeholder needs, existing technological capabilities, or how both can be combined.

  • Ideal outcome a design process should include activities that enable the three investigations (conceptual, empirical, technical) in any order.

  1. ii.

    Value elicitation

  • Assessment method this is assessed based on whether the process calls for using pre-defined or elicited values which coincide with a using top-down or bottom-up VSD approach, respectively [16], or both.

  • Ideal outcome using elicited values or a combination of both approaches is favoured over only using pre-defined values.

  1. iii.

    Value embodiment

  • Assessment method this is assessed based on whether (i) the design process includes prototyping activities to create intermediary outcomes and (ii) whether those outcomes are shown to/used by/evaluated by any domain experts, stakeholders outside the design team, or potential end-users, to be able to assess their value embodiment.

  • Ideal outcome the design process should account for creating an intermediary artefact, outcome, or prototype which is given to relevant parties to explicitly evaluate value embodiment before the end of the design process.

4 Findings

Table 2 introduces and summarises the design processes reviewed, whilst Table 3 highlights the findings of the critical review with relation to the dimensions mentioned earlier.

Table 3 A summary of how the reviewed design processes compare in terms of the dimensions discussed throughout the paper
Full size table

Lack of specified target technologies and empirical use-cases The reviewed design processes lack specificity regarding the types of AI-based systems they are intended for, such as conversational AI or autonomous vehicles. It is unclear what type of AI-based systems these design processes can facilitate the creation of, and this information was not provided by the creators of these processes. Additionally, they lack empirical validation, relying primarily on conceptual case studies and envisioned benefits, and therefore, including no verification of real-world effectiveness. These limitations emphasise the need for future work to provide clearer targeting and empirical evidence when creating or adapting design processes for AI-based systems.

5 Results

5.1 IEEE 7000

5.1.1 Process description

The IEEE 7000 employs a “value-based engineering” or “value by design” approach where it moves away from checklist-style regulations [97] and away from static, overly generic value lists [98] and more towards a methodological approach. It not only focuses on the risks of neglecting values, but on modifying the non-functional requirements engineering process to lead to more value-sensitive and human-centred outcomes [97]. The design process’ steps are outlined in Fig. 2.

Fig. 2
figure 2

Visual summary of the steps of the IEEE 7000 process

Full size image

The first phase is the Concept of Operations (ConOps) and Context Exploration Process. The ConOps phase is synonymous to outlining the goals and desired outcomes of the project and identifying relevant multi-disciplinary stakeholders. The Context Exploration Process aims to have the team understand the contexts in which the system will operate, viewing the system as a socio-technical entity that will have varying effects on different parts of a wider ecosystem. The next step is the Ethical Values Elicitation and Prioritisation Process. The aim of this phase is to elicit and rank core values from stakeholders. Such a process should also focus on values for each stage of AI creation, such as data collection, data preparation, modelling, deployment, and evaluation and should be transparently communicated between all stakeholders. The outcome of this phase is a ‘Value Registry’ with the chosen core values, their demonstrators, and any conflicts. The following stage is the Ethical Requirements Definition Process. The aim of this phase is to translate the core values into system requirements and risk treatment plans that show how the values are reflected and protected within the system. This ensures technical and social feasibility and provides a traceable trail between the core values and subsequent design and development decisions. These requirements should be validated with stakeholders. The final stage is for an Ethical, Risk-Based Design Process to take place. There are no constraints as to how this design process could look like, only that it follows the guidance of the Value Registry and involves the identified stakeholders consistently.

5.1.2 Process analysis

Comprehensive value sensitivity with a lack of explicit mapping The process ensures value sensitivity across stakeholder identification, value elicitation, and assessment of value impact, reflecting socio-technical contexts and stakeholder concerns. ConOps and Context Exploration Process explicitly calls for identifying multi-disciplinary stakeholders in terms of both intended and potentially unintended users. The Ethical Values Elicitation and Prioritisation Process describes how to elicit and rank stakeholder values and the Ethical Requirements Definition Process operationalizes those values into more usable system requirements. The ConOps and Context Exploration Process also calls for reflecting on the socio-technical environment the system will be deployed and used in and the potential benefits and harms that the system might introduce to it. Nevertheless, the absence of explicit mapping between the process’ activities and the VSD investigations and steps hinders a clear understanding whether coverage of those investigations was intentional.

Incorporation of diverse values The process combines pre-defined and elicited values, embracing both top-down and bottom-up approaches, emphasising the importance of elicitation from stakeholders. It provides a list of values to consider and recommends supplementing those with more context-specific values from stakeholders.

Practical guidance offered The process offers detailed instructions, examples, and diagrams, aiding understanding and implementation.

Partial focus The process only focuses on the initial phases of the AI-based system design process and does not go into details regarding the rest of the phases, which also require value-sensitivity across their activities. The lack of descriptiveness for what an ethical, risk-based design process lowers the overall comprehensiveness of the process.

Missing prototyping emphasis While emphasising ethical, risk-based design, there is no mention of prototyping or assessing value embodiment midway, potentially impacting the iterative development process.

5.2 Z-Inspection

5.2.1 Process description

The Z-Inspection process only targets the initial phases of AI systems design. Instead of having one initial reflection, contextualisation, or requirements elicitation phase as other processes do, the Z-Inspection process breaks this phase down into several sub-phases and activities. Its steps are shown in Fig. 3. The first phase is set-up, where a trans-disciplinary team is formed including AI designers and domain experts, a list of initial questions are tackled, and the boundaries between the different contexts that the system will exist within (such as where it is designed, where it is created, where it is deployed, where the data is stored, etc.) are defined. Afterwards comes the assessment phase, where four main activities take place. Firstly, socio-technical scenarios are collaboratively designed, leveraging domain experts’ expertise, to map out different actors, interactions, technologies, and processes that can affect or be affected by the system. Secondly, ethical issues and tensions are identified based on multiple stakeholder perspectives. Thirdly, the identified issues are mapped out using a list of values provided by the EU for creating trustworthy AI,Footnote 2 to ensure the issues identified are comprehensive, and link them to a common resource that can be used to establish shared definitions and solutions. Finally, a strategy is created to address issues and resolve tensions. The final phase is resolution, where the strategy is executed and feedback and recommendations are given to relevant stakeholders, along with the trade-offs associated with different decisions and the honouring of different values to varying extents.

Fig. 3
figure 3

Visual summary of the steps of the Z-inspection process

Full size image

5.2.2 Process analysis

Comprehensive value-sensitivity with a lack of explicit mapping The Z-inspection process achieves value sensitivity across the three VSD investigations. It guides practitioners towards identifying stakeholders, understanding relevant ethical issues and tensions, and reflecting on how those can be addressed or avoided. Similarly to the IEEE 7000, there is no explicit mapping to VSD investigations, making it unclear whether this coverage was intentional.

Incorporation of diverse values The Z-Inspection process also works with both elicited and pre-defined values. As such, it uses both a top-down and bottom-up approach, albeit with less emphasis on value elicitation than the IEEE 7000.

Partial focus Similarly, to the IEEE 7000, the process is heavily focussed on the initial phases of the design process and does not go into any detail regarding the following phases and how value-sensitivity can be carried over throughout them, thus lowering its comprehensiveness.

Missing prototype emphasis In terms of value embodiment, there is no mention of prototyping or evaluation activities as the process focuses on earlier phases.

Lack of practical guidance It also does not provide detailed guidance or a supporting toolkit and is less descriptive of how activities should be carried out.

5.3 Value-sensitive algorithm design (VSAD) process

5.3.1 Process description

This design process focuses its scope on all the steps of AI creation, framing its various phases in a more participatory light. Stepping away from processes focussed solely on the machine learning or big data aspects, the process begins with identifying stakeholders and understanding their goals and values. Following, suitable algorithmic approaches are chosen based on the information learned from stakeholders, and prototypes are created for each approach. Following, the prototypes are deployed into the community, and feedback is collected based on carefully planned methods for gaining participants’ insights. The algorithms are refined iteratively in several feedback loops, a practice which is not widely used in the field of AI, and finally, the chosen algorithm is evaluated based not only on performance and accuracy, but also based on acceptance with regards to stakeholder values, and the impacts it has on affected communities. The process’ steps are shown below in Fig. 4.

Fig. 4
figure 4

Visual summary of the steps of the VSAD process

Full size image

5.3.2 Process analysis

Comprehensive value-sensitivity with explicit mapping The VSAD achieves the three investigations of value sensitivity, with a detailed focus on the technical investigation. It also explicitly maps between investigations and the process’ activities. It begins with identifying relevant stakeholders, then recommends understanding stakeholders’ goals and values. Finally, the process calls for choosing different suitable algorithms based on the information learnt about stakeholders and prototyping those to understand their effects on the stakeholders and their goals and values.

Comprehensive coverage The process covers all major design phases in the steps it describes.

Inclusion of stakeholder values The VSAD recommends working with elicited values using a bottom-up approach. It does not mention using pre-defined values or a top-down approach.

Prototype emphasis The process strongly supports value embodiment in that it includes a dedicated and iterative prototyping stage where intermediary prototypes can be created and are evaluated with relevant parties.

Varying levels of clarity Differently to other processes, the VSAD process goes into more detail about how to achieve the technical investigation by selecting suitable algorithms, prototyping them, understanding their effect, and iteratively refining them until the final goal is achieved. It covers the conceptual and empirical aspects in less detail, simply advising that stakeholders be identified, and their values and goals understood.

5.4 Requirements specification for machine-learned components (RSMLC) process

5.4.1 Process description

Another framework inspired by requirements engineering processes is the RSMLC. Consisting of four steps, the framework promotes contextualization and transparency as follows: the first step is “benchmarking the domain” which consists of understanding the components, features, actors, and values present in the domain within which the AI-based system will operate to ensure all stakeholders are in consensus before moving on to defining requirements. Secondly, they recommend that the dataset which will be used to capture the domain be visualised, to understand how accurately it represents the real-world context of the domain. Following, the domain is learned by several different AI models, prioritising explainability and legibility, and comparing the trade-offs of different models. Finally, they recommend highlighting the gaps present due to domain concepts that were unprioritised or undiscovered, dataset limitations, ambiguous requirements or values, etc. The steps of the RSMLC process are outlined in Fig. 5.

Fig. 5
figure 5

Visual summary of the steps of the RSMLC process

Full size image

5.4.2 Process analysis

Comprehensive value-sensitivity with a lack of explicit mapping Similarly to the IEEE 7000 and Z-Inspection, the RSMLC process covers the three VSD phases but does not provide an explicit mapping. The ‘benchmarking the domain’ phase mentions identifying values present in the AI-based system’s domain. The process also recommends visualising the chosen dataset to make sure it captures stakeholders’ values and context accurately and then training different ML models to identify which ones respect relevant values the most. It also includes highlighting gaps in the work, which calls for a reflection of potentially unaddressed values or similar elements.

Inclusion of stakeholder values Similarly to the VSAD, the RSMLC process recommends working with elicited values using a bottom-up approach. It does not mention using pre-defined values or a top-down approach.

Varying levels of clarity Also similarly to the VSAD, the RSMLC covers the conceptual and empirical investigations in little detail, with almost no guidance on how to identify stakeholders or elicit and understand their values. It covers the technical investigation in much more detail, outlining different activities that can be conducted to achieve it across different phases.

Partial focus The RSMLC has less comprehensiveness than the VSAD, focussing on earlier phases of the design process only similarly to the IEEE 7000 and Z-Inspection.

Missing prototype emphasis In terms of achieving or assessing value embodiment, there is no specific mention of prototyping and any value-based evaluation.

5.5 Value-sensitive design for social good (VSD2AI) process

5.5.1 Process description

This design process aims to restructure and redefine the tripartite investigations of VSD in order to better suit the nature of AI-based systems. The process begins with Context Analysis, which aims to identify stakeholders and understand the context of the system in terms of norms and values. Value identification follows, where values are obtained from various sources providing AI-related values, as well as the United Nation’s Sustainable Development Goals (SDGs). The values obtained from secondary sources may also be supplemented with more context-specific values through the adaptation of identified stakeholders’ values. The next step is Formulating Design Requirements which aims to translate identified values into design requirements. Finally, Prototyping should take place to create prototypes adhering to the created design requirements. This stage should last throughout the entire life cycle as new constraints, undesirable consequences, and re-designs emerge or take place. The figure below (Fig. 6) shows the steps of this process.

Fig. 6
figure 6

Visual summary of the steps of the VSD2AI process

Full size image

5.5.2 Process analysis

Comprehensive value-sensitivity with explicit mapping Being the main purpose of the process, the VSD2AI process covers the three value-sensitive investigations comprehensively and explicitly, and adapts VSD activities to the context of AI. The process also explicitly outlines how each activity maps onto a VSD investigation in their own work, helping to show the importance of conducting each activity and how it might look in a real project using a case study. The context analysis step includes stakeholder identification, as well as an understanding of their context. The value identification and prototyping steps also include elements of conceptual investigations. The value identification step directly calls for value elicitation. The design requirements step also includes working with those values to translate them into requirements. Both the design requirements and prototyping phases look at reflecting on how technology can address the identified values and the resulting design requirements.

Incorporation of diverse values In terms of value elicitation, the VSD2AI uses a hybrid top-down and bottom-up approach. It starts with pre-defined values from responsible AI recommendations and then supplements those with more context-specific values from stakeholders, similarly to the IEEE 7000 and Z-Inspection.

Prototype emphasis Regarding value embodiment, there is a dedicated prototyping phase, and the prototype is expected to be viewed by relevant parties for value-based evaluations.

Varying levels of clarity It provides some guidance in terms of pointing to methods and frameworks for conducting some of the mentioned activities but is not too extensive regarding the instructions it provides for others.

Partial focus While it mentions that prototyping should continue throughout the implementation phase, the details of this are not clear, and it does not mention evaluation, and deployment and monitoring phases.

5.6 Microsoft’s HAX process

5.6.1 Process description

The process presented in Microsoft’s HAX Workbook is meant to guide teams who are in the early stages of requirements discovery for an AI-based system or who are evolving an existing system. The process focuses on the initial requirements planning phase and breaks that down, similar to the IEEE 7000 and RSMLC processes. The process is centred around Microsoft’s principles for responsible AI and consists of five steps. Teams are meant to select a principle, brainstorm how that principle can benefit or harm users in the context of their project and to what extent, describe requirements needed within various aspects (AI, data, user interface, etc.) to enact high-impact principles and estimate the resources they will need, prioritise specific requirements or features, and then assess their progress throughout the development process. The steps of this process are described below (Fig. 7).

Fig. 7
figure 7

Visual summary of the steps of Microsoft’s HAX process

Full size image

5.6.2 Process analysis

Lack of comprehensive value-sensitivity The HAX process does not touch on the three investigations of value-sensitivity per-se. It does cover the technical investigation by exploring how the existing principles affect users, although it is not clear how those users will be identified. The process does not explicitly recommend or describe identifying users or stakeholders. The process also includes brainstorming how different principles can harm or benefit different stakeholders and to what extent.

Clarity and illustration The process comes with an accompanying workbook to use when carrying out the activities. This makes following the process more concrete and practical.

Use of static values This is because the process works with pre-identified values originating from Microsoft instead of identifying stakeholders and eliciting their values, therefore it only uses a top-down approach.

Partial focus and lack of prototype emphasis The process does not cover the full design process, instead earlier design phases are targeted. As a result, there is no consideration of value embodiment as prototyping and evaluation phases are not included.

5.7 PwC’s responsible AI process

5.7.1 Process description

The responsible AI process by PwC is an incredibly detailed process that aims to provide accountability and governance to the AI design process by focussing on risks, responsibilities, and evaluation. The steps in their process are outlined below:

  1. 1.

    Strategy (Corporate strategy, industry standards and regulations, internal policies and practices)

  2. 2.

    Planning (Portfolio management, delivery approach, internal policies and practices)

  3. 3.

    Strategy (Corporate strategy, industry standards and regulations, program oversight)

  4. 4.

    Ecosystem (Technology roadmap, sourcing and vendor assessment, change management)

  5. 5.

    Development (Context-business-data understanding, solution design, data extraction, pre-processing, AI-building)

  6. 6.

    Deployment (Impact and integration, transition and execution, performance monitoring, evaluation and check-in)

  7. 7.

    Monitor and Report (Continuous monitoring across life cycle, compliance reporting)

It is worth noting that whilst the first three steps are sequential, the following steps are presented as cyclic and iterative. Development and deployment form a large application cycle, with monitoring and reporting occurring concurrently throughout. This contrasts with the more linear nature of most other processes presented. These steps are shown linearly in Fig. 8 for comparability with other reviewed processes.

Fig. 8
figure 8

Visual summary of the steps of PwC’s responsible AI process

Full size image

5.7.2 Process analysis

Lack of comprehensive value-sensitivity PwC’s process does little to address the conceptual investigation. The process does not explicitly recommend or describe identifying users or stakeholders. With the empirical and technical investigations, they could be included within some of the activities described, however, there is no explicit mention or emphasis on those investigations and potential activities to cover them, thus it is unclear whether they are effectively covered by the process or not and will depend on a case-by-case basis. The development phase includes context, business, and data understanding, which could include understanding the needs and values of stakeholders within the AI-based systems’ context, although this is not explicitly mentioned. The ecosystem phase includes creating a technology roadmap and the development phase’s solution design activity could include a reflection on how the technology is going to address identified needs, values, and issues. This is again not explicitly mentioned, however.

Use of static values This is because the process works with pre-identified values instead of identifying stakeholders and eliciting their values, therefore it only uses a top-down approach.

Lack of prototype emphasis There is no consideration of value-based evaluation when prototyping in order to achieve or assess value embodiment.

Lack of practical guidance The process does not provide any practical tools or resources and does not go into clear details on how to carry out the different phases.

Comprehensive coverage The process covers all major design phases in the steps it describes.

6 Discussion and recommendations

6.1 Discussion

We discuss here the similarities between the selected design processes and identify broader trends.

6.1.1 Supporting tripartite investigations and design phase coverage

As mentioned previously, value-sensitive design calls for three layers of investigation: conceptual investigations into who the relevant stakeholders are and their broader contexts, empirical investigations to collect stakeholders’ needs and values, and technical investigations to reflect on how the target technology can enable or hinder those values. This section discusses the extent with which different processes address these tripartite investigations.

Early-stage processes The IEEE 7000 and Z-Inspection include value-sensitive investigations but need more comprehensiveness in terms of covering the entire process in more detail to ensure this value-sensitivity is maintained throughout and is embodied in the final outcomes.

Business-oriented processes Microsoft and PwC’s processes both revolve around the use of pre-defined values. This is perhaps due to their business-oriented focuses and in the interest of saving time and resources. In the context of Microsoft’s HAX process, this causes the process to skip the conceptual and empirical levels of value-sensitivity in favour of other benefits. The technical level also suffers as it is unclear how to identify users in order to brainstorm the effects of the chosen values on them. The process also fails to cover design phases where it may be more difficult or impractical to include stakeholders and end-users: design, implementation, deployment & monitoring. PwC’s focuses more on broader organizational and infrastructural activities that need to take place. As a result, they cover all design phases in detail. Nevertheless, PwC’s process is vague regarding whether the empirical and technical levels of VSD are covered. While they are not explicitly covered by any of the phases, they could be included in some of the activities described, but this will depend on the party following the process. They also do not cover the conceptual level.

Value-sensitive processes Processes like the VSAD process and the VSD2AI process show that it is possible to modify traditional AI-based system design processes and augment them with mechanisms to capture and embed stakeholder values. The VSAD also shows a great example of providing more detailed descriptions of how to achieve those levels through its description of the technical level but lacks to replicate this attention to detail for the conceptual and empirical levels. The RSMLC follows a similar pattern, focussing on the technical level and neglecting the other two, but is not as comprehensive as VSAD, following a similar pattern to Microsoft’s HAX where some design phases are neglected. The VSD2AI process explicitly outlines the mapping between its activities and VSD investigations, helping to highlight why and how each activity is important and brings value to the process.

6.1.2 Value elicitation and embodiment

As discussed earlier, value elicitation refers to the bottom-up surfacing and understanding of stakeholders’ contextual values within a given project, as opposed to using top-down ‘universal’ lists of pre-defined values, or at the very least, a combination of both. Value embodiment refers to the verification that the elicited values do indeed ‘show up’ in the technology as intended, or are respected and enabled by it. This requires the creation of intermediate artefacts such as prototypes and a consideration of whether the required values have been successfully embedded, ideally conducted with stakeholders.

In terms of how the design processes address value elicitation, there was a mix of top-down and bottom-up approaches, with some design processes using a combination of both. Interestingly, both processes that work with pre-defined values using a top-down approach both originate from industry settings, perhaps to reduce the resources used by the design process in terms of time and other costs. Two of the ‘value-sensitive processes’ work with elicited values using a bottom-up approach, which is the approach advocated for to increase value-sensitivity [55, 56]. The two more general, engineering-based processes use a combination of both as different contexts require (except for VSD2AI which also uses a combination of both).

Regarding value embodiment, only two processes, namely the VSD2AI and the VSAD processes, included prototyping activities. In order to meaningfully assess the outcomes of embedding value-sensitivity in design processes, it is crucial to include more prototyping phases where intermediary artefacts are created, and their value embodiment can be assessed. It comes as no surprise that the two design processes that support these activities are the ones focussed on value sensitivity; however, such a practice needs to be more widespread.

6.1.3 Offering practical guidance

Only two of the surveyed processes offer a sufficient level of guidance: the IEEE-7000 and Microsoft’s HAX Workbook process. The IEEE-7000 provides detailed instructions on how to practically carry out the steps they detail, making it reasonably easy for a practitioner to follow their design process. The HAX Workbook offers an accompanying workbook to conduct the corresponding activity for each step. The workbook, combined with the pre-defined list of values, makes it straightforward to actualize their process. The VSD2AI process provides some lists of values and references a framework which can be used to convert values to design requirements, offering some level of practical guidance, but can still be ambiguous when describing how other activities should be carried out.

6.2 Recommendations for designing value-sensitive AI

Based on this review and the derivation of criteria for design processes enabling VSAI, we end this study with several recommendations for how AI-based systems design processes can address value-sensitivity:

  • Comprehensiveness Focussing on providing steps and activities that more comprehensively cover the various phases of the design process, instead of just focussing on early planning and ideation phases and late evaluation phases. This can include involving stakeholders in centring design decisions around their values or value-based prototyping activities or making sure intermediary artefacts and outcomes embody stakeholder values. This consistent inclusion of stakeholder values ensures practitioners have a more active and sustained consideration of values and therefore can ensure a clearer reflection of those values in the outcomes produced.

  • Offering practical guidance Providing clearer and more practical guidance on how to carry out the steps and activities mentioned. This helps ensure that they are implemented in standardised and effective ways and not left to the interpretation and discretion of different practitioners. Such guidance can come in the form of instructions or accompanying toolkits.

  • Supporting tripartite VSD investigations Explicitly mapping between activities and VSD investigations and emphasising the importance and value of activities supporting VSD in order to move away from mindless compliance and towards active reflection and empathy-building. While this also helps ensure standardisation, it also works towards fostering a more human-centred culture amongst AI practitioners.

  • Supporting value elicitation Alongside using pre-defined values as a starting point, accounting for value elicitation by including activities that help identify stakeholders, make values explicit, allow for the ranking and discussing of values, and so on.

  • Assessing value embodiment Including activities that allow for the creation of intermediary artefacts or prototypes and the evaluation of their value embodiment during the design process, instead of relying on potentially unrealized intended values or delayed, after-the-fact realised values.

These recommendations are summarised in Fig. 9 along with some actionable points on how to adhere to each recommendation. These recommendations are tailored for a broad audience involved the conceptual and technical design of AI-based systems. This includes academic researchers and industry practitioners, who are either adopting, adapting existing, or formulating new design processes. By following these recommendations, they can ensure a targeted and systematic integration of stakeholder values into their processes, thereby fostering the development of AI-based systems that are not only technically robust but also ethically sound and socially responsible.

Fig. 9
figure 9

A summary of the recommendations provided for design processes enabling VSAI along with actionable points on how to implement each recommendation

Full size image

7 Conclusion

This work reviewed several existing design processes for socio-technical AI-based systems with objective of creating ethical, human/user-centred or responsible AI. Design processes were assessed based on several dimensions introduced to assess the extent to which design processes enable VSAI. These criteria were based on their ability to enable value-sensitivity and in response to criticisms of current interventions. As such, the design processes were assessed based on (a) whether they were comprehensive in spanning across the major design phases; (b) the level of guidance they offered with regards to how to carry out value-sensitive activities; and (c) their methodological value-sensitivity in terms of (c.1) supporting the tripartite investigations of VSD, (c.2) encouraging hybrid value elicitation and (c.3) allowing for the assessment of value embodiment.

Overall, socio-technical AI design processes need to more explicitly account for the different dimensions discussed. Deliberate comprehensiveness is needed to ensure elicited information is passed down through each design phase and can be clearly tracked and identified in final outcomes. Practical guidance in the form of instructions, case studies, or toolkits on how to conduct these activities most effectively is needed to avoid following in the footsteps of overly abstract guidelines. While many processes already include activities that account for the tripartite VSD methodology, there needs to be a clearer mapping between both and an intentional emphasis on the value of these activities and why they should be conducted. Such a mapping can help practitioners understand the benefits of VSD and can help change their mindsets and attitudes. By helping practitioners understand the importance of embedding value sensitivity in their processes and that this does not hinder their workflows, rather it augments them with extra benefits, AI-based systems can show a clearer correlation between their resulting value-sensitivity and the activities that helped them get there. Value elicitation should ideally use elicited values, or a combination of pre-defined and contextually elicited values, and value embodiment should take place through the creation and evaluation of intermediate outcomes to ensure values have been effectively proliferated. Based on the work done throughout this paper, some resulting recommendations were given to aid with adapting existing processes or creating new ones.

Given these findings and recommendations, and as new socio-technical design processes for human-centred AI continue to emerge, new investigations will need to take place. Future work will aim to consider the implications of these recommendations on design processes which choose to follow them, as well as assessing the resulting value-sensitivity of the processes and the value embodiment of their outcomes. These recommendations form part of a still-developing movement towards increasing value-sensitivity in AI-based systems. Their aim is to act as a source of inspiration for embarking on a long journey, rather than a short-cut to a final destination. As [88] writes in their book Value-Sensitive Design: Shaping Technology with Moral Imagination, “we need not require perfection, but commitment to practice – and through practice, progress” [p.180].