Introduction

Within the last decades, we have witnessed an increase in design approaches that seek to anticipate ethical challenges related to the design of computer technologies. Primarily, attention has been directed toward VSD, a tripartite and iterative methodology for proactively integrating values in the design of computer technologies in various contexts. Here, values are broadly understood as what is important to people in their lives, with a focus on ethics and morality (see, e.g., Friedman et al., 2017; Friedman et al., 2006). Furthermore, VSD holds an interactional position for “how values become implicated in technological designs”, implying that the shaping of technology and social systems interact (Friedman & Kahn Jr, 2002).

VSD is defined as “a theoretically grounded approach to the design of technology that accounts for human values in a principled and systematic manner throughout the design process” (Friedman et al., 2017, p. 63).Footnote 1 Recently, Hendry et al. (2021) emphasized VSD as a formative framework, noting that:

While value sensitive design can be considered to be many things—a perspective for exploring the human-technology relationship, a theory of design, a methodology, a set of design tools, a community of practice, a field of research—it is surely a design framework. When employing value sensitive design in projects, designers bring together conceptual, empirical, and technical investigations. (Hendry et al., 2021, p. 39).

VSD outlines three interdependent types of investigations. The conceptual phase is often the preferred starting point for initial value elicitation through philosophical and conceptual investigations and clarification of which direct and indirect stakeholders to involve. In the empirical investigations, the value perspectives of both direct and indirect stakeholders are included by applying design methods from social science and design studies (Friedman et al., 2017), including the use of creative design tools and methods, such as, e.g., the renowned VSD envisioning cards (Friedman & Hendry, 2012), and the Value Dams and Flows method to address value tensions in design (Miller et al., 2007). Several VSD articles favor the discussion of conceptual (see, e.g., Hayes et al., 2020; Umbrello, 2020) or empirical investigations (see, e.g., Aizenberg & van den Hoven, 2020; Yoo, 2018) deliberately leaving out the technical investigations.

Informed by insights from the conceptual and empirical investigations, technical investigations focus on the technology itself and proactively design for values or reactively redesign existing technologies. In accounting for the technical investigations, the authors emphasize that:

At times, technical investigations (…) may seem similar to empirical investigations insofar as both involve technological and empirical activity. However, they differ markedly in their unit of analysis. Technical investigations focus on the technology itself. Empirical investigations focus on the individuals, groups, or larger social systems that configure, use, or are otherwise affected by the technology. (Friedman et al., 2006, p. 352) (Friedman et al., 2008b, p. 73).

Compared to Computer Ethics, Social Informatics, CSCW, and Participatory Design, VSD is a unique methodological framework (Friedman & Kahn Jr, 2002). As such, van den Hoven (2007) refers to a “design turn” in ethics and sees VSD as dealing with ethics “in a new and fresh way: by ‘front-loading’ ethics” (van den Hoven, 2007, p. 70). Later, he notes that “ethics can be the source of technological development rather than just a constraint and technological progress can create moral progress rather than just moral problems” (Van den Hoven et al., 2012, p. 154).

The VSD approach has been refined, and particularly efforts have been made to address the critique that VSD is a normative framework lacking a normative foundation. Also, the assumption that values are universal, although they may be instantiated differently in particular cultures (Friedman & Kahn Jr, 2002), has been debated (see, e.g., Manders-Huits, 2011). The same goes for a suggested list of 12 “human values with ethical import” (Friedman & Kahn Jr, 2002, pp. 347, 1187; Friedman et al., 2006) often involved when designing computer technologies. Moreover, attention to the empirical investigations has resulted in valuable insights and new design methods (see, e.g., Friedman & Hendry, 2012; Miller et al., 2007). Issues tied to conceptual or empirical investigations have been dominant in discussions of VSD’s strengths and weaknesses. In addition, Davis and Nathan concluded in 2015 that the field has not yet matured—“development of methods and theories to support that work [design for human values] is still at a nascent stage” (Davis & Nathan, 2015).

More recently, a literature review by Winkler and Spiekermann (2018) identifies 17 studies using the complete tripartite methodology of VSD. They conclude that these studies generally suffer from methodological and reporting issues. The technical investigations in the 17 studies “relied mostly on the background knowledge of the authors, but in some cases additionally considered input from stakeholders” (Winkler & Spiekermann, 2018, p. 19). Therefore, the authors strongly encourage the use of structured and reproducible methods. Also, they find it disturbing that most projects do not engage at the technical level:

Keeping in mind that conceptual, empirical and technical investigations are interdependent, and that iteration is the bond between them, it is alarming that only four projects reported iterations that promise enhanced designs (Winkler & Spiekermann, 2018, p. 19).

Against this backdrop, this systematic review documents how technical investigations have been adapted in VSD studies from 1996 to 2023. The review distinguishes between “theoretical studies” and “applied studies” that either discuss or carry out technical investigations in VSD. Thereby, this systematic review can contribute to further developing the methodological framework for carrying out technical investigations in VSD investigations.

Methods

This systematic review was planned and is reported according to existing guidelines for the conduct and dissemination of systematic reviews (Page et al., 2021; Shea et al., 2007).

We wanted to include studies that focus on the technical phase in VSD, i.e., studies that explicitly refer to “technical investigations”, but also studies that adapt technical investigations in VSD without explicitly referring to them as such. We, therefore, developed a search strategy that allows for some variations in the description of these methodological developments. We included all study and publication types and limited to publications in English. In our search for relevant studies, we searched the following databases without any date or geographic restrictions:

  • Scopus (searched 03 January 2022)

  • The online libraries of the Association for Computing Machinery (ACM) (searched 03 January 2022)

  • The Institute of Electrical and Electronics Engineers (IEEE) (searched 03 January 2022)

  • Science Direct (searched 01 January 2022)

  • Springer (searched 01 January 2022)

The searches were updated in January 2023 (searches were performed on January 19th). Furthermore, all papers listed at vsdesign.org and the seventeen included studies in the review by Winkler and Spiekermann (2018) were screened. Finally, citations to (Friedman, 1996) were identified in Scopus. All search histories are available in the appendix (Gerdes & Frandsen, 2022).

After removing duplicates, all retrieved references were added to a reference management database using Covidence, which is developed for screening of literature for systematic reviews (Cleo et al., 2019). Two reviewers independently assessed each publication to assess inclusion eligibility. Studies were included using the following two-step process: First, all publications were screened on the title/abstract level. Second, full texts of the potentially eligible publications were retrieved, and two reviewers independently assessed each full text. Any disagreements were resolved through email correspondence and face‐to‐face discussions to achieve consensus. There are measures for assessing the degree of agreement in judgments called "interrater reliability". We used two measures that are also typically used: percent agreement and Cohen’s kappa (McHugh, 2012). The data extraction took place using a pilot-tested template and included the following information:

  • Applied studies: aim, methods, stakeholders, and results.

  • Theoretical studies: the contribution to methodological aspects of technical investigation.

All data extractions are available in the appendix (Gerdes & Frandsen, 2022).

Findings

The searches retrieved 2232 publications, and the other methods identified 547 publications. After screening, 54 studies were included in this review. The screening on title and abstract resulted in 103 conflicts for the studies identified via databases and 23 conflicts for studies identified through other methods, which results in 94% and 91% agreement in these two screenings (Cohen’s kappa is 0.39 and 0.36). The full-text screening on title and abstract resulted in 3 conflicts for the studies identified via databases and eleven conflicts for studies identified through other methods, which results in 93% and 66% agreement respectively (Cohen’s kappa of 0.84 and 0.32). For the updated search there were 34 conflicts in the screening of title and abstract (94% agreement and Cohens’s kappa of 0.37) and there were 21 conflicts on full text screening (38% agreement and Cohen’s kappa of 0.09). The process is illustrated in Fig. 1.

Fig. 1
figure 1

Study selection flow diagram

Full size image

Theoretical studies

We identified 22 theoretical studies contributing to the methodological developments of technical investigations. Table 1 provides an overview of the studies and their contribution to the methodological development.

Table 1 Overview: Theoretical studies
Full size table

Applied studies

We identified 32 applied studies. Table 2 provides an overview of these studies and their main characteristics.

Table 2 Overview: Applied studies
Full size table

Thematic analysis of the included studies

VSD reflects a tripartite integrative and iterative design process with conceptual, empirical, and technical investigations. Yet, it still makes sense to zoom in on the technical investigations to discuss how this component is approached in VSD studies. Hence, our thematic analysis outlines theoretical studies’ methodological contribution to the technical investigations. Subsequently, we present an analysis of how technical investigations are addressed in applied VSD studies.

The 22 theoretical studies reflect (a) critical accounts of the feasibility of VSD, or (b) contribute useful Theoretical studies’ methodological contribution to the technical investigations.heuristics for technical investigations, which rely on theoretical reflections or draw on insights from other applied studies. The theoretical studies are organized under the following headings: Contributions conceptualizing the role of the technical investigations; Contributions describing how to adapt and adjust the technical investigations; Critical voices discussing foundational challenges and suggesting revisions.

Contributions conceptualizing the role of the technical investigations

The nine studies presented here define the role of the technical investigations in VSD and describe heuristics.

Before the formation of VSD, an early study by Friedman (1996) emphasized the need for a systematic account of bias “during the earliest stages of the design phase (…) Then comes the task of remedying” (Friedman, 1996, p. 21).

In a positioning article, Friedman and Kahn Jr (2002) present VSD as an interactional position toward inscribing values in technology, which holds “that whereas the features or properties that people design into technologies more readily support certain values and hinder others, the technology's actual use depends on the goals of the people interacting with it” (Friedman & Kahn Jr, 2002, p. 1179). Also, they point to limitations in the fields of computer Ethics, Social Informatics, CSCW, and Participatory Design and emphasize that VSD is unique by emphasizing proactively integrating values into the design (Friedman & Kahn Jr, 2002, p. 1187).

Likewise, illustrating the strengths of VSD, two identical studies (Friedman et al., 2006; Friedman et al., 2008b) present three best practice VSD cases concerning (1) Web browser cookies and informed consent online; (2) the design of a video window displaying the outdoor environment (in offices without windows) while balancing employees’ well-being (direct stakeholders) with the privacy expectations by those, who pass by (indirect stakeholders); and (3) an urban simulator, UrbanSim, facilitating community deliberation and informed decision making related to urban planning. Moreover, illustrative examples are provided. Both studies refer to studies on how video-based collaborative work systems can provide blurred views or not of office settings as an example of value tradeoffs between individuals’ privacy and group members' desire for awareness of peoples’ presence and activities.

In describing technical investigations, the authors (Friedman et al., 2006; Friedman et al., 2008b) note that VSD “adopts the position (…) that technologies (…) provide value suitabilities” following from the properties of the technology. Against this backdrop, technical investigations may address how existing technological properties hinder or support human values. Likewise, in developing new technologies, technical investigations proactively design for values identified in the conceptual and empirical investigations (Friedman et al., 2006; Friedman et al., 2008b). Both studies draw attention to the distinctions between technical and empirical investigations (as mentioned above in the introductory section) and note that empirical activities related to technical investigations focus on the technology itself, whereas empirical investigations focus on stakeholders, “groups, or larger social systems that configure, use, or are otherwise affected by the technology” (Friedman et al., 2006, p. 352), (Friedman et al., 2008b, p. 73).

Both studies provide heuristics for technical investigations encouraging designers to:

(…) make explicit how a design tradeoff maps onto a value conflict and differentially affects different groups of stakeholders (…). Unanticipated values and value conflicts often emerge after a system is developed and deployed. Thus, when possible, design flexibility into the underlying technical architecture so that it can be responsive to such emergent concerns. In UrbanSim, for example, Borning et al. used agile programming techniques to design an architecture that can more readily accommodate new indicators and models (Friedman et al., 2006, p. 367), (Friedman et al., 2008b, p. 93).

Recapitulating the same cases as mentioned above, a corresponding outline is found in Friedman and Hendry (2019), who note that “[i]n one form—retrospective analyses—technical investigations focus on how existing technological properties and underlying mechanisms support or hinder human values. In a second form—proactive design—technical investigations involve the proactive design of systems to support values identified in the conceptual investigation” (Friedman & Hendry, 2019, p. 34). Similarly, Friedman et al. (2017) present a survey of methods helpful during conceptual, empirical, and technical investigations. For technical investigations, they present three heuristics (Friedman et al., 2017, p. 101) identical to the abovementioned heuristics (Friedman et al., 2006; Friedman et al., 2008b). Finally, a reprint of Friedman et al. (2006) exists, including an addendum which do not contribute to the technical investigations (Friedman et al., 2013).van de Poel (2013) conceptualizes values hierarchies, which may be applied to relate values and design requirements and translate values into design requirements guiding the design process. As an illustrative example, the author refers to the design of chicken husbandry systems, i.e., battery cages. Later, van de Poel (2021) follows up on developing a value change taxonomy. Against this backdrop, he identifies technical features (system adaptability, flexibility, and robustness), which “can be designed into products or systems so that they can better adapt to changing values in the later phases of the life cycle of a product or system” (van de Poel, 2021, p. 29).

Contributions describing how to adapt and adjust the technical investigations

The nine studies below document how the technical investigations are presented in theoretical studies, which summarize insights from applied studies or seek to refine and/or merge VSD with other design approaches.

Chen and Zhu (2019) argue that a “more systematic push for the application of Value Sensitive Design in learning analytics is needed.” The authors present two previous studies, one study exemplifies conceptual investigations, and the other (which is included in this review under “applied studies”) presents a VSD algorithm design process. The authors hope these examples can “strengthen the future design of learning analytics systems” (Chen & Zhu, 2019, p. 351). Similarly, Schikhof et al. (2010) recapitulate and reflect upon their approach in an earlier study on the development of a remote monitoring system in dementia care (also included in this review under “applied studies”). Here, they combined VSD with a Human Centered design process and noted that “[b]est practices of incorporating human values (…) are welcomed (…) to improve current design practices” (Schikhof et al., 2010, p. 421).

Longo et al. (2020) evaluate the usefulness of VSD in the field of Industry 5.0 and present three use cases with prototypes to make concrete how VSD can guide the development of Industry 5.0 solutions. Technical investigations are informed by bi-directional values hierarchies, which may be helpful for engineering teams to “specify as best as possible how different design requirements, norms and values relate to one another in any give design project and determine how to most aptly satisfy them” (Longo et al., 2020, p. 14/25). Likewise, Tsunetomo et al. (2022) suggest a design process for smart product-service design (PSS) with participatory and systemic design approaches integrating a value typology based on VSD. The design process is tested in an idealized case study concerning a monitoring service for older adults. Here, technical investigations are carried out using existing technologies and role plays. The authors conclude that the design process contributes valuable insights. The authors conclude that more case studies are needed in future research.

Mykhailov (2022) outlines an analysis of computer intentionality to account for challenges related to self-learning capabilities in AI applications. As such, the author addresses problems of opacity and unpredictability in machine learning, which arise due to the noncausal relation between the designer and systems behavior. The author enriches the technical investigations by illustrating the relation between the notion of computer intentionality and the value of fairness. By the same token, Umbrello and van de Poel (2021) present an AI for Social Good VSD methodology (AI4SG-VSD). They provide conceptual guidelines by modifying VSD to accommodate specific ethical challenges related to AI, which arise due to the self-learning capabilities of machine learning systems. The AI4SG-VSD design process consists of four iterative phases context analysis, value identification, design requirements, and prototyping. They apply values hierarchies to translate abstract values into design requirements (van de Poel, 2013). Following up, Capasso and Umbrello (2022) adapt this design model to a use case with AI-driven digital personal assistants to illustrate how to design responsible nudging in AI health care (Umbrello & van de Poel, 2021).

Similarly, Umbrello and Yampolskiy (2021) explore the usefulness of VSD in the field of autonomous vehicles (AVs). Applying values hierarchies, they show how to design for explainability and verifiability to ensure human control over autonomous vehicles by minimizing “opaque architectures”. They argue that VSD provides “at least a strong enough foundation from which designers can begin to anticipate design needs and formulate salient design flows that can be adapted to the changing ethical landscapes required for utilisation in autonomous vehicles”. Correspondingly, in the context of Industry 4.0, Vernim et al. (2022) introduce a VSD approach to a robotic AI-based assistance system. In the technical investigations, a functional description of this technology is given. The authors note that “this paper centers on a primarily technical investigation given the novelty of the assistance system in question” (Vernim et al., 2022, p. 508). The authors outline ethical redesign and urge researchers to validate their approach in future practice-based case studies.

Critical voices discussing foundational challenges and suggesting revisions

Four studies question the feasibility of VSD and suggest ways to move forward.

Alrechtslund (2006) draws attention to a “positivist problem” in VSD, i.e., the assumption that a correspondence exists between design intention and use context. He suggests that “it is necessary to envision as many multistabilities as possible while designing technology to anticipate future ethical problems and dilemmas” (Albrechtslund, 2006, pp. 71–72). In addition, he problematizes that VSD seems to be “a neutral tool,” lacking an ethical foundation. He rhetorically asks if “VSD [could] operationalize the ethics and values of, for instance, Nazi Germany?” (Albrechtslund, 2006, p. 67). Likewise, Manders-Huits (2011) notes that VSD appears “an attractive approach.” Yet, she argues that VSD is descriptive and cannot properly integrate values in design proactively or refine existing systems because VSD lacks an ethical theoretical foundation. Thereby, it becomes impossible to deal with value tradeoffs and makes value prioritizations in a principled manner. Kozlovski (2022) argues that three commonly used methods to tackle value conflicts (Value Dams and Flows, values hierarchies, substantive ethical theory methods, i.e., selecting a normative ethical theory to guide the design) fail to account for value incommensurability. Instead, the author introduces axiology and normative ethics to incorporate the evaluative relation of ‘parity’, which, combined with rational decision theory, may help resolve problems concerning value incommensurability.

Davis and Nathan (2015) propose guiding heuristics based on VSD critiques, case studies, and related value-based work. In responding to the criticism that VSD lacks an ethical theoretical foundation, the authors note that, in some cases, committing to an ethical theory may help identify and prioritize relevant values. Still, they do not agree that VSD should always adopt a normative ethical theory. The authors suggest that researchers clarify their personal values and their position toward the universality of human values or value pluralism. Specifically addressing technical investigations, they remark that these investigations “are primarily concerned with specific features of technologies. (…) These studies may include designing a new technology to support particular values or analyzing how particular features of existing technologies implicate certain values in a context of use” (Davis & Nathan, 2015, p. 16).

To sum up, the 22 theoretical studies provide guidance or heuristics, often distilled from descriptions of applied studies or otherwise presented as high-level conceptual guidelines. In addition, philosophical discussions of challenges related to VSD suggest directing attention to specific foundational problems, which must be overcome to move VSD forward.

Applied studies—approaches to the technical investigations

The 32 applied studies contribute valuable design insights (redesign or new design) and design guidelines for future technical designs within specific contexts. The applied studies are organized under the following headings: Exploration of a given design space; Design guidelines: redesign (excl. prototypes); Design guidelines: redesign (incl. low- or high-fidelity prototypes or mock-ups); Design: technical development and implementation.

Exploration of a given design space

The seven studies presented here explore a design space to enhance understanding of how values play out in a specific technologically mediated context. The purpose is to provide guidance to designers and inform design in both specific contexts and on a general level.

Two studies explore how to enhance the design of features to facilitate user-driven contributions to public goods. Hence, to understand the motivational aspects of user engagement in the Wikipedia project, Kuznetsov (2006) uses technical investigations to explore and reflect upon how values play out in human-technology interactions on Wikipedia. For example, he notes: “Although the Wiki technology undoubtedly supports altruism and reciprocity through its ease of use and quick navigation, these values are predominantly fostered by the Wikipedians themselves” (Kuznetsov, 2006). Similarly, Yetim et al. (2011) seek to understand motivational values in two cases and review technical design features to suggest features facilitating the motivation to contribute to semantic web content.

In the context of assistive technologies in dementia care, Dahl and Holbø (2012) study stakeholders’ attitudes to sensor-based technology. They identify a set of value-biased features, which provide stakeholders and decision-makers with guidance for the assessment of sensor-based assistive solutions and provide insights for designers within the field of dementia care.

Boyd et al. (2016) report on work-in-progress concerning values and tradeoffs related to “in-context data about computer-mediated work” (ibid, p. 233). In the future, this study may inform the development of activity-tracking systems for researchers while preserving privacy. The authors apply a method for exploring the design space in which participants are asked to rank different scenarios concerning work activities apply the VSD Value Dams and Flows method (Miller et al., 2007) to report bus operators’ attitudes and values toward existing and possible future rider information tools. The authors aim to help developers “deciding the next transit tools to develop and identify barriers (…) to be cognizant of in future design” (Watkins et al., 2013a, 2013b, p. 978). In a follow-up study (Watkins et al., 2013b), the authors apply brainstorming (informed by the conceptual and empirical investigations) and present a list of prioritized applications.

Bleumers et al. (2015) report on young children’s play experiences in hybrid contexts (online/physical) and parental mediation practices. They guide designers on how to consider the role of parental involvement in the design and how to be aware of the manner in which design features “fosters parental play beliefs and support adult–child interaction” (Bleumers et al., 2015, p. 170). In the technical investigations, the authors use online customer reviews to carry out empirical investigations.

Design guidelines: redesign (excl. prototypes)

Nine studies present design choices and suggestions for the redesign of specific applications. In addition, these studies aim to inform design in general by guiding designers. An early VSD study by Millet et al. (2001) refers to “the emergent field of value sensitive design” and specifies criteria for online consent based on a retrospective evaluation of web browsers. The authors “document relevant design changes in Netscape Navigator and Internet Explorer over 5 years, starting in 1995” (Millett et al., 2001, p. 46), identify problems with cookie technology, and provide “design remedies”.

Brush and Borning (2005) report on a field study concerning the value implications of using a group awareness system. They identify privacy and accountability-enhancing design features. Also, the technical investigations are used to inform their field study of “how participants felt about using email for ‘today’ messages” (Brush & Borning, 2005). Miller et al. (2007) study a similar use context and report on a groupware system, CodeCOOP, to support software engineers in sharing knowledge. They develop and apply the Value Dams and Flows method to balance values, inform the design of technical features, and suggest “managerial policies to support system use” (Miller et al., 2007, p. 289). The authors note that more research is needed on calibrating thresholds for value dams and flows in projects.

Setting out to strike the appropriate balance between privacy and surveillance, Czeskis et al. (2010) present strategies “to inform design and technical direction” of parental mobile phone monitoring tools, noting that, e.g., “[t]he design of parent-teen mobile phone safety application must thus support different pathways for sharing the data collected on the teen’s phone” (Czeskis et al., 2010).

Likewise, relying on stakeholder inputs, van Andel et al. (2015) present four design choices concerning a home-based nocturnal seizure detector design. They note that “each design choice has two extremes with corresponding risks and benefits” (van Andel et al., 2015), which provides developers with indications of the consequences of making a specific design choice by showing which risks they need to mitigate and how they might maximize benefits.

Cawthorne and Cenci (2019) report work on designing ethical drones and present a retrospective analysis of a prototype cargo drone, followed by a prospective VSD analysis and redesign suggestions. For instance, the authors note that “[t]o counteract the potential risks to psychological welfare identified in the retrospective analysis such as privacy and the”chilling effect,” it would be beneficial to make the drone highly visible with lights and to paint it in bright colors” (Cawthorne & Cenci, 2019, p. 1123). In addition, Cawthorne and van Wynsberghe (2019) report on a preliminary case concerning a health drone project for blood sample transportation. The technical investigations apply a values hierarchy to translate values into design requirements.

Cummings (2006) applies VSD as a didactic tool in engineering classes. The author applies Just War Theory and presents a VSD redesign case concerning a decision support tool for a Tactical Tomahawk supervisory control interface.

Jaljolie et al. (2022) investigate a geographic information initiative, i.e., the Humanitarian OpenStreetMap Team, focusing on stakeholders, who use or create OSM data and codes of conduct. They present ethically improved recommendations based on the identification of “gaps between the existing OSM’s ethical aspects and the VSD investigation” (Jaljolie et al., 2022, p. 10).

Design guidelines: redesign (incl. low- or high-fidelity prototypes or mock-ups)

Ten studies present new designs based on existing technologies or redesigns of specific applications and include prototyping to test design features or explore the design space. In addition, these studies aim to inform design in general and provide guidance to designers.

Applying insights from Millett et al. (2001), Friedman et al. (2002) present and test three high-fidelity prototypes, including an implementation of a cookie watcher prototype “in the Mozilla browser [which displays] how specific technical mechanisms—of peripheral awareness and just-in-time interventions—can be employed to support informed consent for cookies in the context of individuals’ Web browsing (…)”. They argue that their contribution illustrates the usefulness of VSD in real-life contexts implying large-scale software systems. Likewise, Xu et al. (2012) present the design of privacy-enhancing tools to protect privacy and empower users to control their personal information. Furthermore, they develop and test a prototype that integrates privacy-enhancing tools in an ad on toolbar for web browsers representing a “near-complete” design solution.

Yet another high-fidelity prototype is presented by Davis (2008), who develops a working prototype implementation of indicators, that can easily be implemented in the UrbanSim system, which is a simulator system facilitating community deliberations and informed decision-making about future urban planning by modeling long-term consequences of choices on a legitimized foundation. Here, the household indicators offer “a new approach for enabling citizen interaction with UrbanSim results. Using personal information provided by the user, this web application attempts to address the question, “How will this decision affect me?”” (Davis, 2008).

Schikhof and Mulder (2008) combine Human Centered Design and VSD to develop and test a remote monitoring system, based on existing monitoring aids, for watching people with dementia at night. Here, a working prototype was tested in an improvised bedroom. At a later stage, the remote monitoring system was tested in a real-life usability test at a nursing home context over 4 weeks. The authors report that based on their work, healthcare decision-makers have decided to implement “a remote monitoring system in a small-scale housing project next year” (Schikhof & Mulder, 2008, p. 8). Similarly, Smits et al., (2022a, 2022b) apply a VSD approach to the design of an ambient intelligence solution for remotely and continuously monitoring the quality and safety of patient care. They use care scenario mock-ups and develop a working prototype of an ambient intelligent solution (trained on video clips). Validation of the solution in a real-life setting is planned to collect evidence on safety, privacy and inclusiveness to further inform design recommendations.

Epstein et al. (2013) hope that further evaluation of their VSD approach can “strengthen our ability to make prescriptive design recommendations”. The authors discuss values and tensions in interfaces for sharing step activity data. They suggest a set of data transformations that balance the benefits of detailed data sharing while minimizing harm. Also, prototypes “are implemented in C#, using the Windows Presentation Framework, accessing step data via the FitBit Intraday API (…). Although our current implementation is effective and certainly sufficient for our empirical investigation, it could also be extended” (Epstein et al., 2013, p. 492).

Zhu et al. (2018) developed a VSD algorithm design method to facilitate the early stages of algorithm design to avoid bias or compromising stakeholders. They present prototype implementations of a value-sensitive recruitment algorithm for WikiProjects, developed in close collaboration with the Wiki community.

Prototypes or mock-ups are also used exploratively to enhance stakeholder reflections on design and to delve into the design space. To this end, Walton and DeRenzi (2009) note that health information system (HIS) projects in developing countries often fail to succeed and apply VSD to provide guidance for the design, development, and implementation of HIS in rural African districts. The authors present a specific technical investigation of a vaccine delivery project, including design suggestions for respect and accountability. Here, low-fidelity prototypes, i.e., an onscreen mock-up, allows users and designers to reflect on how different options for answering via multiple-choice and freeform text box may influence values concerning respect and accountability.

A similar explorative use of prototyping is seen in Rector et al. (2015), who explore exercise technologies for blind and low-vision persons. The technical investigations evaluate existing technologies in light of the empirical investigations and provide “stakeholders an opportunity to brainstorm new technologies that address their concerns” (Rector et al., 2015, p. 211).

Similarly, in the context of healthcare, Strikwerda et al. (2022) present a case with a GDPR compliant and ethical app supporting healthcare students carrying out preventive health checks. The authors apply a value sensitive co-creation approach with an ethical matrix to design choices informing a mock-up app.

Design: technical development and implementation

The following six studies conduct technical development, implement systems or features, and inform software development processes.

Three studies are related to the UrbanSim system (described above): Freeman-Benson and Borning (2003) combine VSD with an agile system development method to facilitate the development, iterations, test, project, and project management of the entire UrbanSim system development process. Following up, (Borning et al., 2005) present an “interaction design around indicators for UrbanSim” (p. 456). The authors design an Indicator Browser to enhance functionality and promote stakeholder values while ensuring transparency to legitimize UrbanSim simulations. They emphasize technical documentation. Correspondingly, Schwartzman and Borning (2007) demonstrate how prototyping and agile SW-development can be combined. The authors identify a gap in a previous version of UrbanSim and develop a graphical interface for the Indicator Browser that enables exploring the results of computing indicator values and requesting additional values of indicators to be computed. For example, the authors report on a trade-off between values and usability. Here, a new open-source platform (Python, C + +) has increased efficiency in the latest version of UrbanSim. Earlier, computing indicators could take longer than the runtime for the simulation. Now, indicators can be defined using Opus variable definitions and SQL queries, which is a less transparent format, but indicators can now be computed in minutes rather than hours.

Two studies develop (Thornton et al., 2018) and iterate (Thornton et al., 2019) on an autonomous vehicle speed control algorithm. They model the problem as a partially observable Markov decision process (POMDP) and illustrate “the formal connection of human values to a POMDP design” (Thornton et al., 2018, p. 1161) by aligning POMDP design choices with values from the conceptual investigation to “justify the engineering and explicitly record the embedding of said values” (Thornton et al., 2018, p. 1159).

Finally, Boyd (2022) develops an open-source prototype ML ethics search tool—“[a]nswering the call for supports for ethical algorithm design that are integrated into technologists’ workflows and adaptable to organizational and industry context”. The tool may help data scientists and ML engineers when employing ethical mitigations. Moreover, the tool may be helpful in educational and organizational settings.

To sum up, the 32 applied studies suggest designs, redesigns of existing features or systems, or design of systems to be realized and implemented in the future. The applied studies demonstrate the impact of VSD by highlighting the variety of contexts to which VSD has been adapted.

Discussion

In what follows, we summarize and reflect upon the implications of the abovementioned thematic analysis by organizing the discussion in alignment with perspectives presented in a recent article by leading proponents of the VSD community, which identifies grand challenges in the field (Friedman et al., 2021).

The VSD investigations are interdependent. However, it is still worthwhile to single out and document how studies address technical investigations. Hence, this systematic review includes applied and theoretical VSD studies addressing technical investigations. Thereby, we leave out applied studies, which do not claim to contribute to the technical phase, but which may still inform technical design. Consequently, studies like (Debrabander & Mertes, 2022; Friedman et al., 2008a, 2008b; Kowe et al., 2022; Mohamed & Han, 2022; Smits et al., 2022b; van Wynsberghe, 2013; Woelfer et al., 2011) do not aim to contribute specifically to the component concerning technical investigations (and are hence not included in this review). Yet, their contributions share similarities with studies organized above, particularly in the sections Exploration of a given design space and Design guidelines: redesign (excl. prototypes). This observation indicates that no consistent use of the VSD-methodological terminology has crystalized.

The theoretical studies can be divided into studies that (1) account for the technical investigations drawing on insights from selected exemplary cases, (2) present heuristics for technical investigations while seeking to merge VSD with other design approaches or seeking to refine VSD, (3) question the practicability of VSD and discuss how to overcome foundational challenges in VSD. Several of these studies develop their contributions based on reflections anchored in identical examples of best practice VSD cases.

The applied studies describe technical investigations related to concrete cases and contribute to design explorations, new designs, or redesign of existing systems. All applied studies share the hope that their results can serve as guidance and inform future design directions. These studies articulate a concrete design process and demonstrate that VSD offers a framework for scaffolding design activities with attention to values and stakeholders. Design activities tied to technical investigations spans a continuum from explorative value investigations to translations of values into design requirements for technical implementation. Only one unique project, including three studies, develops a computer system within this continuum: the UrbanSim simulation system. In addition, only one unique recent project, including two studies (Thornton et al., 2018, 2019), presents highly technical systems iterations. Only six studies develop prototypes or systems representing near-complete systems (Boyd, 2022; Davis, 2008; Friedman et al., 2002; Schikhof & Mulder, 2008; Smits et al., 2022a, 2022b; Xu et al., 2012). One of these studies (Davis, 2008) is part of the UrbanSim project.

The VSD methodology is tripartite, iterative, and integrative. However, it gives rise to concern that the technical investigations present technical reflections and solutions to a minor degree. Especially so since proponents of VSD distinguish VSD from Computer Ethics by arguing that Computer Ethics highlights ethical problems related to computer technologies without commitment to their technical solution (Friedman & Kahn Jr, 2002):

(…), Computer Ethics often remains too divorced from technical implementations. How exactly, for example, can HCI professionals build interfaces that enhance trust within a community of users? Or how exactly do we address the value problems that arise through invisible computing? (Friedman & Kahn Jr, 2002, p. 1184).

Likewise, van den Hoven (2007) has praised VSD for its commitment to ‘front loading’ ethics (sec. 1). A similar message is sent by Nissenbaum (2001), who coined the term “engineering activism” to call out for technical engagement within the humanities and social sciences:

Sometimes a fine-grained understanding of systems—even down to gritty details of architecture, algorithm, code, and possibly the underlying physical characteristics—plays an essential part in describing and explaining the social, ethical, and political dimensions of new information technologies (Nissenbaum, 2001, p. 118).

Thus, to frame the discussion of our findings, we apply the article Eight grand challenges for value sensitive design from the 2016 Lorentz workshop (Friedman et al., 2021), which lists challenges concerning how to (1) account for power, (2) evaluate VSD, (3) frame and prioritize values, (4) enhance professional and industry appropriation, (5) influence and inform tech policy, (6) account for values and human emotions, (7) account for challenges related to AI, and finally, (8) how to settle value tensions. Against this setting, we discuss our findings on the backdrop of the three selected challenges concerning issues about evaluation (2), framing and prioritizing values (3), and appropriation (4).

Concerning challenge (2) evaluation, the authors note that in the early days of VSD, evaluation consisted of illustrating that VSD is feasible, thereby providing “proof-of-concept” projects that values can be integrated into technical design. According to the authors, the challenge for evaluating the VSD framework and projects is no longer “can this be done?” (Friedman et al., 2021, p. 8). Instead, the time has come for an evaluative framework, which tunes in on “ “how can this be done well” and “what are the impacts?”” (Friedman et al., 2021, p. 8). Consequently, Friedman et al. (2021) propose to evaluate projects that succeeded or failed.

This approach certainly has merits, but based on findings in this systematic review, we suggest that the VSD community prioritize increasing the stock of VSD projects which actually engage in developing technical solutions. Therefore, we see an urgent need in the VSD community to revitalize the original ambitions of VSD.

Concerning challenge (3), framing and value prioritization. Here, the authors recapitulate standing discussions concerning the role of ethical theories in VSD, the usefulness of operating with a list of prototypical values in technical design (not a canonical list, but a place to start), and whether values ought to be understood as universally held or culturally embedded. Hence, critical questions for further exploration concern issues, such as how researchers can evaluate the effectiveness of committing to or connecting VSD to an ethical framework. Clearly, with the increasing stock of VSD projects, which include an ethical framework (see, e.g., Cummings, 2006; Umbrello & van de Poel, 2021; van Wynsberghe, 2013; van Wynsberghe, 2016), it might be interesting to compare these projects with VSD projects without an ethical framework. Such comparative studies raise complex issues essential to discuss but doing so presupposes the existence of a consistent practice for applying VSD in the first place.

However, this systematic review reveals that design activities in the technical investigations span a broad continuum from explorative brainstorming sessions to the development of algorithmic models, which makes it hard to demarcate constitutive activities in the technical investigations and distinguish between the types of activities that count as technical investigations and those that definitely do not constitute technical investigations. This observation may also partly explain the existence of challenge (4) professional practice and industry appropriation. Here the authors point to three problems related to the lack of transfer from VSD to professional practice and the industry, i.e., lack of alignment of VSD to engineering practices, lack of concrete guidelines, and missing clear organizational roles to make VSD work (Friedman et al., 2021, p. 11). The authors set a clear direction for problems to solve when seeking to facilitate a broad uptake of ethics in system development practices. Nevertheless, it won't be easy to define concrete guidelines and align VSD with external professional practices if the internal research practice of the VSD framework is not well-aligned.

In sum, there is little engagement at the technical level in the VSD community, and this observation confirms the results from the review by Winkler and Spiekermann (2018), who identified and investigated seventeen publications that apply the complete tripartite methodology.

Conclusion

This systematic review documents how technical investigations have been approached in theoretical and applied VSD studies in the period from 1996 to 2023. Thereby, the systematic review provides a contribution to the VSD community when seeking to further develop and refine the methodological framework for carrying out technical investigations in VSD. Based on our systematic review, we suggest a revised research agenda for VSD committing to informing the technical design.

The VSD methodology’s three components, i.e., conceptual, empirical, and technical investigations, are interdependent. Yet, the absence of technical analyses, reflections, and solutions is a problem for the VSD approach. It is a problem because of the ambition behind VSD, and even though VSD studies contribute valuable insights concerning design, redesign, and design guidance.