Introduction

Value sensitive design (VSD) can look back at over 20 years of constant development and is considered by many as the most comprehensive approach to account for human values in technology design (e.g., Manders-Huits 2011; Davis and Nathan 2015). Against this background, the goal of this paper is to review the methodological practices in VSD projects. We want to understand how VSD has been applied, especially in terms of the reported use of selected methodological elements for design projects that employ the VSD tripartite methodology. To achieve successful incorporation of human values in the design process, VSD employs an integrative and iterative tripartite methodology, consisting of conceptual, empirical and technical investigations (Friedman et al. 2006). In the conceptual investigation, direct and indirect stakeholders are identified, followed by an analysis of how these could be harmed by or benefit from a new technology. Additionally, values implicated by the use of technology are identified and defined (Davis and Nathan 2015). As soon as values are identified and discussed, value tensions can emerge (Friedman et al. 2006). Under empirical investigation, qualitative and quantitative methods are employed to evaluate how stakeholders experience a technology with regard to the values they consider important (Manders-Huits 2011). One aim of the technical investigation is to combine insights from the other investigations and explore how a technology might be designed to support the values identified (Manders-Huits 2011). All three investigation types are interdependent and inform each other (Manders-Huits 2011). Burmeister (2016) shows how an identified value can be refined by empirical insights, which support its in-depth conceptualization and understanding. Due to the interdependency of the three investigations, we consider iterations between them as an important cornerstone of the VSD tripartite methodology.

Method

We conducted a literature review on VSD from 1996 to 2016, using the online libraries of the Association for Computing Machinery (ACM), the Institute of Electrical and Electronics Engineers (IEEE) as well as Science Direct and Springer. We considered all peer reviewed journal publications and conference proceedings with the term “value sensitive design”, “value-sensitive design” or “vsd” in the title, abstract or as a keyword. Furthermore, we included all papers that are listed at vsdesign.org, a website maintained by the Value Sensitive Design Research Lab and Batya Friedman, the pioneer of VSD (Friedman 1996).

We identified 229 contributions in total, of which ten were not accessible. When examining the 219 accessible papers, we found that 106 are grounded in related research domains such as ethical computing or responsible innovation and do not apply any investigation of the tripartite methodology. This left us with 113 papers, of which 63 focus solely on the conceptualization of various values, 22 contribute to individual methodological techniques of VSD (e.g.: Friedman and Hendry 2012) and 11 propose extending views (e.g.: Le Dantec et al. 2009).

Our analysis is focused on the remaining 17 publications that develop technology with the complete tripartite methodology. To review them methodologically, we consider five VSD outcomes: (1) identification of direct and indirect stakeholders; (2) identification and conceptualization of values; (3) understanding of value harms and benefits; (4) development of mitigation strategies for value tensions; and (5) presentation of technical measures to address values. In addition to these five essential outcomes, we consider quality of method description and therefore emphasize the importance of reproducibility.

Findings

Table 1 shows that VSD projects have been conducted in a wide range of technology domains, suggesting its applicability across contexts. Of the 17 projects, 16 start with a conceptual value investigation, followed by nine with an empirical analysis. Seven go right into technical investigation after conceptualization and later validate values empirically. Only four of the projects report iteration between investigations.

Table 1 Considered projects and their investigation procedure
Full size table

Conceptual investigation

Three of the 17 projects mention a specific or reproducible method for stakeholder identification: Rector et al. (2015) had a brainstorming session, Boyd et al. (2016) reviewed the literature and conducted interviews, and Miller et al. (2007) identified stakeholders according to their interaction roles with the proposed technology. Only seven projects reported the demographic data of their participating stakeholders. Nine studies explicitly mention the identification of possible harm and benefits during this phase. Identification of values relied on a mixture of methods such as literature reviews from earlier technology implementations, content analysis, guided brainstorming techniques, etc. (Azenkot et al. 2011). The depth of value conceptualization varies heavily between studies and we were unable to clearly discern how conceptualization was achieved methodologically in any of the VSD projects.

Empirical investigation

Empirically, three projects employed semi-structured interviews and three used a survey. Four combined these to a mixed-method approach. The other seven studies used rich creative formats to inform their design thinking, including value scenarios (Czeskis et al. 2010; Epstein et al. 2013), prototypes (Friedman et al. 2002), focus groups (Walton and DeRenzi 2009), field experiments (Dahl and Holbø 2012), co-design workshops (Azenkot et al. 2011) or content analysis (Brush and Borning 2005). Unfortunately, only 11 studies reported on their methods in a reproducible way.

Technical investigation

All projects provided technical measures to address the identified values. 11 reviewed existing technology, connecting known technical features with the values to be addressed (i.e. encryption for security). In doing so, they relied mostly on the background knowledge of their authors, but in some cases additionally considered input from stakeholders, such as customer product reviews (Bleumers et al. 2015), rankings (Boyd et al. 2016) or semi-structured interviews (Davis 2008). The other six projects utilized stakeholder collaboration, envisioning, brainstorming or the value dams and flows method as a way to derive necessary technical requirements (Miller et al. 2007). Seven projects explicitly recognised the possibility of value tensions, but only one structured and reproducible method was proposed to deal with them—notably the value dams and flows method mentioned above. In the remaining six projects it is unclear how recognized value tensions were resolved. It is mostly the case that authors let stakeholders envision or brainstorm alternative technologies as a solution. Figure 1 summarizes our findings.

Fig. 1
figure 1

Percentage of 17 VSD projects containing selected methodological elements to achieve core outcomes

Full size image

Discussion

Keeping in mind that conceptual, empirical and technical investigations are interdependent and that iteration is the bond between them, it is alarming that only four projects reported iterations that promise enhanced designs. VSD does not prescribe an order for the three investigations (Watkins et al. 2013). Yet, if iterations cannot be made (for example, for lack of time), then it seems reasonable to have an empirical analysis that follows upon the conceptual work in order to deepen value insights. It is only then that a technical investigation would seem advisable.

Considering technological complexity and ubiquity, identifying stakeholders (especially indirect ones) is increasingly challenging. Many studies did not employ a good methodical approach for stakeholder identification. As a result, only broadly defined stakeholder groups were used. This practice can undermine value sensitivity as the accidental exclusion of stakeholders and their values can infringe other relevant values (Manders-Huits 2011).

A key factor in advancing any scientific field is a good description of methods, as this allows for the replication and advancement of methods. This emphasizes the importance of good documentation (see Dahl and Holbø 2012) for the future of VSD. There is limited guidance on how to accomplish certain tasks (Burmeister 2016), which can be especially challenging for researchers new to the field. A detailed report on how value conceptualization is achieved methodologically is particularly rare (see: Rector et al. 2015). A good overview of available VSD methods can be found in Friedman et al. (2017).

In order to derive technical design solutions, most authors used known features from existing technology. Our suggestion is to use a more structured approach, such as the value dams and flow method (Miller et al. 2007), to determine whether such features are in conflict with each other. It also seems promising to explore other approaches for VSD that have been proposed in the classical field of requirement engineering to support human value.

Conclusion

VSD has generated a rich body of literature on value conceptualizations, methodological papers and projects. Despite this success, we see a lack of methodological guidance, especially for researchers new to VSD. There is room for experienced VSD researchers to provide more best practices, which are evaluated on their long-term effects. There is also the need for more reproducible guidance and methodological descriptions to ease entrance barriers to the field. Lowering method barriers in combination with the existing openness of VSD towards other disciplines and methods is a fertile ground for evolving the discipline and its relevance over the next 20 years.