Keywords

1 Introduction

Intensive growth of virtual, augmented and mixed reality solutions opens up new possibilities for better immersive experience of software solutions for end users. However, the rapid growth of those areas that comprise the immersive extended reality continuum (XR), brings new challenges to the software development process and teams. Despite the wide range of existing methods, tools and approaches for developing diverse mobile and desktop environments, including user-centered design coupled with end-users participation, i.e. co-design and participatory design approaches, there is still a need to select and tailor the most promising solutions to new interfaces and modes of interaction of the XR continuum. Recently increasing pace of both hardware capabilities and its proliferation puts pressure on software developers to deliver more content as more diverse groups of users enter the market for Immersive Extended Reality Environments (IERE)Footnote 1. Therefore, the objective of this paper is to present our findings regarding the methods, practices and tools aimed to facilitate direct involvement of users of varying ICT-proficiency levels, particularly from vulnerable groups, in the participatory design process for the development of new interfaces and modes of interaction related to virtual, augmented and mixed reality (VR, AR, MR). In a series of empirical and field studies [1, 2, 6, 9,10,11, 18, 23] we examined various experimental setups for stakeholders’ participation within and across different steps and phases of the software development processes. Therefore, we provide an overview of selected methods, practices and tools, that we found best support communication, collaboration, and cooperation among stakeholders, especially members of vulnerable groups who are often excluded from the main technological discourse.

2 Related Work

Participatory design, or co-design, engages users in the development processes [20, 21], allowing them to directly shape the designed solutions at their core. Users can provide insights at different stages of the process [5, 15] - or, according to Ladner [14], become designers themselves, in Design for User Empowerment. Without this, even solutions meant to work towards Social Good can be tinted by stereotypes [24], which can become evident in unexpected places due to unconscious biases held by the project team, putting into question the ethical aspects of such solutions, as some some stereotypes may trickle down into the immersive environments built. [4] To prevent this, user engagement during the design and development is crucial. One way of approximating it is by forming Living Labs. The term Living Lab itself was coined by William Mitchell from MIT [16] and indicated a space where routines and everyday life interactions of users and new technology could be observed and recorded [3, 22], to examine the users’ needs and avoid business risks [19]. However, besides problems with maintaining long-term and sustainable user communities [17] a realistic Living Lab, requires significant investments. Some alternatives, like lightweight living labs [9], were proposed where users were animated with interesting activities which keeps them engaged. This approach is also useful for new and emerging interfaces such as extended or mixed reality (XR), like VR, AR or voice interaction. There is also a lot of room for further research to establish a comprehensive and ubiquitous interface that combines all of the extended reality technologies. In this vein, it is possible to engage users in a distributed Living Lab environment [11] aimed at testing and developing these solutions, and eventually even engage them in start-up development teams, as in the SPIRAL method described in one of the previous CHASE workshops [9]. Our RAPID approach draws from all of these methods, while opening the discussion for further exploration of new and emerging interfaces on the extended reality continuum in the context rapid software development.

3 RAPID Approach Outline

Based on our previous experience with empowering older adults and other vulnerable groups for participatory design, we developed an instant environment called RAPID (Rapid and Agile Participatory Interactive Development) devoted to lightweight Living Lab conditions. The RAPID approach consists of three phases divided into steps and for each of these we list related methods and expected outcomes.

3.1 I. Preliminary Phase: Team Formation

This preliminary stage consists of two steps, that can be realized in a few days or partially omitted, in particular step 1. in experienced teams or even step 2. based on communities such as Living Labs or local activity groups.

Step 1. Core XR Team Formation and Empowerment. Internal workshops with methods presentation, discussion and roles assignment. This is an important step when we prepare the team for direct cooperation with users as they may hold some stereotypes about them as well. Internalization of methods is important, especially to ensure unbiased and open cooperation with vulnerable users and among each other in the in-team role assignment, i.e. product owner, team leader, technology advisor, developer, designer etc.

Methods and Tools that Can be Used:

  • evidence from previous successful projects with direct user participation in different contexts, i.e. videos, artifacts (actual applications, products) which may come from outside of the ICT-area, for example, from participatory Social Design projects,

  • interviews with designers successfully engaging in such projects, sharing the benefits and challenges from their points of view

  • ideas about what the team knows about the users, and what is unknown, which can later be verified

  • brainstorming for ideas and insights about the solution with core team members, further facing their stereotypes of the other group (users),

  • affinity diagramming for collection, categorization and analysis of ideas,

  • mind mapping for methods, ideas, team roles and responsibilities.

Expected Result: Team members understand the importance and goal of the process and are open to direct cooperation with end-users, while remaining aware of their unconscious biases.

Step 2. XR Team Extension: User Engagement. Preliminary trials with representatives of the group of end users, user engagement, recruitment and selection in order to extend the core team (from local communities, groups, Living Labs). User engagement session. Preliminary trials with end-users as potential team members. Controlled audio-visual immersion into the IERE world. Direct interaction (i.e. controllers) optional, not required.

Methods and Tools that Can be Used:

  • showcasing interaction with various IERE interfaces (Brain-Computer Interface (BCI), Voice User Interface (VUI) and various types of XR solutions)

  • brainstorming possible everyday uses of such interfaces to invoke creativity

  • engaging users in a simple, largely passive, VR environment (i.e. cardboard or video pass-through)

  • advanced XR environment (i.e. full headset or an immersive AR game)

  • fully fledged demos and trials (i.e. fluent and attractive)

  • semi-structured individual and group interviews with users, screening forms.

Expected Result: Recruitment of two to six end-user representatives with high motivation and a creative flair for the main co-design phase session and team participation.

3.2 II. Main Phase: RAPID IERE Development

Main phase consists of two stages: user empowerment stage (steps 3 and 4) and co-design development stage (steps 5, 6 and 7). Each stage can be realized in one day, or extended as needed.

Step 3. Introduction: Discussion of the Goal of the Workshops and Current Pre-workshop Expectations of Participants. An informative and motivating introduction is key, as both the potential users and the development team need to realize the purpose and importance of their work. Additionally, the participants ought to introduce themselves, to provide their background on technology use, and their experience with the specific topic and their own expectations of these workshops. When engaging vulnerable groups it is crucial to not strain their resources and to ensure that the benefit is mutual for the parties involved. Next, in order to set baseline expectations, we introduce our participants to the idea of the project without putting it in the context of specific technologies nor equipment.

Methods and Tools that Can be Used:

  • ice-breaking games to learn about everyone’s motivations and aspirations

  • division into sub-teams, each with at least one representative of a different group (target user, programmer or designer), to facilitate debates and allow everyone enough speaking time to voice their opinions

  • semi-structured interviews

  • brainstorming, affinity diagramming, mind mapping

  • co-creating mood and context boards exploring the potential of the project

  • sharing insights accross different sub-teams

Expected Result: The participants understand the importance and goal of the workshops and share their experiences with the subject matter of the workshop (e.g. in our cases: banking technology and ATMs, potential uses of Smart Home technology with VUIs and BCI, workplace stress, intrusive thoughts and relaxation techniques).

Step 4. Immersion with Interaction: Hands-On Presentation of the IERE Technology. This step is crucial for the target group to get a feel of what is possible in the technology of their choice, so that they are not limited by their presuppositions or lack of experience in the development stages that follow. It may also be beneficial for the target group to witness the development team also discovering something new and engaging within the demos. Moreover, experiencing immersion may get the users excited about this technology, awakening their creativity. In this step the target users try IERE gear and engage in interactive activities, which to some extent may be similar to the ones being developed later, either in scale, means of interaction or goal (Fig. 1).

Fig. 1.
figure 1

In one of our workshops older adults played the NVIDIA® VR Funhouse game, to become aware of the wide range of what is possible in VR

Full size image

Methods and Tools that Can be Used:

  • Showcasing passive commercially available low-cost products that are within their reach and their key functions: Cardboard 360 movies and experiences, XR experiences, VUI in their own smartphones

  • Engaging the users in active experience creation: capturing 360 photos with their own phones, setting functions of their own voice assistants

  • Explaining the idea of the Internet of Things and how experiences can be combined and co-dependent

  • Letting all of the participants engage with commercial higher quality games and applications available on higher-end devices, which display different and more-advanced aspects of IERE and the range of possible interactions with the real world and each of them through the Internet of Things

Expected Result: The participants understand the range of possibilities that exist in IERE and get excited about the technology.

Step 5. Development - Environment in IERE. While working on any type of project that involves software engineering many environmental factors come into consideration, such as type of devices, software choices or general theme. While in our cases most of those factors were predetermined by the concepts of the project, hardware solutions available to us at the time and team’s skill, we were able to involve all of our participants in the process of designing XR playing space and brainstorming on the potential future uses of other IERE solutions, while also gaining valuable insights in their real life preferences.

Fig. 2.
figure 2

We printed out VR Sketch Sheets, to be used for web-based XR environment design from https://blog.prototypr.io/, however, using them directly with members of our target group proved to be too reliant on an unknown metaphor and too detached from the familiar experience.

Full size image

Methods and Tools that Can be Used:

  • Semi-structured interviews

  • Showing existing large scale projects and their interactions, eg: Google Earth VR Street View in different locations, engaging in unscripted interactions with audio assistants

  • Evaluating existing solutions as design case studies: eg: Rating different aspects of the presentation of locations, talking with virtual assistants

  • Presenting the specific case and evaluating it in the same format

  • Using paper prototyping tools to jot down insights: XR Paper Prototyping with the use of Sketch Sheets 2)Footnote 2

  • Creating flow diagrams to depict the discussed interaction methods and aspects

Expected Result: The participants are involved in project defining decisions, can visualize and evaluate the suitability of different environments, weigh the dangers and benefits associated with each choice, and come up with their own ideas regarding environmental variables.

Step 6. Development of the Concept. Working in IERE environments opens a whole new range of possibilities regarding UI and UX, as such interfaces rely on approximations of real interactions to a greater extent, therefore it is of utmost importance to have as much feedback as possible. By using quick photoshop mockups, 3D modeling software and interaction and flow mock-ups in web-based tools like Proto.io, we were able to gather important insights on what our participants expect from our projects, while allowing them to quickly see mock-ups of their ideas (Fig. 3).

Fig. 3.
figure 3

Older adults were shown web-based VR mock-ups from Proto.io on smartphones

Full size image

Methods and Tools that Can be Used:

  • Paper prototyping of interface elements and their functionalities, which are later turned into quick clickable mock-ups on e.g. Adobe Experience Design

  • Proto.io for UX tests on the initial UI

  • Adobe Photoshop rapid UI prototyping

  • Unreal Engine as a quick means of creating simple VR project

  • Autodesk Maya 3D for 3D modeling purposes

Expected Result: The participants conceptualize and create UI and UX elements while having the ability to give immediate feedback to our rough interpretations.

Step 7. Development - Functionalities. This step, as one too technical for easy and quick implementation, focuses solely on the participants’ ideas and insight without prototyping the functionalities there and then. Participants explore ideas and conceptualize their implementations and restrictions, to go along with the project concept and future and existing assets. It is important to give freedom to participants’ ideas and leave out any implementation restrictions. While applicable pre-made sets of interactions can be presented during discussion as a starting point, it is not necessary.

Methods and Tools that Can be Used:

  • Passive interactions with environments in Proto.io, Unreal or Unity

  • Wizard-of-Oz method

  • Brainstorming with mind-mapping the functions in each environment

  • Engaging with other fully functional IERE interfaces to remind of the possible range of interactions

  • Affinity diagramming of most commonly suggested concepts

Expected Result: The participants conceptualize project functionalities, without software/engine restrictions.

3.3 III. Closing Phase: XR Product Delivery and Testing

This phase is from the classic user-centered design approach, so we do not elaborate on it.

Step 8. Testing with End Users. We would like to point out three important groups of methods, i.e. qualitative interviews and observations and quantitative sensor-based methods and tools.

Methods and Tools that Can be Used:

  • Semi-structured qualitative interviews,

  • Hands-on usability tests,

  • Eye-tracking and other sensor-based quantitative methods.

Expected Result: Verifying the results with extended team members and new participants, as well as the extensive network of contacts they all have to gather fresh perspectives. Testing is important and can be done outside of the work setting, by presenting the demo at a conference or attending fairs, events where members of the target group may be present.

4 Discussion

Our RAPID approach was developed and verified in design cases, for example a series of design workshops of a virtual simulator of ATM and well-being XR environments [12], while its user-empowerment elements were tested in research related to Voice User Interfaces (VUI) [13] and Brain Computer Interfaces (BCI) [8] for IoT, as well as in a case study of an all round XR-hackathon with several dozens of stakeholders from various groups, including software engineers and content developers [6, 7]. Overall, the RAPID approach is well-suited for creating XR solutions with users who need to be empowered, in our case older adultsFootnote 3, and we expect this to be true for other vulnerable groups at risk of being viewed through the lens of stereotypes or just feeling vulnerable in new circumstances.

4.1 Phase I: Team Formation

In the case of participatory design high motivation and engagement are crucial, which is why RAPID Phase I is essential for the success of the approach. Here are some takeaways from our practice:

  1. 1.

    One good practice is to have the team discuss the most promising target user candidates, in terms of their involvement and the ease of collaboration. As this approach is fast, it is important to gather people who enjoy working together, are interested in the project and want to express themselves. On one hand this may seem counter-intuitive, as we give voice to the target group members who are outspoken already, but on the other hand, they do belong to the target group, and they are more likely to be early-adapters, so we in a way, we do design for them when we work on the early version of the solution.

  2. 2.

    We found that granting the potential target groups access to technology, which may otherwise be out of their reach (either financially, or because of the setup) is a very good way to guarantee engagement, and usually it is enough to convince them to participate in the development process.

  3. 3.

    Vulnerable users, who are shown technologies they had no experience with before, need assistance from dedicated staff who encourage them and ensure the first impressions are smooth.

4.2 Phase II Development

After the users excluded from the main technological discourse join the development team, empowerment is even more important. They are now among people who are tech-saavy, driven and may dominate the discussion. Some takeaways from our studies indicate that:

  1. 1.

    One strong need of our participants was to clearly understand the technology and the nature of the development process, as well as the end goal of the project and each of the involved parties’ stake in it. For example, in XR workshops it was reflected in the preferred use of familiar terminology, such as “simulated” instead of “virtual”.

  2. 2.

    Taming the technology: surprisingly every participant had almost no issues with getting used to VR headset and its controllers. The participants treated both stationary and room scale VR experiences as a novelty. They unanimously agreed that room scale VR felt more impressive; however, some participants concluded that cardboards are better suited for beginner users. The same was true of using VUIs, but in this case some participants decided this solution could work for other members of the target group, but not for them - showing some prevailing intergroup stereotypes to be aware of, even when engaging in participatory design.

  3. 3.

    Giving a place to start: The discussion inspired by Google Earth VR Street View in 3D was valuable, as it gave the participants space to imagine other interactive aspects of the possible solution, without the need to prototype it first. The users explored potential scenarios for the ATM use, as well as locations, pointing to specific places and recalling different situations, as well as mentioning their own insecurities about them.

  4. 4.

    Drawing and prototyping: It is not necessary for the target group to draw and conduct prototyping by themselves, as designers may do this for them following instructions and descriptions they provide. Overall, Design for Empowerment, where users become designers, does not have to mean that they have to become craftspeople as well. For XR setting, paper prototyping did not seem to work for us, as the metaphor of changing 2D depictions into 3D ones was too detached from the target project for users lacking experience, therefore we found low-fidelity prototyping to be better suited for generating insights.

  5. 5.

    Not being shy about half-finished products: In our ATM case, thanks to the participants’ interaction with 3D mock-ups we gathered multiple insights concerning both the build in terms of product design of the ATM as well as its UI, discussed in the context of existing ATMs. We have also gathered ideas regarding preferred relaxation environments, including all immersive aspects such as sounds, colors, movement and avatar placement and their realism, down to the wind movement and animal species.

4.3 Phase III Closing

The UX tests were conducted both by the users and the development team in order to maintain the fresh approach and view on the different aspects of the ATM and its usability, controls, environment and the final purpose as a training simulation and the well-being environment (Fig. 4).

Fig. 4.
figure 4

Example of immersion verification with eye-tracking and psychophysiology

Full size image

4.4 Other Considerations

Scheduling. RAPID is designed to be concise but scalable. This means, that it can extended or compressed to just five days, as follows:

  • Day 1 I. Preliminary Phase - team assembly and empowerment;

  • Day 2 I. Preliminary Phase - user recruitment and engagement;

  • Day 3 II. Main Phase - user empowerment;

  • Day 4 II. Main Phase - RAPID prototyping;

  • Day 5 III. Closing Phase - preliminary product delivery and testing.

Moreover taking into account the typical iterative nature of agile approach and rapid prototyping, some steps can be omitted i.e. core team empowerment or user recruitment, based on previous cycles of software development process or previous projects. Below we present a possible schedule of implementation of the RAPID approach as a sprint.

It is vital to schedule the meetings with enough time for questions and digressions since the whole team and the users alike need to feel their presence and insights are of importance - to an extent even if they relate to the issues outside of the main goal of the workshop. While they may not be crucial for the end product, these questions are essential for the process that relies on open sharing. Often such workshops run long, so ordering catering is another good practice.

Venue. The RAPID approach will work best if the venue changes, depending on the focus of each phase.

  • I. Preliminary Phase: while Stage 1 with the team can be conducted anywhere, it is advisable to get out of the usual place of work to allow the team to think out of the box in a more creative setting, which does not resemble the regular work setup. Stage 2 on the other hand should be done in an open social environment, such as a fair, a community center or any other open venue in which potential participants may feel empowered to take the novel gear on a trial run.

  • II. Main Phase: This phase can happen in any flexible workshop space with the possibility to create a round-table to facilitate open discussion

  • III. Closing Phase: Ideally the venues in this case would be environments similar to where the end product is expected to be used.

4.5 General Discussion

Despite the fact that many methods and tools were developed in recent years, there is still a room for discussion to propose a better, more comprehensive approach for the software development teams to address the challenges of constantly changing hardware landscape of IERE. For instance, many of use cases described at premier software engineering and HCI conferences were based on the rapid prototyping using cardboard platforms, thus became more or less obsolete or hard to implement, as support for these weakened or ceased. On the other hand, there are also some shortcomings of paper prototyping that are not easy to overcome, because new interfaces and modalities are not just a simple extension of the WIMP paradigm (windows, icons, menus, pointers). In contrast, methods for prototyping 2D interfaces are well-established in both software engineering and human-computer interaction with plethora of effective tools, including low fidelity mock-ups that can be prepared even by the end-users themselves.

Thus, drawing from our research experience, we concluded that if even users who are generally excluded from the main technological discourse can creatively and competently engage in such 2D prototyping activities with a proper empowerment. On the other hand, during our research on software engineering for advanced and multi modal IERE interfaces we observed that unlike prototyping mobile or web-based flat environments and applications there is a gap between proficient software designers, developers and end-users. This gap is evident in the field of paper prototyping of immersive interactive environments, which constitutes an important barrier to direct cooperation with end-users, content designers and software engineers. Even empowered users have significant difficulties in bridging the gap between low-fidelity paper prototypes and hi-fidelity immersive multidimensional interaction. This situation resembles the case of architectural design, where drawings and schemes are not enough for many end users to fully understand and imagine the final solutions, hence the industry practice of 3D visualisation of models and spaces, more recently with the use of VR-based solutions. In our case, the same metaphor applies, where prototyping for IERE has to “hit closer to home” to enable the users to fully understand the functionalities of end products and to be able to contribute feedback and relevant insights. These low-fidelity functioning prototypes are generated based on insights from the earlier steps in the development cycle, which rely heavily on the empowerment of users, consisting of demonstrations, free use of technology and discussions with the team, unhindered by fears and false preconceptions, and open to constructive criticism.

5 Conclusions

In this paper we outlined the RAPID approach with example methods and tools based on a series of case studies. It is an attempt to discuss and explore new approaches to rapid software prototyping and development relying on user-empowerment of users typically excluded from the main technological discourse and facing unconscious biases in the context of immersive extended reality environments (IERE). The proposed approach allowed us to gain valuable insights on the development process of Immersive Extended Reality Environments, including such aspects as UX, UI, 3D models, web-based mock-ups and current and future functionalities, but it remains to be seen whether the software produced with this approach perform reliably better and are more likely to be used by these groups. At the same time we are excited about the prospects of the discussions and follow-up studies, further testing the limits of such sprints, and refining such IERE development methods to facilitate the creation of great experiences directly with users excluded from the main technological discourse, empowered to share their opinions, needs and aspirations.