The current paper reports on the design of Project PHoENIX, which stands for Participatory, Human-centered, Equitable, Neurodiverse, Inclusive, eXtended reality. Project PHoENIX seeks to engage autistic users in the co-production of research that has relevance to their lives and to explore alternative research that focuses on the strengths, preferences, and lived experiences of autistic people to create a virtual reality (VR) environment that is highly usable, accessible, and sensitive to the needs and preferences of these individuals. In the following sections, we first provide an overall literature review on VR and autism and issues of limited design precedent of VR environments with autistic participants. Then, we provide information on the design framework of project PHoENIX along with the project description and project design outcomes. Next, three main phases within the design process among the project PHoENIX design team are explained along with how the online VR environment was be co-designed and co-developed through collaborative research with autistic stakeholders while being sensitive to their needs and preferences. Lastly, the “Discussion” section describes what the project team members learned throughout the design process of project PHoENIX and further explains how this project provides much-needed design precedent for advancing the field towards a more inclusive, human-centered, and neurodiverse VR R&D paradigms.

Extended reality technologies have the potential to allow users to feel emotionally, cognitively, and spatially immersed inside of digitally-rendered environments (Bjork & Holopainen, 2004). These technologies fuse a range of digital reality technologies, such as augmented reality, 360 videos, and VR, to immerse users within digitally rendered virtual environments. Among these, VR is an emerging technology that presents a computer-generated environment to a user, allowing for an immersive experience that can simulate a realistic and interactive environment. VR typically employs specialized equipment such as head-mounted displays and hand controllers to present a 3D environment and provide sensory feedback. In recent years, desktop-based VR systems have become increasingly popular, providing a high-quality VR experience without the need for specialized hardware. This technology has broad applications in various fields such as education, training, and healthcare. Desktop-based VR systems, such as the one discussed in the current article, have proven to be a cost-effective solution, particularly in the educational sector, as it eliminates the need for expensive hardware and provides a platform for interactive learning.

Specifically for autistic users, studies have shown that immersive technologies can promote positive benefits (Newbutt et al., 2020). Applications of VR have a long history of being used to address a range of challenges for autistic users, such as social, communication, and daily living (Lee et al., 2003; Pan & Hamilton, 2018; Zhao et al., 2018). For example, Parsons and colleagues (2006) designed desktop-based virtual environments for autistic adolescents and adults to practice social skills. Other studies explored how to design multi-user VR environments to teach social competencies to adolescents on the autism spectrum (Schmidt & Schmidt, 2008). Researchers have also investigated the use of 360 video-based VR technology (Schmidt et al., 2021) to promote the acquisition and generalization of adaptive skills.

The majority of research to-date has focused on how to correct autistic people’s behavior or treat so-called “deficits” (DeFilippis & Wagner, 2016; Genovese & Butler, 2020; Williams, 2010). These studies rest on what is referred to as “medical models” of disability, which position disability as a deficiency that exists within an individual and should be corrected (Schmidt & Glaser, 2021a, b). However, researchers have recently begun advocating for studies that support and enable autistic individuals, as opposed to focusing on how to treat or cure them (Newbutt et al., 2017; Parsons & Cobb, 2011; Andrunyk et al., 2018; Newbutt et al., 2016). Therefore, calls are increasing for research in VR that is socially oriented and developed according to the needs and preferences of autistic people. In light of this, Project PHoENIX seeks (1) to engage autistic stakeholders in participatory design and collaborative research within an immersive VR environment, (2) to explore alternative research that focuses on the strengths, preferences, and lived experiences of autistic people, and (3) to include autistic preferences throughout the design, such as by using the preferred terminology of autistic communities: “autistic”, “person on the autism spectrum’, and ‘autistic person’” (Bury et al., 2020). The current design case details how we went about addressing design considerations and technology affordances in developing a collaborative VR environment that intentionally embodies autistic people’s needs and preferences.

The last two decades have seen the prevalence of autism steadily increasing. Currently, around 1 in 44 children in the USA have been identified as being on the autism spectrum (Maenner et al., 2021). Autism is a complex, life-long neurodevelopmental condition that begins in early childhood and can significantly impact daily functioning. According to the American Psychiatric Association, this condition is characterized by social communication and interaction differences, restricted interests, and repetitive behaviors (APA, 2013). In the literature, the concept of a spectrum is used to refer to the continuum of challenges in development or the type and severity of symptoms experienced by autistic people. For instance, some autistic individuals might be “twice exceptional,” with remarkable learning and problem-solving abilities. Others might be very severely impacted, requiring lifelong care to perform daily activities. Most people on the autism spectrum experience a range of comorbidities such as epilepsy, cognitive impairments, and ADHD. Although a great deal of variation exists in the degree of impairment from one autistic person to another, nearly all autistic people benefit from additional support (i.e., training, therapy).

Evidence suggests that autistic users engaging with interventions built using immersive technologies have experienced moderate improvements in social, communicative, and emotional competencies (Herrero & Lledó, 2019; Mosher et al., 2021). However, typical XR interventions for individuals on the autism spectum are limited in that they (1) do not focus on adults, (2) ignore daily living and lifespan issues, and (3) are developed without the input of end-users (Roberts et al., 2022). For autistic adults, daily living interventions are particularly valued (Harris et al., 2021b), and non-immersive (i.e., traditional, non-XR) technology interventions have demonstrated efficacy for improving outcomes related to activities of daily living (Domire & Wolfe, 2014; Gardner & Wolfe, 2013). XR technologies are believed to be particularly promising for providing training and therapy in a very realistic manner but without real-world risks, thereby potentially promoting generalization of knowledge and skills (Dixon et al., 2019; Herrero & Lorenzo, 2020). Research suggests that of all applications of XR technologies for autistic people, the technology is most effective at promoting daily living skills (Karami et al., 2021). Yet immersive interventions to promote daily living skills for autistic adults are remarkably limited.

Researchers and practitioners in the field of learning/instructional design and technology (collectively “LIDT”) recently have become increasingly aware of the phenomenon known as learning experience design (LXD). LXD is defined as “a human-centric, theoretically-grounded, and socio-culturally sensitive approach to learning design, intended to propel learners towards identified learning goals, and informed by UXD [user experience design] methods” (Schmidt & Huang, 2021, p. 141). A hallmark of LXD is the use of participatory design approaches in the development of learning technologies, which has been shown to enhance relevance, quality, validity, sensitivity, and practicality of research (Israel et al., 2005), and to accelerate the pace of discovery and development of clinically meaningful interventions in the area of autism in particular (Yusuf & Elsabbagh, 2015). Participatory LXD approaches allow for iterative improvements over time based on participant input and therefore represent an attractive approach to XR intervention design for autistic people.

Project PHoENIX utilizes participatory design within an LXD frame to intentionally locate autistic people, their caregivers, and their providers centrally in the processes of immersive technology design and development, as well as research design and execution. Known areas of training needs for autistic people include employment, transportation, accessing services, and functional and independent living skills. These needs align remarkably well with known affordances and benefits of immersive technologies (Stokes, 1997); however, they are under-researched (Harris et al., 2021a). Project PHoENIX seeks to establish design precedent for advancing the field towards more inclusive, human-centered, and neurodiverse paradigms for immersive technology R&D with autistic people.

Literature Review

Over the past two decades, a growing body of research has investigated the use of virtual reality (VR) technologies as a medium to provide various treatments, vocational support, and training for autistic individuals. These interventions were developed to teach a range of life skills, including emotional skills and daily living skills, as well as social and communication skills (Adjorlu & Serafin, 2018; Mesa-Gresa et al., 2018; Newbutt et al., 2017). Such interventions are often delivered through various devices, including (1) desktop-based systems or screen-based VR, (2) projection-based systems, (3) Cave Automatic Virtual Environments (CAVE), and (4) head-mounted displays (HMD). For example, Ke et al. (2022) evaluated the use of a virtual world social skills learning environment for seven (7) autistic children over 16–31 sessions lasting 0.75 to 1.25 h each. Using Open Simulator, the authors constructed their virtual world with various play- and design-oriented social interaction tasks. Data were collected pre- and post-intervention using the Social Communication and Skills Questionnaire and screen recordings, and observations were collected during the intervention. Findings suggest that social skills performance increased from baseline to post-intervention. Zhao et al. (2018) found a similar pattern of results in a study in which they developed a novel collaborative virtual environment (CVE) social interaction platform for autistic children. Their desktop-based VR allowed two children to play a series of interactive games using a naturalistic approach by coordinating their hand gestures to complete tasks collaboratively. Additionally, participants were able to share information and discuss game strategies using gaze and voice, which helped to promote communication and cooperation among users. The CVE was tested with 12 autistic children and 12 typically developing peers. Results suggest that this CVE was well-accepted by users and that cooperation in play gradually improved. Researchers concluded that the CVE has the potential to recreate and foster authentic communicative experiences.

Other researchers have focused on using CAVE as a medium to provide interventions, as this technology can allow for greater fidelity and interaction with objects and avatars in virtual scenarios (Yuan & Ip, 2018). For instance, in a randomized controlled study conducted by Lorenzo et al. (2016), researchers designed an immersive virtual reality system (IVRS) focusing on emotional skills for 40 autistic children using social stories. Results suggest significant improvements in the children’s emotional behaviors in the real school environment after IVRS training. Further, Tsai et al. (2021) conducted a two-phase study over a period of 5 weeks aimed at improving emotion recognition in autistic children through a CAVE-like system. In phase 1, participants worked with traditional figure card emotional recognition, and in the second phase, they entered the 3D-CAVE to engage with interactive games. Through third-person role playing in the immersive virtual environment, results suggest participants experienced substantial growth in their ability to recognize and understand six basic human emotions compared to baseline levels.

Recently, research interest in HMD-based VR for autistic individuals has been increasing steadily, partly due to the affordances of HMD, as well as the emergence of more affordable HMD such as the Meta Quest 2. In a study conducted by Almazaydeh et al. (2022), researchers examined the effectiveness of a new VR-based learning environment designed to enhance daily living skills, especially street crossing skills and social attention in a sample of nine autistic children aged 8 to 11 years. Using Wilcoxon’s signed-rank test to assess the change in each child’s skills compared to a baseline, results suggest significant improvement, supporting the promise of VR-based learning interventions targeting daily living skills. With similar motivations, Adjorlu and his colleagues (2017) developed a head-mounted display (HMD) VR-based simulation of a supermarket to teach everyday shopping skills to autistic adolescents. The experiment started with a pre-measurement and ended with a post-measurement in a real supermarket. After seven VR training sessions, the researchers compared the treatment and the control groups. The results suggested positive effects of HDM-VR simulations to train independent skills in autistic children. Further studies have been conducted with the aim of evaluating the effectiveness of HMD-based VR in addressing the difficulties that some autistic people have regarding social and communication skills. For example, in a single case multiple probe design study, Cheng et al. (2015) investigated the effectiveness of a HMD-based 3D virtual environment to enhance social skills of three autistic children. Over a 6-week period, participants were involved in a series of virtual social scenarios in which they answered multiple-choice questions assessing their nonverbal communication, social initiations, and social cognition. Preliminary results suggest that participants’ understanding and ability to initiate social overtures were enhanced after training.

Although the aforementioned studies provide evidence in support of the effectiveness of various VR technology-based interventions for autistic people, the majority of research in this area is largely technocentric; that is, researchers tend to refer all questions and solutions to the technology itself, and not to the action possibilities that the technology affords (Schmidt & Glaser, 2021a, b). Virtual reality interventions can be experienced through a variety of low- to high-tech devices, which present different levels of immersion (i.e., low, moderate, and high) and degrees of realism, meaning that the ecological validity of a VR intervention and the generalizability of a learned skill can be influenced by which type of VR device is used. Further, in the realm of VR for autistic users, understanding is lacking regarding “which technologies work for whom, in which contexts, with what kinds of support, and for what kinds of tasks or objectives?” (Parsons, 2016, p. 153). This is seen as particularly problematic given the well-known complexity of VR design in general, which is exacerbated when designing for a remarkably heterogenous population—an issue that has been referred to as a “wicked problem” (Schmidt & Glaser, 2021; Parsons, 2016; Schmidt et al., 2019).

Based on the current literature, very few scholars have proposed guidelines when approaching the design and development of VR interventions for education (Dalgarno & Lee, 2010; Fowler, 2015; Fracaro et al., 2021; Johnson-Glenberg, 2018; Mulders et al., 2020). For example, in their framework of learning in three-dimensional virtual environments, Dalgarno and Lee (2010) identified two distinguishing characteristics of this technology that can impact learning, which are “representational fidelity” and “learner interaction”. Representational fidelity refers to the quality of the display as well as to the degree of realism offered by the 3D objects and scene content. On the other hand, the learner interaction component refers to a user’s ability to construct their own virtual space or object and to engage and experience the richness of VR learning environments. However, Fowler (2015) argued that representational fidelity and learner interactions are technical affordances, but a clear perspective about how VR applications can lead to optimal learning opportunities for students is needed. Therefore, he extended Dalgarno and Lee’s model by adding pedagogical input and a design emphasis, which could guide practitioners to design VR experiences that meet the intended learning outcomes. His expanded model recognized three necessary aspects that must be considered to effectively use VR for training and teaching, namely, conceptualization, construction, and dialogue. In the conceptualization stage, learners gain a basic understanding of the concepts and what needs to be learned. Then, in the construction phase, learners engage in constructivist activities by beginning to explore, manipulate, or ask questions. Finally, learners deepen their knowledge through interaction or discussion with others. As Fowler argued, it is important when designing VR systems for learners, whether neurodiverse or neurotypical, that the benefits of learning be first articulated and motivated by the pedagogical requirements of a given learning experience rather than the possibilities that this technology can offer. In addition, perhaps most notably, Parsons and Cobb (2014) proposed a learner-centered design approach called the “triple-decker sandwich model” (3 T) to support autistic people. This model aligns three principles, which are (1) theory (i.e., top-down insights derived from research evidence), (2) technologies (i.e., specific affordances of the technology that support learning and interaction), and (3) thoughts (i.e., perspectives of the members of the autistic community). This model was applied by Parsons (2015) to illustrate how the 3 T factors informed the design of a novel collaborative VR environment called Block Challenge to promote collaborative and communicative reciprocity between autistic children. Findings suggest that by using the 3 T factors, Block Challenge was a useful tool to help promote reciprocal social communication and perspective-taking. While this model may inform the development of VR interventions, questions such as how to effectively design immersive learning experiences for autistic individuals and what are the best practices for designing VR interfaces for this group remain unanswered. Therefore, for researchers in this area to better explicate the parameters and principles that underscore the design of their VR interventions for autistic people (Glaser & Schmidt, 2022), further research is needed.

Based on the aforementioned issues, project PHoENIX adopts a paradigmatically divergent approach to designing our VR spaces that largely eschews dominant perspectives. This paper aims to detail the learning experience design of our VR environment for conducting research with autistic adults.

Project Description and Design Outcomes

Project PHoENIX has developed a framework of novel methods and processes for supporting co-design and collaborative research with autistic stakeholders in XR. Autism can severely influence independent functioning, exacerbate employment problems, and lead to social isolation (Eaves & Ho, 2008; Hedley et al., 2017; Müller et al., 2008). XR technologies can support autistic people with these challenges. Among XR technologies, virtual reality (VR) has received significant attention (Durkin, 2010; Strickland, 1996). For this reason, Project PHoENIX seeks to foreground the voices of the autistic community in advanced technology R&D efforts (Schmidt et al., 2021). Project PHoENIX approaches this issue with a focus on a critically under-researched group of autistic people—those who are transitioning from secondary education into adult life—in a specific context, support programs for autistic adults. The project has two outcomes: an iteratively co-designed and a co-developed VR environment that is sensitive to autistic stakeholder needs and preferences (project outcome 1) and novel methods and processes for engaging in co-design and collaborative research with autistic stakeholders within a multi-user, online VR environment (project outcome 2).

Project Outcome 1: Participatory-Designed and Developed a VR Environment

We have performed research with autistic stakeholders within the VR environment itself. In support of this, we developed this environment in collaboration with autistic stakeholders, actively involving them throughout the process. This was built on activities starting in Jan. 2021, when the principal investigator began co-designing and developing a prototype VR platform in collaboration with autistic stakeholders from a local, university-based autistic support center. The result is a VR space suitable for conducting further co-design and research that not only has been built by and for autistic adults, but also is highly usable, accessible, and sensitive to the needs and preferences of these individuals.

Project Outcome 2: A Framework for Co-Design and Collaborative Research with Autistic People

A handful of studies have engaged in real-world participatory design of VR environments with autistic people (e.g., Benton et al., 2012; Parsons et al., 2011), and a single project has been conducted with autistic children within VR (Roper et al., 2019). Preliminary guidelines and methods have been developed for in-person participatory design activities (Frauenberger et al., 2011; Millen et al., 2011), but whether and how well these will work in VR—and specifically for adults—remains unknown. For this reason, we began preliminary work in this area in Jan. 2021 and have completed a rigorous product review of low-cost, common, off-the-shelf VR tools that can support a broad range of activities for autistic adults. We have integrated technology tools, research methods, design processes, and user needs into an operable framework for supporting co-design and research with transition-aged autistic participants. To this end, we conducted a series of user-centered design cycles with autistic stakeholders in VR to (1) design and evaluate a range of co-design activities that our VR environment can support and (2) align these activities and technologies to the needs, values, and preferences of participants. The result of this is a preliminary framework that allows us for conducting inclusive research and supporting autistic adults within VR spaces in a way that minimizes known adverse effects associated with using VR (Schmidt et al., 2021).

The PHoENIX Design Process

The design of Project PHoENIX unfolded across three design phases. Phase 1 describes the VR software internal product review process that was performed, and the scoping activities carried out by the initial VR space design team such as empathy mappings. Phase 2 explains the refinement and evaluation process of the VR space including how the team collaborated at a distance, iterative refinements of the design space, and usability testing sessions. Phase 3 sought to pilot our methods and processes for conducting focus groups within the VR space with autistic users. Through these three phases of continuous revision and improvement, the design team gained an increasing understanding of autistic user needs, preferences, and values (Fig. 1). All research performed was approved by our university’s institutional review board.

Fig. 1
figure 1

Project PHoENIX design phases

Full size image

Study Participants

A total of 13 participants engaged in this study across the three design phases, with four in phase 1, five in phase 2, and four in phase 3, respectively (see Table 1). All participants in this study self-identified as autistic. Details about formal autism diagnosis were not collected for several reasons. First, specific diagnostic information was not relevant to our study. Second, asking for a formal medical diagnosis might have excluded those who have a school-based diagnosis or those who are self-diagnosed. Finally, the limitations of autism diagnoses, such as underdiagnosis of autistic girls and women, also played a role in this decision. Therefore, in this study, we did not collect data on formal autism diagnoses and instead relied on self-reported identification as being on the autism spectrum.

Table 1 Research participants involved in each design phase
Full size table

Phase 1: Product Review and Initial Prototyping with Autistic Users

To establish the design requirements and scope of the project, a rigorous product review was performed alongside scoping activities that applied LXD methods to bound the problem space. We describe these activities in the following sections.

Product Review

The PHoENIX VR environment was designed to present examples of different kinds of commercially-available, off-the-shelf VR software tools to autistic participants to help focus discussion. To identify high-quality examples of these VR software tools, the team engaged in a rigorous VR software product review. To this end, three team members conducted structured and unstructured searches to identify and evaluate 2–3 promising VR tools per week. Team members presented findings weekly and discussed specific features and drawbacks of each software tool. Through multiple discussion sessions, the product review team established consensus on which of the identified tools would be more formally evaluated. To inform our evaluation, VR software was tested directly. However, not all software tools were available to evaluate on a trial basis, meaning that the product review team had to glean information from product websites, promotional videos, and online reviews. The information found online regarding VR tools sometimes lacked sufficient details, which led the team to exclude these tools.

In Spring of 2021, the product review team evaluated all included software packages.

First, all VR tools were categorized into three main types: (1) educational single-user VR, (2) 360 video-based VR, and (3) social VR. The team then developed a VR software rating system using a rigorous process that spanned multiple iterations. The result of this was a product rating system consisting of both subjective and objective criteria (Table 2). Finally, the team used this product rating system to evaluate a total of six educational single-user VR software tools, six 360 video-based VR software tools, and 20 social VR software tools. Figure 2 illustrates output as radar charts for the three different types of VR tool that were included in this evaluation effort. The information gleaned from these product reviews was summarized and incorporated into a VR gallery space, where autistic participants were able to review the various tools and associated information, as well as to discuss the tools from the perspective of our research questions.

Table 2 VR software product review criteria, descriptions, and rating scales
Full size table
Fig. 2
figure 2

Example product review results for educational single-user (left), 360 video-based (middle), and social VR (right) software tools

Full size image

Scoping Activities

Scoping activities were carried out by initial VR space design team and were guided by the question of how can a VR environment be designed to promote the inclusion of autistic people in the research and development of VR systems that are designed for them? This question was predicated by the mismatch between the researchers’ agendas and the kind of research autistic people value (Parsons et al., 2020) and the need for more co-constructive and inclusive design practice in this area (Nind, 2014). Our process consisted of four primary scoping activities: (1) empathy interviews, (2) development of personas, (3) evaluation of personas and the initial prototype of a VR space, and (4) refinement and evaluation of the VR spaces to support focus groups with autistic people.

Empathy interviews involve interviewing individuals to gain a deeper understanding of their experiences, emotions, and perspectives related to a particular topic or situation. The goal of empathy interviews was to put the design team into the shoes of the interviewee and understand their perspectives as fully as possible. Empathy interviews are often used in user-centered design approaches to better understand the needs and perspectives of the target audience and to inform the development of products or services that are more responsive to their needs. In our empathy interviews, interviewers asked open-ended questions and actively listened to the responses, seeking to gain a deeper understanding of the interviewee’s experiences, emotions, and perspectives. Empathy interviews were conducted with four autistic users following a pre-designed protocol (see Appendix), with a goal to gain a deeper understanding of autistic users’ lived experience with technology in general and VR technology specifically. Grounded in the empathy interview results, the initial VR space design team performed empathy mapping and further developed four personas aimed at representing the autistic users in this study. To promote the inclusion of autistic users in every step of the design process, the design team created an initial rapid prototype of a VR environment, within which we presented the four personas in both poster and video modalities (see Fig. 4). Upon completion, the design team invited the four autistic users to experience the VR environment and to provide feedback and suggestions on both the personas and the design of the VR space while they were in the VR space.

The four personas were co-designed with autistic participants. The design team used what was learned from the empathy interviews to create a draft set of personas and then invited four autistic participants to critique the personas and provide feedback inside of the VR space. The critique sessions took the form of an interview. In each interview session, participants were introduced to the critique activity, what they were expected to do, as well as the technological tools and systems they used in the session. After joining the VR space, participants were asked to interact with the designed VR system and then answer a series of questions. Example interview questions included: (1) What do you want to improve on the design and presentation of the personas? (2) What do you like? and (3) What do you not like? The design team then refined the personas and modified/expanded the VR space after each interview session.

As the final step of the initial design phase, multiple autistic users visited the VR spaces simultaneously and provided another round of feedback and suggestions on both the personas and VR environments. Below are screenshots of the developed persona example and the VR environments that the initial phase of Project Phoenix designed and developed (Figs. 3 and 4). The four personas were used throughout the multi-phase project to help the team design and develop from autistic users’ perspective. Many of the VR environment design features and autistic participants’ design feedback and suggestions in the initial phase were instantiated in the following design iterations.

Fig. 3
figure 3

An autistic persona that was developed based on empathy interviews with autistic participants

Full size image
Fig. 4
figure 4

Prototype VR spaces for conducting research with autistic people: (1) persona presentation and evaluation and (2) space for conducting multi-user focus groups

Full size image

Phase 2: VR Environment Refinements and Evaluation

Phase 2 focused on initial ideation and rapid prototyping within the design team. The lead researcher and five research assistants iteratively designed low-fidelity prototypes based on the personas and prototype VR space that we had usability tested with autistic users in phase 1. A range of LXD methods were used to work in this ill-structured design space to move from initial ideation to a functional prototype, such as brainstorming, concept mapping, user journeys, and storyboarding. Design artifacts and design team journals were collected as data in this phase. Design efforts during phase 2 resulted in a minimum viable product (Lenarduzzi & Taibi, 2016) in the form of three low-level prototypes suitable for evaluation using cognitive walkthrough methods. These prototypes included (1) a VR training space that taught users how to navigate and use the various features of Mozilla Hubs, (2) a virtual gallery space in which users learned the history of VR for autistic people, and (3) a second gallery space in which users learned about and rated a range of different commercial, off-the-shelf VR software tools.

Concept and Ideation Process

After completing the VR software product review and getting familiar with the Mozilla Hubs software, the research team began to explore how a VR space can be designed so as to be sensitive to the needs, preferences, and strengths of autistic users. Therefore, mapping the needs and strengths of autistic users to the affordances of VR technologies was needed, as detailed in Antonenko and colleagues (2017). The team met to map out the matrix of needs and abilities to the affordances of VR that have been identified by Bozgeyikli et al. (2018) as being particularly promising for autistic people. A draft needs-affordances matrix was created that sought to connect autistic users’ needs and abilities with VR design recommendations so as to guide the PHoENIX VR environment’s task design, information architecture, and message design.

Beginning in May 2021, the team pursued an agenda of (1) co-designing and developing a prototype VR platform in collaboration with autistic stakeholders, (2) exploring a range of co-design activities and technologies to support autistic people, and (3) designing a VR space suitable for engaging in co-design and research within a VR space with autistic people. To these ends, the team met on a daily basis to map the functionalities of the Mozilla Hubs environment such as avatar personalization, navigation, chat messaging, and teleporting. The team also explored VR design precedent by playing and discussing commercial, off-the-shelf VR games. These design explorations were accompanied by questions of research logistics, including (1) how computers would be set up in different locations, (2) what possible VR activities would be conducive to our research questions, (3) how training activities could be simplified and made easy-to-use and accessible, and (4) how team roles and rules should be assigned to build VR activities with a team who has no prior experience designing in VR. In addition to this, we were concerned with how we would control for issues of motion sickness and/or cybersickness, as this is an issue that has been identified in the literature as a particular concern for individuals with sensory integration disorders (Glaser & Schmidt, 2022; Schmidt et al., 2021). Mozilla Hubs is relatively constrained in terms of features. Primary features are focused on communication (chat function, draw function, bring in 3D model figures) and navigation (keyboard including arrow functions), with limited features for manipulating objects or none for triggering events. Therefore, to explore the action possibilities of these features, each team member would build out small tasks and then present and discuss these prototypes with the team using a rapid prototyping approach.

Team Building at a Distance

Brainstorming began in early August 2021. Due to the COVID-19 pandemic, establishing a team (many of whom had never met in person), scheduling meetings, and collaborating remotely (with none of the team members having worked on a design research project before) were remarkably challenging. The team regularly met using Zoom, but also met a few times in-person in the studio wearing masks and social distancing. One of the most challenging issues was establishing a shared understanding and vision of the project. The team engaged in multiple meetings and discussions, but developing a shared understanding of critical aspects of the project remained elusive, such as concerns for vulnerable participants, complexities of VR design, ambiguity, and lack of design precedent in this space, how design research is an ongoing (not summative) process, and complexities of VR logistics, procedures, and implementation. After experimenting with multi-user collaboration tools such as Google Docs and Google Sheets, we learned that shared whiteboards were remarkably effective at organizing brainstorming sessions, sharing collective understanding, and identifying gaps, tensions, and friction. Figure 5 below illustrates how the lead designer facilitated a brainstorming session by first asking team members to write out any questions on sticky notes, then having the team organize those questions using affinity mapping techniques, then labeling the different groups of sticky notes, and finally prioritizing the groups.

Fig. 5
figure 5

Brainstorming session for project PHoENIX using shared whiteboard tool Miro

Full size image

Rapid Prototyping

Design tensions emerged towards the end of the summer of 2021 in that team members’ prototypes were not improving over time, but instead were developed, reviewed, and then abandoned. Many of the designs that the team members came up with were abandoned due to the process of several iterations. For example, there was one environment prototype that seemed like a complex conference hall with multiple rooms including stairs; however, given the logic of the user movements within the space and the study protocol were misaligned, the initial VR environment prototype needed to be abandoned. This trial-and-error approach was productive for team members to learn the ins-and-outs of the Mozilla Hubs platform, but was not advancing the design of the underlying PHoENIX VR environment. The lead designer surmised that this was in part due to an established lack of design precedent in this space (Schmidt & Glaser, 2021a, b), but also due to a lack of design expertise on the part of the team.

As a result of this tension, the team pivoted in September 2021 away from exploratory methods and towards more intentional design and project management methods. The lead designer conducted a workshop in the design studio in which the design team learned how to work in an agile project management design environment. A second workshop focused on how to engage in productive design critiques so as to more efficiently advance the design of the PHoENIX VR environment using rapid prototyping methods. Following this, the team created a backlog of user stories and organized them into a series of design sprints. Virtual daily stand-up meetings allowed the team to share what they had been working on, and what they were currently working on, and to openly discuss any issues or “blockers.” More structured and consistent design critiques led to productive failure, identification of design flaws, and iterative improvement of the VR spaces. Figure 6 below illustrates the iterative rapid prototyping process for designing one of the VR gallery spaces.

Fig. 6
figure 6

Incrementally improving virtual gallery space designs, created using Miro

Full size image

The changes made to project management and to performing critiques ultimately led the team to develop a shared understanding and focus, which accelerated design and development substantially and led to more effective designs. Examples of this with the VR training space include (1) consistently improving clarity and consistency in instructional message design, (2) incorporation of progress indicators, and (3) standardizing the shapes and sizes of the virtual geometry in the VR space. Examples of this with the virtual gallery spaces include (1) incorporating navigational aids such as maps that allowed the user to select a room and teleport there instead of navigating their avatar; (2) adding media boards that allowed users to add their own media to the VR space such as selfies, animated GIF memes, and text messages; and (3) grouping learning content into various “stations” within the gallery to segment the materials logically and allow for sequencing activities. After just 1 month, the team was able to invite an expert on VR and autism and another expert on VR for education to perform a heuristic evaluation of the prototype spaces and provide recommendations for improvement. Their feedback suggested that the prototype VR spaces were user-friendly and showed promise, but were in need of refinements such as (1) making the spaces feel more inclusive for autistic users, (2) improving loading times, and (3) making improvements to accessibility.

Data-Based, Iterative Refinements

Several revisions were incorporated in October 2021. We approached the issue of making our spaces more inclusive by adding original artwork created by autistic people and including quotes from autistic people about their experiences with VR on the walls. We made accessibility improvements by increasing image resolutions, making our content presentations more hierarchical and systematized, slowing the narration speed of videos, fixing issues with sound overlaps and loud noises, simplifying the language used in messaging, and adding more instructions to the VR training space. Loading times were optimized by using smaller-sized files and streaming videos versus hosting locally and removing any unnecessary 3D models.

As these changes were being incorporated, the team also created the second VR gallery space focusing on commercial, off-the-shelf VR software tools. Design challenges included (1) how we should showcase different VR software, (2) how we could design ways to rate the software and where it should be placed, and (3) how we should sequence activities, i.e., rating software immediately after users have reviewed it versus after all software is reviewed. Multiple iterations led to consensus and a reusable template for how to present and rate VR software tools. We then populated the space using this template and invited a different expert on VR and autism to conduct a heuristic evaluation of our VR spaces. This expert provided encouraging feedback and suggestions for improvement. These recommendations were incorporated during November 2021, after which the team conducted an internal usage test (Fig. 7) focusing on what worked well, what did not work well, and what needed improvement. Further refinements were made based on this usage test, after which a usability test was conducted.

Fig. 7
figure 7

Screenshots from internal usage testing with multiple users

Full size image

Usability Testing

A usability test was performed with five autistic adults in December 2021 with the purpose of evaluating the usability of the designed spaces and refining them based on issues identified by participants. Convenience sampling was used when it comes to participant recruitment. The five autistic adults were recruited through an autism center at a large university in the southeastern region of the USA. Five white males were in the usability testing session. Each testing session was performed with one autistic participant online and at-a-distance. Screen video and audio recordings, field notes, a self-developed social validity instrument, and the Computer System Usability Questionnaire (Sauro & Lewis 2016) were used for data collection. After each session, the research team debriefed on identified usability issues and assigned a severity level (0–4) to each of them (Nielsen, 1994). Issues with a severity higher than 3 (i.e., usability catastrophes) were addressed before the next testing session. Of note is that no high-priority usability issues (severity higher than 2) were revealed after the first four participants (see Fig. 8).

Fig. 8
figure 8

Usability study-informed iterative improvements to the PHoENIX VR software review gallery showing how the rating system was moved from a stand-alone object (1) to an in-line element that flows with the sequence of the learning experience (2)

Full size image

We provide a brief overview of the usability study findings here. The mean score on CSUQ from all five participants who engaged in phase 2 usability testing was 1.98 (sd = 0.53). This suggests above average overall perceived usability according to benchmarks from Sauro and Lewis (2016). In qualitative analysis, descriptive quotes from participants suggest they perceived their experience thus: “Cool, I liked it”, “smooth and effective”, and “I liked seeing different programs.” Additionally, participants stated that “VR is the future”, “[VR] helped reduce anxiety”, and “this software can be used for kids with autism or other developmental challenges”. Preliminary analyses suggest PHoENIX has high acceptability, satisfaction, and relevance for the target population and that participants greatly valued that it was a product designed with autistic people.

Phase 3: Pilot Testing

In phase 3, pilot testing was conducted to evaluate the capability of the VR spaces to support multi-user focus group research. During this study, four autistic participants engaged in a pilot of one complete focus group. The pilot test was performed online, at-a-distance, and within the designed Project PHoENIX VR spaces (i.e., training space, VR for autism gallery, and VR software review and rating gallery). Data collection methods used in this phase were identical to those used in phase 2 usability testing. For the pilot testing session, there were 3 white males and 1 white female participating as a focus group.

We provide a brief overview of the pilot testing findings here. Qualitative methods such as video analysis and thematic coding were used for data analysis. Analysis procedures followed techniques outlined by Hsieh and Shannon (2005) for conducting a directed qualitative content analysis. To analyze quantitative data from the CSUQ and social validity instruments, descriptive statistics were computed, and these were used within a multi-methods frame to support the qualitative findings. To analyze video recordings from previous sessions, four researchers engaged in the coding process and all videos were coded in dyads. A coding scheme with reference to usability characteristics defined in the product quality model (ISO/IEC 25010, 2011) was adapted, and results were recorded in a shared Google Spreadsheet. In order to control coder drift, the two coding teams coded one 10-min clip every week and sent their results to the lead researcher for quality assurance. Gwet’s AC1 coefficient (2008) was computed in R (R Core Team, 2022) with γ = 0.77, which can be considered as “Good” agreement using Altman’s (1999) criteria or “Substantial” agreement based on Landis and Koch’s (1977) criteria. The mean score on CSUQ from all five participants who engaged in phase II usage testing was 1.45 (sd = 0.57), again scoring above the average benchmark for perceived usability, and outperforming the version used in the phase 2 usability test. Results from video analysis confirmed the improved performance of PHoENIX, as the group described PHoENIX as highly relevant and user-friendly. Ultimately, phase 3 pilot testing revealed that our VR environment was highly usable, remained stable with multiple participants, and was able to support all focus group activities we had designed. Excerpts that supported this finding include “It’s nice to know we are being thought of”, “What VR helps with is that it’s not a place where people would be watching you specifically, because you can mess up all you want in VR”, “[T]he society is probably a bit more overall accepting of autistic behaviors. The more projects there are like this, the more people that know that autism is a thing. Have a positive technology actually feed the positive change”, and more. On this basis, the software is now considered to be in a final release state and ready for deployment for a series of focus groups with autistic stakeholders, caregivers, and providers.

Discussion

The present study aimed to develop the Project PHoENIX environment for autistic people using a comprehensive design process that involved three phases. The overarching goal was to create an immersive VR space that is sensitive to the needs, preferences, strengths, and abilities of autistic users. Our study employed a range of best practices in human-centered design, including usability testing, iterative design, and rapid prototyping, to ensure that the VR environment was optimized for its intended user group (Nind, 2014; Parsons & Cobb, 2014).

The results of our study demonstrate that the PHoENIX VR environment is highly usable, relevant, and user-friendly for autistic users. The mean score on CSUQ from all five participants who engaged in phase II usage testing was above the benchmark for perceived usability. Furthermore, the qualitative feedback from participants indicated that they found the VR environment enjoyable and that it could be a useful tool for reducing anxiety and promoting positive change. The pilot testing phase also confirmed that the VR environment is stable with multiple users and can support all focus group activities as designed.

Our study contributes to the limited research on co-design and inclusive design practices in the field of VR for autism, as among a handful of studies that report co-design or inclusive design with autistic participants (Francis et al., 2009; Giaconi et al., 2021; Magkafa et al., 2021). This approach to co-designing with the intended user group allowed the creation of a VR environment that is more sensitive to the needs, preferences, strengths, and abilities of autistic individuals, and is thus more likely to be effective in engaging and empowering them in research and co-design activities. Moreover, our approach to co-designing with individuals with autism is noteworthy, as it required the adoption of a human-centered learning design lens, which sought out human perspectives and feedback in all steps of the design process (Frauenberger et al., 2011; Millen et al., 2011). By establishing co-design methods in a VR modality, and utilizing user experience design methods, we were able to create an effective, efficient, and satisfying experience for users.

While our study presents promising results, some limitations should be noted. The small sample size of participants who engaged in the usability testing and pilot testing phases is a significant limitation. However, the use of a rigorous and multi-method approach to data collection and analysis provides confidence in the findings. Furthermore, the limited number of VR features available in Mozilla Hubs constrained the design possibilities and affordances aligned around realism for the PHoENIX VR environment (Dalgarno & Lee, 2010). In addition, the participants only experienced the VR desktop-based environment and did not use a VR headset. The affordances of a VR desktop software based environment versus the VR hardware headset need to be considered in future research as well towards individuals with autism (Anthes et al., 2016; Kugler, 2021). Lastly, researchers might explore the use of other off-the-shelf VR platforms or technologies to develop more advanced and customizable VR environments for individuals with autism.

The present study has meaningful implications for the development of VR technologies that are inclusive and accessible for diverse user groups, including those with neurodevelopmental disorders such as autism (Francis et al., 2009; Giaconi et al., 2021; Magkafa et al., 2021). Our study highlights the importance of co-designing with individuals with autism to create VR environments that are more sensitive to their needs, preferences, strengths, and abilities. Moreover, our approach demonstrates how inclusion can be foregrounded in the delivery of service and training using extended reality technologies for a population that is in critical need.

Overall, our study presents much-needed design precedent for advancing the field towards more inclusive, human-centered, and neurodiverse VR research and development paradigms. The use of VR to design and perform research on VR for autistic people represents a promising direction for future research that is cost-effective, accessible, and fully digital. Our study offers insights into how the voices of neurodiverse participants can be foregrounded and celebrated in both the design process and the resultant design products.