Introduction

Autism and intellectual disability

Autism Spectrum Disorder (ASD) is described as a neurodevelopmental condition that is manifested by challenges in social communication and thinking and behaving flexibly (APA, 2013, 2022). Social communication difficulties involve challenges communicating with and without words, as well as challenges understanding the behavior of others (APA, 2013, 2022; Jones et al., 2001). The latest data reported by the Center for Disease Control and Prevention (CDC) indicate that in the US 1 in 36 children (2.76%) are autistic (Maenner et al., 2023). Recent systematic reviews on the global prevalence of autism indicate a somewhat lower prevalence between 1 and 100 children (Zeidan et al., 2022) and 1 in 166 (Salari et al., 2022). A relevant percentage of autistic individuals also have an intellectual disability (ID). According to Maenner et al. (2023), 37.9% of people with ASD also have ID. Intellectual disability, or intellectual development disorder, is characterized by difficulties in intellectual functioning and adaptive functioning (APA, 2013, 2022). Intellectual functioning is considered to include the abilities to reason, problem solve, plan, think abstractly, make decisions, and learn. Adaptive functioning refers to the capacities necessary to live independently and responsibly. According to a prevalence study by Patrick et al. (2021), 1.2% of children present with ID, of which 39% are autistic. In relation to supporting autistic people, digital technologies can provide an important opportunity to promote the autonomous functioning of autistic people, helping them to communicate, participate and lead an independent life. There are numerous arguments and studies that justify the use of digital technologies with autistic people (Zergovianni et al., 2023). One key justification relates to the ‘fit’ between the way technologies present information and the way autistic people prefer to process information, in addition to the possibilities of personalization and control of the learning situation.

Development of technologies for autistic learners

In the field of video game development, Al-Azawi et al. (2014) consider two archetypal development models, the predictive and adaptive models. Predictive models create work planning as a separate pre-development task and are preferred when the goals and customer requirements are clear and fully defined. In the case of adaptive models, the requirements and objectives are not completely clear, and the client or user can add new objectives and requirements at each stage of the project. In these cases, the process is based on prototypes, tests, and refinements. Each iteration includes analysis, design, implementation, testing, and evaluation. Predictive models can be very useful for non-flexible application areas, but highly inadequate for creating technology solutions for autistic people (Herrera et al., 2018). Adaptive models are also most useful when the backgrounds of the participants vary greatly from one to another, as occurs in a multidisciplinary design or participatory design process (Benton et al., 2012). This is because not everyone involved can predict the consequences of decisions made during design and development. Adaptive models also facilitate the refinement of tools to increase their usability (Herrera et al., 2018). However, the work of Herrera and colleagues, conclude that there are no quality standards for the development of VR/AR learning content for autistic people and that this lack of standards and methodologies can partially explain the scant scientific evidence-base available on the effectiveness of commercially available technologies for autistic users (Herrera et al., 2018; Kim et al., 2019). More recently, Zervogianni et al., (2020, 2023) have developed a framework for research with digital technologies for autistic groups called BETA (Building Evidence for Technology in Autism). A Delphi study was used to create the BETA framework to develop a consensus on what constitutes good evidence for digital supports among the broader autism community, including autistic individuals and their families, as well as practitioners and researchers concerned with digital supports. In that investigation (Zervogianni et al., 2020), a four-phase Delphi study consultation was conducted with a 27-member panel, resulting in agreement on three categories for which evidence is required to support technology tools for autistic people. These are: (1) Reliability: Is it technically robust/functional? Works correctly?; (2) Ability to connect with the user: How usable, friendly, pleasant, and accessible is the product for the users for whom it is intended?; and (3) Effectiveness: What impact has the product had on the people who have used it? Does it make an observable difference in the user's life or behavior? Consensus was also reached on four key sources of evidence for these three categories: practical experience, academic sources, expert opinions, and online reviews. These categories were weighted differently as sources of evidence within the three categories described above (reliability, user connection, and effectiveness) to obtain an overall score on the level of evidence for a given product. Based on these parameters, a rating scale was developed and validated on a random sample of 211 studies drawn from the results of a systematic review of technology and autism (Zervogianni et al., 2023). The User Information-Based Design Scale (UIDS), which is one of the results of the BETA project, is used to assess to what extent the design of a technological tool has taken into account the knowledge available in the scientific literature on autism, to what extent the design has been based on real information and the experience of the end users, and to what extent the end users have been involved in the design.

Immersive virtual reality applications for autism

Immersive virtual reality (VR), as a programmable environment, has the potential of providing customized instruction and learning-support for autistic students. Immersive VR benefits have been investigated over the last three decades (Strickland et al., 1996). Initial studies tested immersive VR using heavy and wired head-mounted displays (HMD) with a reduced field of view and low-resolution images, together with wired sophisticated data gloves for hand-tracking. Even with those limitations the acceptance rate was good both in the seminal study of Strickland et al. (1996), where the two participants accepted the technology, and on a later study by Alcantud et al. (2002) where 39 out of 40 students accepted it. Nevertheless, the high cost and, in some cases uncomfortable nature of these technologies, have inhibited their general adoption in educational practice during this time. Schmidt and Glaser (2021) have studied immersive VR applications with autistic users in the context of complexity theory, arguing that “no single factor in such a complex ecosystem individually affects the learner experience” and that, instead, “it is the nature of the interaction between the interconnected and interdependent components that gives rise to the phenomenon of learner experience as an individually perceived unique phenomenon” (p. 1670). They describe the learner experience of autistic individuals in HMD-based VR interventions as a complex, interconnected, and interdependent system that includes but is not limited to, the designed intervention, learner-as-user, and learning context (Schmidt & Glaser, 2021).

Recent developments related to the relatively low-cost and high-quality of immersive VR HMD have helped to rapidly change the landscape of their use in autism education (Newbutt et al., 2020). Those devices, such as Meta Quest®, provide people opportunities to use them in standalone situations (i.e. wireless) and include hand-tracking sensors, which make them a priori more suitable solution than those available in previous decades. The level of immersion, image quality, and other features of this type of device have been compared with more basic forms of virtual reality by Parsons et al. (2017), such as those where a smartphone is inserted into a cardboard HMD, reporting a higher level of immersive experience with HMD despite the higher cost of those devices by that time. Other studies have tested cardboard or smartphone-based VR as an intervention tool to learn and develop critical practical life skills (McCleery et al., 2020) having found it safe, feasible, and usable for verbally fluent autistic users more than 12 years old without intellectual disability. The latter study did not include hand tracking (or hand manipulation) of objects by autistic participants. This is an inherent limitation of the currently available cardboard with smartphone VR technology option; together with a reduction of 20º in the field of view within the HMD. However, bespoke HMDs like Meta Quest 2® include the necessary depth sensors for hand tracking/manipulation and has recently been identified as the most accurate HMD when using head-tracking, compared with other HMDs of their type (Holzwarth et al., 2021). According to the International Organization for Standardization (ISO), usability refers to “the extent to which a product, a system or a service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO/IEC 9241–11., 2014, Bevan et al., 2015). Several studies have already provided initial positive findings of immersive VR HMD use amongst autistic users, specifically focused on feasibility, usability, and safety (Newbutt et al., 2016, 2017, 2020; Schmidt et al., 2021). In fact, these studies have highlighted intentional steps to ensure safety with these groups; in doing so report limited sensory problems or barriers to HMD use. Recent work (Malihi et al., 2020) has identified similar findings and concluded that immersive VR HMDs could be a suitable and safe technology for supporting autistic people. Technology usability should be a precondition for any technology-based investigations in autism, as it is something that may positively or negatively impact the magnitude of the intervention effect (Mazon et al., 2019). This is in consonance with the more general research on technology and autism where reliability, engagement and effectiveness of the technology have been identified as the most important categories where evidence is required (Zervogianni et al., 2020). Therefore, continued and further research is needed to demonstrate the feasibility, usability, and safety of these tools in educational settings with autistic students; especially for those who also have an intellectual disability (American Psychiatric Association [APA], 2022), as limited research includes autistic users with intellectual disability (Newbutt et al., 2016, being one example). This deployment of technology must be integrated into methodologies that promote a structured, safe, and highly usable environment where students on the autism spectrum with or without intellectual disabilities can learn independently.

The individual work system applied to autistic groups and students with intellectual disability

TEACCH (Treatment and Education of Autistic and related Communication Handicapped Children) is defined as an approach that supports the individual’s ability to learn, comprehend, and apply learning across situations (Mesibov et al., 2005). This program has been widely used with groups of autistic people in different parts of the world (Goin-Kochel et al., 2007; Green et al., 2006; Hess et al., 2008; Kielinen et al., 2002). The TEACCH educational approach is called Structured Teaching (Mesibov et al., 2005; Schopler et al., 1995) and is based on the evidence and observation that autistic individuals share a pattern of neuropsychological strengths and differences (Mesibov et al., 2005). In this context, the term “structure” describes the organization of time, space, and sequences of events within the environment to support autistic individuals understand and execute learning activities. Structured Teaching is not a curriculum but a framework within which educational goals can be achieved (i.e. to facilitate access to the curriculum).

Several review studies have been conducted about TEACCH (Mesibov & Shea, 2010; Sanz-Cervera et al., 2018; Virués-Ortega et al., 2013) with results reporting gains of different magnitude in different domains. As Mesibov and Shea (2010) highlight, it is problematic to demonstrate the effectiveness of a comprehensive program like TEACCH as this is formed of multiple components and mechanisms. As those components are deployed in natural settings, the measurement of individuals’ data about their performance and progress is more difficult than in laboratory settings, making research more difficult.

The individual work system (IWS) is one of the TEACCH components and is defined as a visually organized space where children practice acquired skills (Schopler et al., 1995). It is a work organization system that can be and is usually applied to individuals with autism spectrum conditions and/or intellectual disabilities. A work system visually communicates at least four pieces of information to the student: (1) the tasks the student is supposed to do; (2) how much work there is to be completed; (3) how the student knows they are finished; and (4) what to do when they are finished (or “what’s next”). The work system provides a structured opportunity for students to practice skills deliberately and independently. It can be considered a “low-tech” instructional technology, as it uses furniture, printed and laminated images, and Velcro® type or magnets to provide clear instructions to students so that they can learn and practice independently. At least four empirical scientific studies have been conducted (Bennett et al., 2011; Hu et al., 2019; Hume & Odom, 2007; Hume et al., 2012) and all of them have found positive results for promoting independent learning for autistic groups.

The individual work system may also promote students’ generalization of skills across settings. As a best practice in autistic interventions and its use, involves the application of three heuristics categorized by Stokes and Osnes (2016, cited in Schmidt & Glaser, 2021): (1) using elements of the natural environment that already function to maintain the target behavior, in our study this is the traditional IWS; (2) train diversely, refers to maintaining the minimal level of training control possible while still producing behavior change, with this being achieved through the co-design and use of a diverse range of activities contained in the trays of the IWS; and (3) taking advantage of relevant discriminative stimuli in the training environment that can be transferred to other environments to promote generalizations, which in our case involve using similar elements within the immersive VR environment to those they can find in the real world setting.

Potential advantages of immersive virtual reality

Immersive VR has the potential of overcoming some research difficulties identified in previous studies about TEACCH, such as those related to capturing research data in natural settings: virtual environments combine the benefit of being a simulation of reality and, at the same time, the possibility of automatically gathering and analyzing data on the student performance as in a laboratory setting. More specifically, the implementation of a virtual version of IWS has the potential of sharing the advantages of implementing it in a real setting (to promote independence, understanding and access to the curriculum) and has additional potential advantages. Similar to real IWS, a VR-IWS allows (1) customized instructions (being possible to personalize every single educational activity contained on a tray), and provides an opportunity for (2) accessing the curriculum in a format they can easily understand and manipulate and (3) provides means for independent learning, effectively using individuals’ strengths to overcome their challenges, promoting self-reliance and independence. However, these similar potential benefits of VR-IWS have not been developed or evaluated previously. Besides, there are other unstudied potential advantages of implementing a VR-IWS:

  1. 1.

    It is possible to drive the student's attention by augmenting the saliency of the elements with extra light or sounds;

  2. 2.

    It is possible to implement error-less training (i.e. not allowing putting a small ball in a big hole in a size-matching task);

  3. 3.

    It is possible to have full control of the sensory load of the environment;

  4. 4.

    It is possible to provide contingent reinforcers (attractive sounds or visual effects);

  5. 5.

    It is low-cost and scalable: educational activities only need to be created once and then can be used by thousands of students with no chance of being deteriorated or getting lost;

  6. 6.

    It can facilitate the transfer of the pedagogical principles to different cultures and territories, including those where teachers had limited access to qualified training;

Virtual reality and the individual work system

The authors of this article first developed a VR-IWS prototype in 2018 as a proof of concept with two autistic students with intellectual disability who showed a significant decrease in the percentage of time they received human support in VR-IWS compared to non-VR-IWS (Sevilla, et al., 2018). In that study, Fove VR Headset (that requires cables and an additional hand-tracking sensor) was used. Hand-tracking was considered important as it allows the manipulation of the VR environment with the users’ hands, so it is not necessary to use (and learn to use) any kind of controller, as they require specific motor skills, arbitrary motor sequencing skills and a higher abstraction level than using their own hands. When designing technology for autistic students, one size does not fit all, and educational solutions need to be fully customizable to each student’s needs. After this initial study, the authors started the co-design of a new, more complete, and fully customizable VR-IWS that could be used with Meta Quest® (wireless and equipped with hand-tracking sensors). In this co-design action, teachers from four schools across the UK, Spain and Turkey were involved. Students were indirectly involved as the activities reported by teachers were those that the students regularly developed autonomously (or mostly autonomously) at the schools. This was conducted to create a wide and rich database to reflect activities from different cultures and different skill levels. More than half of the children in this co-design process had an intellectual disability in addition to a formally assessed autism diagnosis. In each region, this differed, in terms of the process of diagnosis, but each school reported a formal diagnosis based on local standards. The school from the UK reported 45 activities from 5 students. One school from Spain reported 70 activities from 7 students. Another school from Spain was involved in the co-design and reported 81 activities from 9 students. The school from Turkey reported 9 activities from 9 students. In total, 205 activities were reported and analysed to find commonalities, skills, and themes that could be implemented in the virtual version of the IWS to connect with the interest and daily practices of the schools and autistic students involved in the project. This co-design process (Vera et al., in preparation) resulted in the definition of an immersive VR-IWS including a series of adaptations to facilitate access to autistic participants with intellectual disability. Several types of tasks were identified and implemented: (1) moving balls from one side of the tray to the other, (2) classifying objects according to different categories (size, colour, number, width, length, shapes and textures) or a combination of them, (3) associating different sets of objects according to semantic fields (different types of vehicles, different types of animals and differentiating between fruits and vegetables) and (4) half-and-half puzzles related to creating a figure based on a model, or more complex ones in which they should resolve simple mathematical operations (addition and subtraction). Figure 1 shows the immersive VR implementation of the IWS used in this study. Figure 2 highlights the interface and setting of the VR-IWS. Another requirement emerging from the co-design process was the need for adaptable sensory load in the VR-IWS (see Fig. 3).

Fig. 1
figure 1

immersive VR implementation of the IWS

Full size image
Fig. 2
figure 2

Interface and setting of the virtual reality IWS

Full size image
Fig. 3
figure 3

Levels of sensory load within the VR-IWS

Full size image

Objectives

The current study is a feasibility, usability, and safety trial examining the previously described virtual implementation of the IWS. Feasibility was tested to check if students could complete the activities independently, usability was tested to demonstrate that students find it easy to perform autonomous tasks within this VR setting, and safety was tested to check if any adverse effect was present when using the VR-IWS with an HMD.

This is the first study to analyze the feasibility, usability, and safety of a highly immersive standalone, equipped with hand-tracking and completely wireless new-generation HMD with autistic children, some who are younger than 12 years old, that included participants with and without intellectual disability.

Methods

Participants

Participants were recruited from three different special education schools in the UK (n = 7), Spain (n = 7) and Turkey (n = 7). Prespecified inclusion criteria included: (1) a documented diagnosis of autism (according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR) criteria (APA, (2022)) with or without accompanying intellectual difficulties; (2) being between 6 and 18 years old; and (3) willing to accept and wear VR glasses at first attempt (without the need for a desensitization procedure). Positive previous experiences with immersive VR in two of the three participating schools justified this latter inclusion criterion. Only one candidate participant from one of the schools was not included in the study for this reason, and he was granted access to the proposed immersive VR solution with a desensitization procedure afterwards. Exclusion criteria included having a personal history of seizures, migraines, or vertigo. A total of n = 21 autistic students were enrolled in the study: 3 females and 18 males aged 6 to 17 (mean age: 11 years and 8 months; standard deviation: 3 years and 6 months). Six participants presented an accompanying intellectual difficulty. The study was approved by the Ethics Committee of Research in Humans of the Ethics Commission in Experimental Research of the University of Valencia (Spain). As part of the ethical review process, every student had permission from their legal guardian/caregiver to participate in the study and signed a consent form for their participation. The teachers in charge of the application of the VR-IWS knew the students who participated. This was necessary due to the significant communication difficulties of students (they had to interpret what happened during the sessions and include that information in the corresponding record sheets). In addition, by knowing them so well, the teachers were able to advocate and ensure their safety and locate ways for them to withdraw from the study should they want to. This knowledge also meant they were able to create and prepare VR sessions according to the skills and themes of interest of each student.

As a usability study for an educational tool designed by/for autistic participants, including a comparison group of non-autistic participants wasn’t planned as they were not the target population of the tool. In the same way, as a usability study is not an intervention study and no intervention effects were measured, there was no need to establish a comparison or control group of autistic participants who were not involved in the usability testing.

Table 1 provides details of the age, sex, ethnicity, family income and whether participants presented an accompanying intellectual difficulty. Family income was calculated considering the median household income (MHI) for each country according to official data and establishing these scores: low (< 0,5 MHI), medium (0,5–1,5 MHI) and high (> 1,5 MHI).

Table 1 Participant characteristics for this study
Full size table

Materials

Immersive virtual reality head-mounted display

All three schools were equipped with Meta Quest 2® HMD and a tablet or TV screen where the VR contents shown to the participant were streamed to allow teachers to monitor the participant’s activity.

Immersive virtual reality tool

This was the software that has been initially described in the introductory section that implements a VR classroom containing a version of the IWS, based on three components: (1) an "input shelf" where the trays with the tasks to be carried out are located, (2) a "work table/area" where they have to be deposited to be carried out and (3) an “output shelf” where the tasks already completed will be stored. For the configuration of the sessions, the teacher had an interface that allowed them to customize some general aspects of operation (such as the level of sensory load, light-guided operation, or whether errors are allowed in performing tasks). They could also configure sessions (composed of a series of activities) taking into account the level of previous progress of the student. They had the ability to launch the sessions, after which they had to give the HMD to the student. The height and orientation of the user could be calibrated to the environment by the teacher, using the buttons and joystick of the right command/controller, to make it easier for the user to reach the different elements comfortably. Teachers could manipulate the different elements with controller devices, but students manipulated the virtual environment only with their hands. Finally, the teachers could analyze the session data and track student progress to adapt the contents of future sessions within the VR-IWS. The software is freely available in English, French, Spanish and Turkish (https://ivrap.adaptalab.org/outputs). Figures 4, 5, 6, 7 highlight the four types of activities included in the software.

Fig. 4
figure 4

Moving balls

Full size image
Fig. 5
figure 5

Classifying objects by a combination of categories

Full size image
Fig. 6
figure 6

Associating objects according to semantic fields

Full size image
Fig. 7
figure 7

Math puzzle

Full size image

Materials used to test safety, feasibility, and usability

Safety was measured to verify that using the VR-IWS does not generate any adverse effects. To this end, we included a series of precautions (exclusion criteria related to epilepsy, migraines, or vertigo), teachers were trained to only run between 3 and 5 activities in each session, and they were instructed to immediately stop the session if observed any indisposition on the part of the students (such as dizziness or signs of feeling sick). They also went through a series of checks after using the tool, with a post-session safety questionnaire that includes open-ended and close-ended questions about the feelings and safety of participants during and after the VR session (adapted from McCleery et al., 2020, provided in (Online Appendix 1)).

Feasibility was tested through the completion of the Tasks Tests for students, using the After Scenario Questionnaire (ASQ) adapted for this particular study (Lewis, 1995, included in (Online Appendix 2)). For each task within the VR session, this test uses Likert scales (from 1 to 7) to ask participants about the easiness, the time spent and the support information when performing a particular task. The participant’s success or failure when performing the task is also gathered.

Usability was assessed to identify interaction issues or any other necessary improvement to the software. Usability is a measure that has three components: effectiveness (verifying that the tool serves what it is designed for), efficiency (it allows it to fulfill its function in a time or with optimal resources) and satisfaction (the participants say they have had a good experience using the tool). The above-mentioned scoring of the tasks test was also used to measure usability at the task level (Online Appendix 2) and the System Usability Scale (SUS; Brooke, 1995) was used to measure the overall usability of the immersive VR tool (Online Appendix 3). The SUS scale is a 10-item questionnaire with ratings from 1 to 5 that asks participants about their general feelings when using the tool (i.e. if they found it easy to use or if they think that previous training is required before using it).

Procedure

Teachers received training from the researchers on the use of the VR tool to enable them to test all the components. They received the training across two different training sessions on two different days. In addition, they were trained on the use of the measures for feasibility, usability and safety described above (Online Appendices 1–3). The teachers were asked to install the software onto a Meta Quest 2® HMD and to set up another screen where the images being viewed by the user (via HMD) were streamed in real-time. They were asked to test everything prior to each VR session.

Students were requested to participate in at least one VR session (n = 21). Some of the participants participated in two (n = 8). The length of each immersive VR session was determined by the number of activities to be developed within the immersive VR environment and the efficiency of the students developing them.

Students were given the HMD with the VR session previously prepared by their teacher. Students were asked to do the activities seated on a chair, but some of them preferred to stand-up. The activities were selected by the teachers according to their skills. For example, those with lower skills were asked to perform “move” tasks in which they only have to place the balls in the hole. Those with a higher level of skill development performed “half and half” puzzle tasks that involved addition and subtraction and knowledge of numbers.

Teachers monitored the session and once finished, they asked participants to complete questions about security (Online Appendix 1), a task test (Online Appendix 2), and the SUS test (Online Appendix 3) together with some open questions about the session. Teachers adapted the questionnaires (or helped students to fill them in) according to the communication skills and profile of each student, providing alternative means when necessary (e.g., laminated picture cards, color codes, faces…) to gather the data. Figure 8 shows one of the participants using the immersive VR-IWS.

Fig. 8
figure 8

A participant in the study using the virtual reality individual work system

Full size image

Community involvement

As described in the introductory section, the individual preferences of a group of 30 autistic students were considered when defining the curricular areas and educational activities of the immersive VR-IWS. Their teachers also participated in the definition and fine-tuning of the tool. Finally, another group of 21 students participated in the usability, feasibility, and safety testing, as described in this section.

Data analysis

Descriptive and text analyses were conducted on the data obtained across feasibility measurements of the tasks developed by participants in the first session of use of VR (N = 127 of a total of N = 164 tasks done). Besides, an exploratory factor analysis was conducted on the adapted ASQ and the factor scores were extracted to use them for a linear regression analysis using jamovi (Leppink, 2019; The jamovi project, 2022). Multiple regression models were created to analyze the effects of the participant’s age, sex, intellectual disability, previous experience with VR, the school and the type of task conducted.

Results

Twenty-one participant students completed independently all the tasks they were offered. They completed a total of 164 tasks, almost 8 tasks per participant on average (SD 2.74) and the distribution per type of task was as follows: 58 tray manipulation (35%), 24 moving balls (15%), 33 classification tasks (20%), 31 association tasks (19%) and 18 half and half (11%). Tray manipulation was a previous necessary step to carry out the activities contained in the trays. The distribution of the other types of tasks is the result of the choices made by participating teachers when preparing the immersive VR sessions for autistic students according to their preferences and skills.

Feasibility

Every task was scored on a 1–7 scale of their easiness, the necessary time and the visual support provided. The results of performance on the task tests ranged from 5 to 7 in most of the cases. Figure 9 reflects the average score on each type of task.

Fig. 9
figure 9

The average score on easiness, the necessary time and visual support per type of task

Full size image

Only four students encountered difficulties, and these were related to only six of the 164 tasks that obtained a score of 3 or lower in some of the dimensions. The performance of these tasks was analyzed with interviews with the teachers involved to help provide a nuanced account of what happened. We next provide some details of four participants. Firstly, Juan (all names are nom de plumes to ensure anonymity) also identified as ES-1 in Table 1, was 8 years old and had an accompanying intellectual difficulty. He had difficulties in the tasks of picking up the tray from the entry shelf and bringing it to the finish shelf after completing it in the first session. He also had difficulties with the task of classifying objects according to a combination of shapes and colors. According to the teacher participating in the study, Juan has a low tolerance for frustration and if something goes wrong or is difficult for him, he gets angry. When picking up the trays, the system did not correctly identify the grip of his hand in the first of the sessions, so he got angry when performing the task. He was able to perform the very same tasks and additional tasks without difficulties in the second session. Next, Pedro a 16-year-old also with accompanying intellectual differences (also identified as ES-3), had difficulties in the tasks of picking up the tray from the entry shelf and bringing it to the finish shelve after completing it. Similar to Juan, this was related to hand-grabbing issues when picking up the tray in the first VR session. Ahmet (also identified as TR-2) is a 16-year-old, with accompanying intellectual differences who had difficulties in the task of moving two balls down. According to his teacher, he went outside the security area without realizing it during these events, so they were surprised when the image changed (in Meta Quest® the virtual images are immediately substituted with the real images captured by the HMD cameras when the user gets close to the borders of the play area). The teacher intervened to keep them in the safety zone. Finally, Fatma a12-year-old, with accompanying intellectual differences (also identified as TR-7) had difficulties in the task of sorting by shapes and also in the task of associating vegetables. According to her teacher, she had the same difficulties as Ahmet, related to getting close to the borders of the fixed play area.

The data analysis conducted over the tasks carried out by participants in their first session (N = 127) revealed sufficient relation between the three variables of the ASQ (i.e., easiness, time and visual information) to consider factor analysis (Bartlett χ2 = 159.387, gl = 3, p < 0.01). Both the overall and the item-specific KMO values were above 0.70 and the scree plot and the K1-criterion supported the 1-factor solution which explained 65.3% of the variance. Cronbach's α and McDonald's ω values were around 0.85, which showed that it is a reliable scale. The factor scores obtained through the Thurstone estimation method were used in a linear regression analysis as an outcome variable that showed that autistic children with intellectual disability scored significantly lower than children without intellectual disability in the scale (β = 0.72, CI = 0.306–1.135, p < 0.01), regardless of the school. Other variables such as the age of the participants, having previous experience using VR or the type of task conducted did not result sufficiently significant to be included in the final model.

On the safety evaluation, only one of the 21 participants, Antonio, who was 6 years old with accompanying intellectual differences (also identified as ES-4), felt a bit dizzy and uncomfortable with his hands and eyes in session 1; however, this did not happen to him in Session 2.

Usability

On the SUS Usability Scale, the student’s average score was 85.36 points (SD 9.91 points). A score over 68 is considered a good usability result according to that scale (Sauro, 2011).

Answers about Safety and Usability in the open-ended questions

In the interviews, open-ended questions about the experience of VR were formulated from the 21 participants, with the aim of gathering their experiences that could be improved for future versions and further VR developments.

Comments were obtained to the question “What did you like the most?” related to both the general experience of VR and to aspects of VR interaction or activities. Among the comments regarding the general experience of using VR mentioned by participants included: “I liked the experience of VR” (comment made by ES-3); “I liked the simplicity of the tasks” (ES-4); “Hand tracking seems pretty good” (UK-4 and UK-5); “It is amazing the fact that you can use hand grabbing in the VR” (UK-6) and “the variety of tasks” (UK-2). They specifically mentioned several activities they liked: “Classifying vegetables” (ES-1 and ES-3); “Dinosaur activities and the physics of elements” (ES-3); “Associating animals” (ES-6, ES-7, UK-1); “Association tasks” (UK-2); “Adding and subtraction” (UK-1); “Categorizing vehicles” (UK-6, TR-6). They also mentioned some peculiarities of the VR world when compared to the real world: “Balls disappear when dropped into the floor, and then they reappear on the same place” (ES-5) and “the dynamism of the control of the space” (UK-7).

Opportunities for improvement of the usability of the tool were present when they were asked “What did you like the least?”: One participant mentioned that” it was not challenging for him” (UK-7) and another stated that “it wasn’t the real world” (UK-2). Other participants mentioned that they would “prefer to have more freedom of movement to explore other areas of the virtual classroom” (TR-1 and TR-6) and that “there were no friends in the classroom” (TR-6).

Tasks-related opportunities for improvement were “difficulties with the shapes task” (ES-1); “Picking up and dropping shelves” (ES-3); and “some animals (such as bugs or birds) were too small to pick up” (UK-1).

Discussion

Intellectual disability

IWS can be implemented in different ways depending on the participants’ skills. For this study, one of the simplest versions has been implemented within a VR classroom to facilitate access to those autistic participants with an intellectual disability. Other adaptations related to the potential benefits of immersive VR were also implemented to facilitate access to this subgroup and to overcome limitations of previous studies and included: (1) an advanced implementation of hand tracking and grabbing, thus avoiding the need for use of VR controllers; (2) the customization of the sensory load and task complexity within the VR classroom; (3) the augmentation of the salience of the relevant interactive elements to drive participants’ attention at each moment; and (4) the provision of contingent reinforcers. This was meant to build a safe environment for them, as a starting point within VR where everything was predictable and cognitively accessible. The results of the study are very positive in this sense, as every single participant (n = 21) has been able to finish all the proposed tasks (164 in total) without major difficulties. The majority (n = 17 out of 21) of the students were able to do it with a high degree of easiness in all the tasks assigned to them. We provide details of the four who demonstrated more complex challenges, which highlights the immersive VR system could be further developed to help overcome challenges of movements and hand gestures in the virtual environment. As shown by the linear regression analysis conducted to identify the effect of the participant profiles on the scores obtained in the adapted ASQ, the most relevant variable that conditioned the ASQ results at the task level was the presence of an intellectual disability. This highlights the need for additional efforts to make immersive VR completely accessible for autistic students with intellectual disability.

Participant’s age

The above-mentioned linear regression analysis has not found a relevant influence of age on ASQ results at the task level, thus demonstrating that the proposed design is feasible and usable for students within the age range covered by the study (6 to 17). The research design carefully approached safety issues including specific instructions to participant teachers to stop the VR session if any difficulty appeared and included post-session safety tests for every single VR session. The fact that one of the youngest participants of the study was the only one who reported some (although minor) adverse effects, highlights the importance of implementing safety surveillance measures and closely observing and monitoring the appearance of possible symptoms and immediately interrupting the sessions when any of them appear, and the need for follow-up observations of younger groups (Schmidt et al., 2021).

Previous exposure to VR

The data analysis conducted on the feasibility results of the first session of this study did not reveal a significant explanation of previous exposure to VR on the feasibility. The technology used was highly immersive and the way of interacting with the virtual environment was a natural one, as participants only had to move their heads to visualize the different areas of the virtual environment (as they do in the real world), and they can manipulate objects using their hands (very similar to the real world but without haptic feedback). These factors may explain the non-relevant role of previous exposure to VR on participants’ results and can be interpreted as a positive outcome, as there was no need for previous training to be able to use the proposed immersive VR-IWS.

Implications for designing immersive VR applications for autistic people and intellectual disability

A key factor of previous digital technology studies and work in this field, supports the personal autonomy of autistic people (Taconet et al., 2023), as the study reported here has done. Personal autonomy is a basic capacity that can serve as a bridge to access many other types of learning.

Usability evaluation should be a prerequisite for primary research studies on the effectiveness of interventions since low usability would condition the results (Mazon et al., 2019). This study has shown how it is possible to evaluate usability in this context.

Data from this study, along with data from other studies, such as those on web accessibility (Frankowska-Takhari & Hassell, 2020; Raymaker et al., 2019), bring us closer to a scenario in which it is possible to offer evidence-based application design recommendations.

Limitations

We recognize several important limitations to our work presented here. First, we only worked with a small group of students and so the results need to be contextualized as such. Further work needs to support larger sample sizes, continue to iterate the design of this tool, and conduct further testing/evaluation. Second, schools were not homogeneous (e.g., none of the UK students had intellectual disability and none of the Turkish students had previously used VR) which limit the number and the statistical power of the analyses conducted. And third, our research, while working across sites in three countries, did not report specific cultural factors, which might have better enabled us to interpret our findings, and reveal new insights to the potential of tools like immersive VR across cultural settings.

Conclusion

Taken together, this study has highlighted the feasibility, usability, and safety of an immersive VR-IWS and worked with a range of autistic young people and their teachers to better understand task performance and identify any barriers to the use and possible safety issues. As in previous work with more basic forms of VR (i.e. McCleery et al., 2020; Newbutt et al., 2016) we did not experience any significant adverse effects or safety issues. Therefore, our findings here further support work locates VR-HMDs as a viable and safe tool used in controlled settings (i.e. classrooms) for young autistic people, including those with intellectual disabilities. These are design and development considerations that will help to improve the overall effectiveness and outcomes for a broader range of autistic young people. By extending this work with autistic people and their educators, we expect the system will become more accessible to a range of users.

The rapid advance in VR technology and the wide range of solutions available, mean that studies conducted so far have been done using different devices and these devices do not always have common features (i.e., image quality, immersion level, field of view and head and hand tracking accuracy). These differences make it difficult to combine the results of preliminary studies into a meta-analysis, as the conclusions may not always be applicable to commercially available devices.

Further research is also needed to evaluate how feasibility, safety and usability evolve across multiple sessions and the effects of interventions based on the use of a virtual implementation of the IWS on participants’ independence and learning. Exploring the contextual and cultural variables that may play a role in the feasibility, safety, and usability of VR IWS across schools and countries may also be important, before recommending its universal use.