Keywords

1 Introduction

Autism spectrum disorder (ASD) is a lifelong developmental disability that may impact people’s understanding of their environment. It can result in difficulties with social relationships, communication and behavior [2]. The latest studies by U.S. Department of Health and Human Services show that today, about 1 in 68 children is identified with ASD [3]. Attention to this specific group and applications for them has increased recently because of an increase in the awareness of prevalence of ASD. According to the Centers for Disease Control and Prevention, National Center for Health Statistics, prevalence of autism has increased by 289.5% from 1997 to 2008 [4]. The three most significant impairments that are associated with autism are listed as; social interaction, communication and behavior [5]. Because of these impairments, individuals with autism often have difficulty in their daily lives, especially while interacting with others. Because of the limiting properties of ASD, it is usually harder for individuals with autism to find jobs and succeed in them without proper training.

There are several advantages of using virtual reality over traditional training methods, such as active participation in accurately represented real-life like situations, opportunities for repetitive practice on simulators, unique training experiences with suitable and customizable difficulty levels, consistent and real time feedback, and opportunity for users to train and correct errors without severe consequences [6]. Due to these positive properties, virtual reality has been used in many different training applications for neurotypical individuals, such as training federal law enforcement agents on interrogation methods [7] and primary care physicians to perform intervention to treat alcohol abuse [8]. Virtual reality is found to be especially useful for populations with ASD, since virtual reality training offers several aspects that resonate with their characteristics, such as the predictability and the ability to repeat the exercises with adjusted difficulty levels until the user feels ready for the task to be performed [9].

There have been many scientific studies for training of individuals with ASD using virtual reality. These studies showed that virtual reality is an effective tool in training individuals with ASD [10, 11]. However, there has been little work to understand which virtual reality locomotion and interaction techniques are useful for individuals with ASD. Although there are many possible advantages of using virtual reality systems for job training, effective interaction techniques should be implemented for the users to truly benefit from the advantages of virtual reality. Since the perception and behaviors of individuals with ASD can be different from neurotypical individuals, using the same interaction techniques that work well for neurotypical individuals may not be a good practice. To bridge this gap, in this study, we examined different virtual reality interaction techniques for individuals with ASD. For this purpose, several different interaction techniques for object selection and manipulation (tangible object manipulation, haptic device, touch and snap, and touchscreen) and locomotion (real walking and walk-in-place) that have been implemented in different modules of the Virtual Reality for Vocational Rehabilitation (VR4VR) system were explored. VR4VR is a highly immersive virtual reality system for vocational rehabilitation of individuals with disabilities. The interaction techniques along with the two different display methods (head mounted display and curtain screen) were explored based on different aspects such as; the ease of interaction, level of enjoyment, frustration, dizziness, nauseousness, tiredness, and user statements.

2 Related Work

Several studies show that advances in technology have been used for assisting individuals with ASD for a long time. Goldsmith et al. collected and published the early examples of the technologies that were used for children with ASD [12]. In general, the developed systems were using an interaction technique that seemed to be the most suitable by the authors for the application and using that technique was rarely justified in the publications. Interaction techniques are one of the most important elements of virtual reality systems since they are directly related to the user’s experience with the system. Existing systems support a variety of interaction techniques for different platforms and input devices, from conventional devices such as mice or joysticks, to modern devices such as touch gestures, speech-recognition devices, and digitally augmented environments. Interaction techniques may affect several aspects of the user experience such as presence, enjoyment, frustration, and tiredness [13].

Most of the precious assistive training applications were implemented using a single interaction technique. The recent applications usually used touchscreen devices since they are easy to use, affordable and available. Furthermore, one of the recent studies showed that the tablet applications with multi touch interactions could make children with ASD more verbally and physically engaged as compared to the traditionally performed similar activities [14]. In a study conducted by Madsen et al., the researchers developed touch screen applications for teaching children with ASD to recognize facial expressions [15]. In this study, lessons learned about the software and hardware design of touch screen applications for this specific population were shared very briefly. In a study on developing an expression recognition game for individuals with ASD using touch enabled mobile devices, the research team has studied the previously existing popular ASD games and tried to consolidate guidelines for designing user interfaces for children with autism [16].

Another popular approach in designing applications for individuals with ASD is using touchless interactions. The availability of the depth sensors, such as Microsoft Kinect and their usage for skeleton tracking made this technique easily usable and popular. Moreover, some researchers suggest not to use wearable sensors since some individuals with ASD may not prefer to wear any sensors on them [17]. A study made on five children with ASD showed that games with touchless interaction helped in improving the attention skills for children with autism. However, the authors stated that the interaction technique was not tested on being appropriate or not for this special user group [18]. Another recent study for individuals with ASD was aiming at improving their motor skills [19]. With this goal, the researchers developed a motion based touchless application and tested the results. This study focused on the importance of physical activity, but did not justify why the authors chose to use this interaction technique while developing the application.

There were also some applications that used more than one interaction technique simultaneously. One study focused on full body interaction techniques for low functioning children with ASD [20]. An environment similar to a virtual reality cave was developed with surrounding projectors, cameras and sensors. Some touchless interaction techniques as well as touch-based interaction techniques were implemented, and the children’s acceptance of the system was discussed. Most of the children accepted the system and used it effectively.

With the emerging technology of virtual reality, some researchers have been integrating virtual reality interaction techniques into training applications for people with ASD. In a study, researchers used a virtual reality system to teach street-crossing skills to children with ASD [10]. The results showed that training in virtual reality improved the related skills of those children. In another study, a virtual reality driving training system was developed [11]. In this system, gaze tracking was implemented to track where the users looked during the training sessions since individuals with ASDs’ gaze positions were reported to be different from neurotypical individuals. The users were trained to look at the important regions such as traffic lights, traffic signs and pedestrians. The results showed that effective training was achieved using the developed virtual reality system with the incorporation of gaze positions.

Although many studies focused on using only one interaction technique per application, there have been some studies in the literature that used two different interaction techniques in the same application or in different applications that were developed for the same purpose for individuals with ASD. One example was a study that aimed at increasing the social engagement of children with ASD [21]. Two different games were used with two different interaction techniques. One was using multiple mice while the other was using a Diamond touch surface. The study did not test the differences observed while using these interaction techniques and did not make any suggestions for researchers. There was a detailed study on a computer-based training system for children with ASD [22]. In the study, a tangible user interface design was compared with the traditional mouse-based approach. The results of the study showed more learning progress using the tangible user interface. Another recent study showed observations on the usability of basic 3D interactions such as translation and rotation for the adolescents with ASD [23]. The authors aimed at finding the differences in the usage of 3D user interaction techniques between neurotypical individuals and individuals with autism. The results showed that the deficits in hand-eye coordination of individuals with ASD caused some difficulties in using the 3D interaction techniques. The authors suggested that developers should add some assistive cues to aid individuals with ASD with the hand-eye coordination.

Although different interaction techniques and their effects have been thoroughly examined for neurotypical individuals; so far, only limited research in this area has been explored for individuals with ASD. The existing studies briefly include a specific interaction technique or two basic interaction techniques. But virtual reality interactions can be more complicated since they are often (but not always) made as similar to real life as possible to increase the immersiveness. In our study, we explored several different immersive virtual reality interaction techniques with user studies and shared the results that may be beneficial for future virtual reality studies that focus on individuals with ASD.

3 Interaction Techniques

In our VR4VR system, there are six modules that were developed for the training of six different transferrable vocational skills: shelving, cleaning, environmental awareness, loading the back of a truck, money management, and social skills. In each different skill, the most convenient interaction technique to be tested was decided by literature review and discussions with the professional job trainers of individuals with ASD. These job trainers have been training individuals with ASD professionally for vocational rehabilitation for a long time and are highly experienced in this area.

To implement the locomotion and interaction techniques, the Unity game engine [24] and MiddleVR software [25] were used. The implemented software was run on a desktop computer with the following specifications: AMD FX-8150 3.61 Ghz Eight-Core CPU, AMD FirePro W600 GPU and 16 GB RAM. For motion tracking, the OptiTrack [26] V100R2 FLEX optical motion tracking system with 12 cameras was used in a 2 m × 2 m tracked area.

3.1 Object Selection and Manipulation

In the VR4VR system, different object selection and manipulation techniques were implemented and used in different skill modules. The interaction techniques were selected according to the requirements of the task and inputs received from the professional job trainers.

For object selection and manipulation, four different interaction techniques were implemented and explored: tangible object manipulation, haptic device, touch and snap, and touchscreen. These were used in different skill modules for interacting with the virtual world. These interaction techniques have been used in different skill modules of the VR4VR system. All interaction techniques and the skill modules they were tested in the VR4VR system along with the relevant interaction tasks are presented in Table 1.

Table 1. The interaction and locomotion techniques and the displays in the VR4VR system. The skill modules they were implemented in and the tasks that were used within these modules.
Full size table

Tangible Object Manipulation.

In this interaction technique, two types of real tangible objects were tracked and represented in the virtual world: (1) identical looking real boxes that were shown in the virtual world with different textures or labels, and (2) a broomstick handle that was represented as a vacuum cleaner or a mop that the user used for cleaning the virtual environments.

In the shelving skill module, there were two physical shelves and one physical table in the real-world environment. The virtual conjugates of those objects were created and placed at the same positions in the virtual world. Furthermore, there were four real boxes that were identical in appearance with reflective markers placed on top of each (for infrared camera tracking). The virtual conjugates of the boxes were created and placed at the same positions in the virtual world with different virtual textures projected on them.

An immersive tangible object manipulation technique was implemented and tested. With this technique, the users could move and rotate the real tangible boxes in the tracked area (see Fig. 1). This enabled a tactile feedback during the interaction, which was expected to increase the presence for the users. Head mounted display (HMD) was used along with hand bands with reflective markers. This enabled real time head and hand tracking. The user was able to see two virtual hands in the virtual world approximately at the same position and orientation with their real hands. We used virtual hand models representing real hands of users since it was reported to increase the realism and the immersiveness in virtual reality applications [27] and help users to better understand virtual distances.

Fig. 1.
figure 1

Tangible box manipulation in the VR4VR system. The user is rotating a real box. The curtain screen displays the user’s view through the HMD.

Full size image

A different tangible interaction method was implemented in the form of a tangible broomstick that was used for controlling a virtual vacuum cleaner and a virtual mop. The real broomstick handle was replaced with a virtual vacuum cleaner or a virtual mop in different tasks (see Fig. 2). To be able to track the real stick by the optical cameras in real-time, three pieces of reflector marker tape were attached around the cylinder. Since the cylinder was symmetric along its longitudinal axis, we used software calculations to visualize the cleaning head (nozzle or mop) according to the angle between the cylinder and the ground. This time, in addition to HMD and hand bands, feet bands with reflective markers were also worn by the user. This enabled real time head, hand and feet tracking. The user was able to see two virtual hands and feet in the virtual world.

Fig. 2.
figure 2

Tangible stick manipulation in the VR4VR system. Real broomstick handle with virtual representations of (left) a vacuum cleaner and (right) a mop.

Full size image

Haptic Device.

Haptic devices utilize force feedback to create a tactile sense of touch. In this module, Phantom Omni® haptic device by SensAble Technologies [28] was used for interacting with the virtual world. Phantom Omni® haptic device created a sense of weight for the users so that they could feel if they were interacting with a light or heavy object. This was expected to help in increasing the immersion. The working area of the haptic device was restricted to a planar surface that was parallel to the display area. This helped the users to relate the haptic device input to the visual output easily and removed the ambiguity coming from the extra degree of freedom for the sake of this task.

The buttons on the haptic device handle were assigned for specific commands (see Fig. 3). One of the buttons was used to hold the boxes similar to the vastly used mouse gesture for drag and drop. The other button was used to rotate the boxes by 90° counter clockwise.

Fig. 3.
figure 3

The haptic device interaction in the VR4VR system. (a) The user controls the cursor in the virtual world by using the haptic device. (b) Haptic device with two buttons on the handle.

Full size image

Touch and Snap.

Touch and snap interaction technique is often used in existing virtual reality applications. In this technique, a virtual object is snapped to a moving object, which usually is selected to be the virtual hand of the user. To trigger the release of the snapped object, different techniques can be used, such as time triggering, position triggering or gesture triggering.

In our implementation, user’s hands were equipped with reflective markers to be tracked in real time by the optical tracking system cameras. Those positions were used to place the virtual hands into the virtual world. Virtual litter object was snapped to the user’s virtual hands when the user bended and their hands came close to the litter. Users carried the litter objects in the virtual world and once the litter arrived in the vicinity of a trash bin, it disengaged from the hand and fell into the trash bin (see Fig. 4).

Fig. 4.
figure 4

Left: The touch and snap interaction in the VR4VR system. The user is (left) grabbing a virtual litter and (right) releasing their hand to throw the litter into a green virtual trash bin (right). Right: The touchscreen interaction in the VR4VR system. The digital cash register interface implemented on a touchscreen tablet computer. (Color figure online)

Full size image

Touchscreen.

With the increasing number of mobile devices such as smart phones and tablet computers, touch interaction became one of the most popular and prevalent interaction techniques in daily lives of users. Nowadays, even some personal computers, televisions and projectors are currently using this technique. Since the visual output and the touch input are aligned perfectly, this interaction method is thought to be very intuitive and easy to use.

In the VR4VR project, a touchscreen ASUS T100 10-in. display tablet computer was used as another interaction method in the form of a digital cash register (see Fig. 4). In this module, only the single touch static touching technique was used instead of the more complicated dynamic or multi touch interactions. A touchscreen keypad similar to the real cash register keypads was presented to the user.

3.2 Locomotion

Locomotion techniques are used for moving viewpoint (and avatar, if used) of users in virtual world. There are many different techniques for locomotion in virtual reality. In this user study, two locomotion techniques were implemented and explored: real walking and walk-in-place.

Real Walking.

To move the avatar in the virtual world with this locomotion technique, the user really walks in the tracked area, as they would do in real life. Although this is a very intuitive method, there is the significant restriction of the limited tracked area. In our implementation, the user was equipped with reflective markers on their hands and head, so that the real position of the user was approximated by these tracked position values and transferred to the virtual world. The virtual world was viewed inside from a virtual camera that was attached to the position of the virtual head and this view was rendered to the HMD. The movement and the rotation of the real head affected the virtual camera’s position and rotation, so that a realistic view of the virtual world could be displayed inside the HMD.

Since this technique is restricted with a limited tracking area, the user was surrounded by two physical shelves and one desk. All the tasks were designed so that they could be performed inside that limited tracked area (see Fig. 5). The design of the module allowed for the use of a limited tracking area and real walking as the locomotion method.

Fig. 5.
figure 5

Left: Picture of the real walking locomotion technique in the VR4VR system. The user walks inside the tracked area in the shelving module. Right: Pictures of the walk-in-place locomotion technique in the VR4VR system. The user navigates in the virtual world by walking in place in the real world. (left) Cleaning module, (right) environmental awareness module.

Full size image

Walk-in-Place.

If the real tracked area is smaller than the virtual world, then real walking technique may not be an effective choice due to the size restriction. To overcome this limitation, walk-in-place technique is commonly used in virtual reality implementations. In this technique, the user marches in the same place while the virtual avatar walks in the virtual world in the direction the user faces. This way, the limitation of the physical tracking area can easily be overcome. But this comes with the additional gesture of walking in place instead of real walking. In our VR4VR system, the implementation of this technique included different walking speeds, depending on the speed of walking in place gesture, so that the user could adjust the virtual speed of the avatar by modifying their marching speed in the real-world. The walking direction of the virtual avatar was controlled by the head direction of the user (see Fig. 5). We assumed that the head of the user was aligned with the user’s body orientation, and the neck of the user was not rotated. If the user turned their head to the left while their body was front facing and marched in place, they would have moved towards the left in the virtual world, where their head was facing.

To be able to detect the walk-in-place gesture, a marker set was attached to the same position on top of the user’s both feet. The difference of the heights (h0) of the left and the right foot markers (h0 = hr − hl) was calculated in each cycle of the program. If the difference of the heights of the foot markers became higher than a threshold (ht), the system entered the ready state for a possible walking action. In a specific time interval (∆t), if the difference of the heights of the foot markers (h1) became higher than a threshold again, but this time in the opposite direction, the walking action was triggered. The walking speed was calculated by collecting the time between the two triggers, and dividing the average step length to the collected time. After each trigger, the system looked for another trigger in a specific time interval. If another trigger occurred, the walking speed was updated, and walking proceeded. If no trigger was initiated in that time interval, the walking was ended.

3.3 Display Methods

Different locomotion and interaction techniques were implemented in different modules of the VR4VR system. Each implemented locomotion and interaction technique was decided to be more suitable for a specific display method following literature review and discussions with the professional job trainers. In this project, head mounted display and a 180° curved screen were used as the available display methods. In the study, the most suitable display method was selected for each locomotion interaction technique and the user preference on the display methods were explored.

Head Mounted Display.

Using head mounted display is an immersive way of displaying the virtual world to the user. A pair of digital displays is placed in front of the user’s eyes so that the user sees the virtual world through them. In our study, VR2200 head mounted display with high resolution XGA (1024 × 768) was used. The main reason for selecting this HMD was to provide individuals with ASD with empty space around their heads since the professional job trainers stated that covering all of their view with the HMD might create a sense of feeling trapped and disconnected from the real world. The job trainers also stated that having open space in the HMD would make sure that they still had some connection with the real world and provide a more comfortable training experience for individuals with ASD. Hence, instead of using a highly immersive HMD that surrounds the user’s whole vision, we preferred to use a more open spaced HMD that could be flipped up when not in use.

Curtain Screen.

For the interaction techniques that were implemented from a stationary point of view of the virtual world, a 180° large curved curtain screen with two projectors were employed in our VR4VR system. The curved screen had 3.5 m diameter and 2.4 m height, and its surface was white fabric to make the projections easily visible. This way, possible discomfort of users while using the HMD could be eliminated and two different display methods could be explored.

4 User Study

Ten neurotypical individuals (10 males, aged between 21 and 50) and nine individuals with high functioning ASD (7 males and 2 females, aged between 20 and 41) participated in the user study. Although there may be different definitions of neurotypical in the literature, a commonly used description would be indicating a typical neurological development and not being on the Autism spectrum [29]. In our study, we defined neurotypical individuals as not having any form of disability. All participants with ASD were previously diagnosed as high functioning by medical professionals. The participants with ASD came to our research facility to try all six modules of the VR4VR system with their professional job trainers. Within subjects experiment was performed. The users completed the skill modules in two different sessions that were scheduled on two different days. Each session was approximately two hours long and there were at least three days between the consecutive sessions. The order of the modules was assigned randomly with counterbalancing. At the end of each skill module, the users were asked to fill out a questionnaire about their experience with the VR4VR system. After completing the final skill module of the system, the users were asked to fill out a general questionnaire including questions about their preferences on different components of the system. Other than these methods of data collection, we also asked the users’ opinions about the interaction techniques during the breaks between the consecutive sessions in the form of interviews. The user study was performed under the IRB with the identification number Pro00013008.

The questions that were asked after each skill module were about the users’ experience with the interaction techniques and the display methods. The questions were about how easy it was to interact with the system, and how much they enjoyed, got frustrated or tired while interacting with the system. We used the answers to these questions to explore the used interaction and locomotion techniques, since they were one of the major differences between the six modules besides the tasks in our VR4VR system. The users were asked to choose from available answer choices based on a five-level Likert scale [30]. The users were also asked if they felt dizzy or nauseous, during and/or after the virtual reality training. In addition, the users were asked to select their preferred interaction technique and their preferred display method after completing all of the modules. The users were also asked to state their own opinions about these explored aspects.

5 Results

The results obtained from the participants are presented in this subsection to provide a general idea on the preference of a cohort of users with autism on several virtual reality locomotion and interaction techniques. However, since our VR4VR system was not designed with the aim of comparing virtual reality locomotion and interaction techniques, these results are only expected to give a general idea instead of generalizable powerful conclusions. Error bars on the charts represent standard errors calculated by dividing standard deviation with square root of sample size.

5.1 Selection and Manipulation Techniques

The results obtained from the users for the four different selection and manipulation techniques –tangible object manipulation, haptic device, touch and snap, and touchscreen– are presented in this sub-subsection. Figure 6 shows the average scores for the neurotypical users, whereas Fig. 7 shows the average scores for individuals with ASD. A score of 1 represents very little, while a score of 5 represents very much of the related aspect. Tiredness and frustration scores were low for all interaction techniques. On the other hand, for ease of interaction, enjoyment and immersion; the averages were above 3.0. The users with ASD found the haptic device hard to interact as compared to the other interaction techniques. Touchscreen interaction received the best results for the ease of interaction, enjoyment, and immersion aspects as compared to the other three interaction techniques for individuals with ASD.

Fig. 6.
figure 6

Survey results for the selection and manipulation techniques in the VR4VR system for neurotypical individuals.

Full size image
Fig. 7.
figure 7

Survey results for the selection and manipulation techniques in the VR4VR system for individuals with ASD.

Full size image

Statistical analysis was performed on the collected data using the IBM SPSS Statistics 23 software. The results of One Way ANOVA with repeated measures with manipulation technique as 4-level factor and alpha value 0.05 showed that there were significant differences for the ease of interaction for both population groups (F(3, 27) = 8.115, p = 0.001 for the neurotypical individuals and F(3, 24) = 4.406, p = 0.026 for the individuals with ASD). Mauchly’s test was used to test the data for sphericity. If the data was not spherical, Greenhouse-Geisser correction was made. Detailed ANOVA results can be seen in Table 2. For further exploration, a Tukey post hoc test was also performed. For the users with ASD, the haptic device was found to be significantly hard to interact as compared with the touchscreen.

Table 2. One Way ANOVA with repeated measures with manipulation technique as 4-level factor and alpha value 0.05 results for selection and manipulation techniques.
Full size table

As the users were asked to select one as their most preferred technique, most of the users with ASD stated preference for the touchscreen interaction and some users with ASD stated preference for the tangible object manipulation. On the other hand, most of the neurotypical users preferred tangible interaction. None of the users stated preference over the haptic device or the touch and snap interaction. A chart of the results is presented in Fig. 8.

Fig. 8.
figure 8

Preference of the two user groups on the selection and manipulation techniques in the VR4VR system.

Full size image

5.2 Locomotion Techniques

The results for the two different locomotion techniques; real walking and walk-in-place, are shown in Fig. 9 for neurotypical individuals and individuals with high functioning ASD. Real walking received higher scores for the ease of interaction, enjoyment and immersion as compared to walk-in-place for both populations. The results for the two locomotion techniques were quite similar for the tiredness aspect. Both user groups found the walk-in-place locomotion technique more frustrating as compared to the real walking. Paired samples t-tests with alpha 0.05 showed that walk-in-place locomotion method was significantly harder to use (t(9) = 4.714, p = 0.002), and introduced significantly more frustration (t(9) = −3.001, p = 0.017) for the neurotypical users as compared to the real walking. Detailed results of these paired samples t-tests are presented in Table 3. No significant difference was found for the data of the individuals with ASD. However, during the breaks, some of the users with ASD complained about the difficulty of walking-in-place while trying to concentrate on the tasks and they stated that they liked the real walking much more than the walk-in-place technique.

Fig. 9.
figure 9

Left: Survey results for the locomotion techniques in the VR4VR system for neurotypical individuals. Right: Survey results for the locomotion techniques in the VR4VR system for individuals with ASD.

Full size image
Table 3. Results of the paired samples t-tests with alpha 0.05 for the locomotion techniques.
Full size table

5.3 Display Methods

The average results for the different display methods are presented in Fig. 10 for neurotypical users and individuals with ASD. Curtain display received higher scores for the ease of interaction for both population groups. The tiredness values were lower for the curtain display for both user groups. This was a predicted result since the tasks performed with curtain display required low or no effort, as they didn’t involve locomotion. The paired samples t-tests with alpha 0.05 revealed that significant differences for neurotypical users for the ease of interaction (t(9) = 3.121, p = 0.012), enjoyment (t(9) = 2.502, p = 0.034), frustration (t(9) = −2.954, p = 0.016) and tiredness (t(9) = −5.679, p = 0.000) aspects. No statistically significant difference was found for the participants with ASD. Detailed results of the paired samples t-tests for the display methods can be seen in Table 4.

Fig. 10.
figure 10

Left: Survey results for the display methods in the VR4VR system for neurotypical individuals. Right: Survey results for the display methods in the VR4VR system for individuals with ASD.

Full size image
Table 4. Results of the paired samples t-tests with alpha 0.05 for the display methods.
Full size table

At the end of the testing, the users were asked about their preference of the display methods: head mounted display and curtain screen. These preference results are presented in Fig. 11. Most of the participants with ASD stated preference for the curtain display. For the neurotypical users, the preference results indicated the opposite.

Fig. 11.
figure 11

Preference of the two user groups on the display methods in the VR4VR system.

Full size image

As the individuals with ASD were asked about their opinions on the HMD display, only one of them stated that they did not like the concept of the HMD due to its resemblance to a television that was placed very near to their eyes. Other than that, no users with ASD stated any negative comments about the HMD. We did not observe any problems in the individuals with autism’s acceptance of the HMD during the user studies. All users with ASD made positive comments about the curtain display method and its ease of use.

6 Discussion

Among the selection and manipulation techniques, touchscreen received the highest score from the users with autism for the ease of interaction aspect, whereas haptic device received the lowest score. We interpret that this may be caused by the users’ previous experience. None of the users with ASD were familiar with the haptic device whereas all of them stated that they used touch enabled devices regularly on a daily basis. We observed that most of our participants with ASD were interacting with their touchscreen phones during the breaks of the user study sessions. The same fact might also be the reason of touchscreen’s receiving the best results for the enjoyment, frustration, and presence aspects. Since we were not expecting the touchscreen to get high presence scores, we asked the users with ASD about the reason behind giving higher scores for this aspect. The participants expressed the reason as the touchscreen’s not requiring any extra thinking or effort for them to use it and hence feeling intuitive. Tangible object manipulation and touch and snap were found to be the most tiring interaction techniques. Those results were expected, since these techniques required more physical activities such as carrying the boxes and bending. The users with ASD stated positive comments about the tangible object manipulation technique. Some of the users stated that it was much easier for them to interact with the tangible objects in virtual reality than the virtual ones. Some users also stated that the tangible boxes gave them physical cues and made the tasks easier to perform for them.

None of the users stated preference over the haptic device or touch and snap. We did not observe any difficulties in using the touch and snap interaction. However, the users with ASD found the haptic device difficult to interact as compared to the other interaction techniques. We observed some difficulties in some users with ASD’s using the haptic device. It took longer for the individuals with ASD to get comfortable with using this interaction technique as compared to the other interaction techniques.

In overall, most of the users with ASD preferred the touchscreen interaction over all of the other techniques tested. In light of these results, user comments and our observations, we interpret that it is better to implement selection and manipulation techniques that utilize commonly used real life interactions as much as possible for the effective use of individuals with autism in virtual reality applications, such as tangible and touchscreen techniques.

For the locomotion techniques, the most significant difference between the real walking and walk-in-place were in the frustration scores. Users got more frustrated while they were walking-in-place than they were real walking. We observed that it was difficult for the users with ASD to comprehend the walk-in-place gesture and keep doing that without really walking forward in the real world. Users gave better scores to real walking for the ease of interaction, enjoyment, and immersion aspects. The users preferred the real walking technique and stated that they did not like the walk-in-place technique. These results aligned with [31,32,33]. Although there was no significant difference in the data of individuals with ASD, based on the verbal comments and their preference indications, we interpret that walk-in-place may be a questionable locomotion technique to be used for individuals with ASD in virtual reality applications. On the other hand, we interpret that real walking can be considered as a suitable method for virtual reality implementations for individuals with autism. Of course, real walking locomotion is not easy to achieve due to the limitation imposed by the motion tracking cameras but as a solution, the tasks in the virtual world would be designed such that the users do not need to go outside the physical tracked area, such as the shelving module of our VR4VR system.

As the display methods were explored, most of the users with ASD preferred the curtain screen over the head mounted display. But as they were interviewed, only one user with ASD stated negative comments about the view through the HMD, mostly about the tired eyes. There weren’t any acceptance or adjustment problems, thus we interpret that both the curtain display and the HMD might be used as virtual reality viewing tools for users with ASD. The neurotypical users stated preference for the HMD over the curtain screen. The results were in alignment with [33, 34].

7 Limitations

The user study was performed with nine individuals with high functioning autism. Thus, it should be avoided to generalize these results. The results may not be applicable for the medium or low functioning individuals with ASD. This study was not designed for a thorough evaluation of different interaction, locomotion and display methods. An existing system was utilized to explore the use of interaction, locomotion and display methods in virtual reality by individuals with ASD with the aimed outcome of providing some insight in this area. These techniques were not examined in isolation, but within a larger scope tasks, hence might have been affected by other factors such as the task design and different virtual environments. In the data analysis, univariate analysis of variance (ANOVA) was performed since not all variables were dependent or correlated. However, some variables such as enjoyment and frustration can be considered as dependent, and a multivariate analysis of variance (MANOVA) could give different results. The study results, interpretations and conclusions were not based on solid data, but mostly on the verbal user statements and observations during the user study sessions. Even still, the long hours of testing sessions with the participants gave us the opportunity to observe, discuss and have an initial idea on the suitability of the several virtual reality interaction, locomotion and display methods for the use of individuals with autism.

8 Conclusions

This study aims at exploring different virtual reality interaction techniques for individuals with autism within the VR4VR system. Several interaction techniques of object selection and manipulation, locomotion and display methods were implemented and tested in different contexts. User experience with these interaction techniques were explored with a user study of ten neurotypical individuals and nine individuals with high functioning ASD. For the object selection and manipulation; touchscreen and tangible interaction methods were preferred by the individuals with ASD. The walk-in-place locomotion technique were found frustrating and difficult to perform by the individuals with ASD. Curtain display method received higher preference scores from the individuals with ASD although they accepted the HMD as well. Based on our observations during the user study sessions and the verbal comments made by the participants, we interpret that users with autism are likely to prefer the most realistic and real life linkable interaction techniques while having some difficulties with the gesture based and more abstract ones. Although not based on data analysis results, we still believe that these insights may benefit future VR studies focusing on individuals with ASD.

Future work will consist of implementing several isolated modules for evaluation of different virtual reality interaction and locomotion techniques for individuals with autism. After the modules are implemented, a user study with more individuals with autism will be performed with the aim of revealing suitable virtual reality interaction and locomotion techniques for this group of individuals.