User Experience Evaluation in Virtual Reality for Autism: A Systematic Literature Review

Abstract

Virtual reality (VR) used for people with autism spectrum disorder (ASD) is gaining a place among researchers lately. The rich interactivity provided by VR makes it a powerful tool for various purposes, one of which is to support interventions for people with ASD. Since people with ASD have a different sensory experience, the user experience (UX) within VR applications for ASD becomes a crucial aspect. There have been past literature reviews of VR applications for ASD. However, to the best of our knowledge, there is hardly one that focuses on the methods used in evaluating the users’ experience towards interaction techniques or interface elements incorporated in various VR applications. In this review, 24 studies met predetermined PICo (Population, Interest, Context) criteria from the search based on three relevant databases, namely Pubmed NCBI, Dimensions.ai, and IEEE Xplore. Direct examination from observers appears as the most popular method used to assess experiences of people with ASD in using VR and followed by self-report questionnaire method. In addition, although all relevant literatures mentioned their UX evaluation method, only a few studies whose evaluation is relevantly subjected towards their interaction techniques or user interface elements. This review is expected to provide insights and considerations in designing UX evaluation method for VR applications for people with ASD in the future.

Appendix

Author, Year	Intervention/Therapy/Treatment	User interface/Interaction technique	UX Evaluation Method	Output/Variable Measured	Interesting Insights
Li et al. (2019) [48]	Social stories using Kolb’s experiential learning model	- Interface components: Visual hints, task list - Interaction: Verbal response - 3D Interaction: Selecting object (Tapping the in-VR rating button)	- Observation from observer; onsite and video. Direct interview to participant	- Output: Case report per individual (qualitative)	Enabling in-VR task lists, hints, and real-time feedbacks are effective for providing in-VR facilitations, which is needed in delivering treatment for ASD users
Junaidi et al. (2020) [49]	Social scenario; selecting food menu, seating, and putting the dirty dishes	- Interface comp: Visual hints, task list - Interaction: Verbal response	- Observation from observer and observer filling questionnaire	- Variables measured: Borg and Gall; convenience of using HMD, object readability in VR, understanding the instructions in VR - Output: Statistics of user experience (quantitative)	- The VR content used has not been able to address LFA problems - LFA users want to use VR devices, but with some limitations
Di Mascio et al. (2020) [37]	Scenes (space, forest, historic site) and games (virtual blocks)	- Interface comp: Scenes - Interaction: Remote control, gamepad, virtual hand (leap motion) - 3D Interaction: Selecting and manipulating an object	- Concurrent Think Aloud (CTA), - Observation from observer. Direct interview to participant - Self-report questionnaire/survey (but not used)	- Output: Acceptability and usability data (quantitative), and engagement (qualitative)	- The evaluation framework could represent the foundations for an innovative IVE evaluation framework
Malihi et al. 2020 [50]	Social scenario (in school bus) and scenes (Blue Planet)	- Interaction: Exploring the VR scene by moving their head (VR) or mouse control	- Self-report questionnaire/survey - Using elastic nets, random forest, AdaBoost, and NN to predict the user-reported ratings of sense	- Variable measured: Demography, sense of presence (Spatial presence, naturalness, engagement) and safety (cybersickness, anxiety) - Output: correlation between IQ and anxiety to sense of presence and safety	- The most accurately predicted target is spatial presence, followed by engagement - IQ and anxiety traits are identified as critical predictors of spatial presence and engagement
Ravindran et al. (2019) [30]	Social scenario consists of reciprocal interaction	- Interface comp: Scenes, Avatar - Interaction: Verbal response	- Observation from observer; onsite and video. Observer filling questionnaire and do a direct interview to participant - Self-report questionnaire/survey	- Variables measured: Joint attention, participant's mood, self-report condition (Alertness, eye discomfort, clarity of vision, headache, stomachache, balance, enjoyment), and observation result (participant’s tolerance of HMD, enjoyment, adverse side effects, value from Floreo)	- Observations from the observer is the primary evaluation method - Participant’s questionnaire response was not consistent as the observer’s note due to limited communication skills
Feng et al. (2018) [31]	Showing virtual faces on the screen	- Interface comp: Avatar with varying realism degree and pupil size - Interaction: Mouse control	- Tracking participant's eye movement (ROI) and combining the user's preference between two different faces with different realism and eye size	- Variables measured: - Region of Interest (ROI) from participant's eye-gaze - Participant's preference	ASD children rather indifferent towards facial features manipulation
Newbutt et al. (2020) [51]	Scenes (Moon, Historical Place) and Game (Throwing Ball)	- Interface comp: Scenes - Interaction: virtual hand (controller) - 3D Interaction: Selecting object	- Observer do the direct interview to participant - Self-report questionnaire/survey	- Variables measured: Enjoyment/Usefulness, physical experience, and preference	The most preferred device is the high-end HMD
De Luca et al. (2021) [52]	Scenario games with CBT	- Interface comp: Scene - 3D Interaction: Full-body movement	- Observation from observer, using GARS - Psychometric assessment - Parental distress (Self-report questionnaire)	Variables measured: Psychometric (Nonverbal fluid intelligence, attention process, visual-spatial functions)	Combined rehabilitation using CBT with VR may be promising in improving cognition
Schmidt et al. (2021) [53]	Train public transportation skills	- Interface comp: Scenes - Interaction: Selecting object	- Expert review, observation from observer. Observer filling questionnaire - Self-report questionnaire/survey	- Variable measured: Usability question set using SUS and adjective scale by Bangor	The spherical video-based virtual reality (SVVR) is easy-to-use and has positive user-experience feedback in general
Bozgeyikli et al. (2019) [39]	Vocational: cleaning, shelving, environmental awareness, loading, money management, and social	- Interface comp: Scenes - 3D interaction: -- Object selection & manipulation; Tangible object manipulation, Haptic Device, Touch and Snap, Touchscreen -- Locomotion; Real Walking, Walk-in Place	- Observation from observer - Self-report questionnaire/survey - Participant’s questionnaire response is processed using ANOVA,	- Variables measured: user experience (difficult in understanding, difficult in operating, in control, enjoyment, effort, tiredness, overwhelmedness, frustration) and task completion	- Participants prefer touchscreen and tangible interaction techniques and real walking - Participants had more difficulty in gesture and more abstract interaction
Bozgeyikli et al. (2018) [40]	Vocational: cleaning, shelving, environmental awareness, loading, money management, and social	- Display method: HMD, curtain screen - 3D interaction:, -- Object selection & manipulation; Tangible object manipulation, Haptic Device, Touch and Snap, Touchscreen -- Locomotion: Real Walking, Walk-in Place,	- Observation from observer - Self-report questionnaire/survey - Questionnaire with a Likert scale is processed using one-way ANOVA	- Variables measured: preference, cybersickness, user experience (ease of interaction, enjoyment, frustration, tiredness, immersion)	- (Same points from preliminary result) - Curtain display is more preferred than HMD - Participants prefer more realistic and real-life linkable interaction techniques
Bozgeyikli et al. (2018) [45]	Scene in virtual warehouse with a realistic appearance, the user asked to go without colliding obstacles	- Interface comp: Avatar - Interface attribute: Instruction methods, Visual fidelity, View zoom, Clutter, Motion	- Observation from observer - Self-report questionnaire/survey - Questionnaire is processed using ANOVA, Mauchly sphericity test	- Variables measured: user experience, presence, motion sickness, user comment	To use animated instructions and avoid verbal instructions, use low visual fidelity and standard view zoom, and use no clutter and no motion in VR training application for HF ASD
Bozgeyikli et al. (2016) [41]	Scene in virtual warehouse with a realistic appearance, user asked to go without colliding obstacles	- Interface comp: Avatar - 3D interaction: -- 3 commonly used: redirected walking, walk-in-place, joystick, -- 2 unexplored techniques: stepper machine, point & teleport -- 3 selected techniques for ASD: flying, flapping, and trackball	- Observation from observer - Self-report questionnaire/survey (Likert scale) - Questionnaire is processed using ANOVA, Mauchly sphericity test	- Variables measured: ease of understanding, ease of operating, required effort, tiredness, being in control, enjoyment, being overwhelmed and frustrated, motion sickness, and presence	- Joystick, point & teleport, redirected walking, and walk-in place are suitable for VR locomotion techniques - Hand gesture-based and automatic movement locomotion are not convenient for HD ASD
Mei et al. (2015) [42]	Game: Imagination soccer (Hand-eye coordination)	- Interface comp: Customizable Virtual Human (CVH) & NonCVH	- Self-report questionnaire/survey - Question set using PIFF2 and Likert scale	- Variables measured: task success rate, user experience (presence, involvement, and flow)	- CVH can increase hand-eye-coordination performance - CVH improved the UX (presence, involvement, and flow) of ASD users
Nuguri et al. (2021) [35]	Social scenario in Orientation Day; recognizing facial expressions, sharing ideas, turn-taking in conversations	- Interface comp: Scenes, bubbles, pop-up boxes, status bars - 3D Interaction: Locomotion (teleportation in VR)	- Self-report questionnaire/survey (using SUS) - Observation from observer towards the app - Observation from observer towards user’s real-time brain signals	- Variables measured: immersiveness, frustration, engagement, task completion	- Teleportation in VR is preferred than moving with a mouse & keyboard - VR is preferred than desktop - Network speed affects the satisfaction of UX
Mei et al. (2018) [32]	Game: Imagination soccer (Hand-eye coordination)	- Interface comp: Scenes, avatar (CVH)	- Eye-gaze logging - Paired t-test (CVH-NCVH comparison) using Bonferroni correction	Variable measured: Total acquired joint, time gazing at Regions of interest (ROI)	- Including CVH results on slower reaction to do the joint attention request - CVH helps ASD to gaze less at unimportant areas
Zhao et al. (2016) [54]	Puzzle game, collection game, and delivery game	- 3D Interaction: Object manipulation (hold, move, and drop)	- Participants fill a questionnaire/survey	Variable measured: user experience, task performance	- Participants are engaged and motivated to play well - Not really adapted to use Leap Motion device
Finkelstein et al. (2013) [43]	Game with physical activity; - Dodging, ducking, jumping - Collecting points	- Interface comp: Scene - 3D Interaction: Full-body movement	- Self-report questionnaire/survey - Observation from observer	Variables measured: demographic, total energy burnt, user experience (enjoyment, replayability, amount of exercise, preference)	- Two non-verbal participants cannot provide feedback; some participants received parental assistance in filling questionnaire - Participants show more interest in familiar themes
Schmidt et al. (2019) [55]	Training to use public transportation (Applied Behavior Analysis + immersive technologies + special education curriculum)	- Interface comp: Scene	- Observation from observer. Direct interview to participant - Self-report questionnaire/survey (SUS) - Cluster semantics coding categories using affinity mapping techniques	Output: Affect (joy/fun/excitement, willingness to return), Accessibility (physical, cognitive, cybersickness), General (usefulness, realism, real-world connections)	The qualitative codes categorization (on output) and operationalizations
Bernardes et al. (2015) [38]	Traveling training; validating the ticket, sitting in the right place, pressing the stop button	- Interface comp: Scene	- Concurrent Think Aloud (CTA) - Task performance (completion time)	Variables measured: Technology acceptance, interface comprehension, task performance	- ASD group takes longer on finishing the task compared to the control group (TD) - Serious game intervention is needed
Kuriakose et al. (2012) [36]	Bidirectional social conversation	- Interface comp: Scene, Avatar	- Observation from observer towards user performance - Observation from observer towards physiological response (Heart Rate and Skin Temperature)	Variables measured: affective states (engagement and likeness) through physiological features (HR & SKT)	- The use of wired physiological sensors possibly induces additional anxiety factors and limiting the user’s movement freedom
Halabi et al. (2017) [44]	Conversation (Role-play and turn-taking)	- Interface comp: Scene, Avatar - Interaction: Verbal response (speech recognition), waving (gesture recognition)	- Self-report questionnaire/survey	- Variables measured: voice and physical motion, time in task completion	- Level of immersion: CAVE > HMD > Desktop - The most liked display method: CAVE
Lahiri et al. (2012) [33]	Bidirectional social conversation	- Interface comp: Scene, 3D Avatar - Interaction: mouse	- Observation from observer - Physiological signal acquisition	- Variables measured: ROI (face, context-relevant objects, and other) and attention duration	This system provides feedback based on quantitative measurement of user’s performance and viewing pattern
Lahiri et al. (2011) [34]	Bidirectional social conversation	- Interface comp: Scene, Avatar	- Observation from observer - Physiological signal acquisition	Variables measured: Physiological data and Affective states (anxiety, enjoyment, and engagement of participants)	The system can predict user’s affective states from objective measures (physiological signals)