Abstract
During extravehicular activities (EVAs), astronauts are heavily dependent on the Mission Center (MCC) and their Intra-Vehicular Astronaut (IVA) counterparts. Each procedure step in a mission is relayed to the astronaut through a real-time voice loop, and emergency procedures are written on cuff checklists that astronauts must read from their spherically shaped helmet. In all situations, crew members heavily rely on IVA or MCC support, especially when they do not understand a procedure or need help with a specific problem. However, it can be hard to communicate procedures effectively due to a lack of visual diagrams and situational awareness between the two parties. To improve EVA efficiency, we investigated the use of a virtual whiteboard on a heads-up display during a lunar surface EVA task with virtual reality (VR). The virtual whiteboard allows MCC to send additional visual guidance (e.g., drawings and annotations) overlayed on the astronaut’s visual field of view to better assist with mission tasks. We conducted a between-subjects experiment where 21 participants were asked to accomplish a rover repair procedure, with (n = 11) and without (n = 10) the virtual whiteboard, using a VR lunar environment with support of a research proctor acting as MCC. The whiteboard group completed the rover procedure 39.1% faster than the non-whiteboard group, and this difference was statistically significant (p = 0.017). The total number of words exchanged during the experimental sessions was not statistically different between groups (p = 0.99). However, participants in the whiteboard group showed a tendency to talk less than their counterparts, while the research proctor in the whiteboard group showed a tendency to speak more. Finally, analysis of the spatial locations during the experiment indicated that whiteboard participants stayed closer to the rover, showing a better focus on the task at hand and therefore short completion times. The results of this experiment inform future development of AR spacesuit technologies for future planetary exploration EVA operations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Abbreviations
- AR:
-
Augmented reality
- CAD:
-
Computer aided design
- EMU:
-
Extravehicular mobility unit
- EVA:
-
Extravehicular activity
- FOV:
-
Field of view
- HID:
-
Heads-in display
- HMD:
-
Head-mounted display
- HUD:
-
Heads-up display
- IRB:
-
Institutional review board
- ISS:
-
International space station
- IVA:
-
Intra-vehicular activity
- KDE:
-
Kernel density estimation
- LEO:
-
Low earth orbit
- MCC:
-
Mission control center
- MMSEV:
-
Multi-mission space exploration vehicle
- NBL:
-
Neutral Buoyancy Laboratory
- OBS:
-
Open broadcaster software
- PGT:
-
Pistol grip tool
- POV:
-
Point of view
- PUN2:
-
Photon Unity Networking 2
- SAFER:
-
Simplified Aid For EVA Rescue
- SUITS:
-
Spacesuit User Interface Technologies for Students
- T2AR:
-
Treadmill 2 augmented reality
- UI:
-
User interface
- VR:
-
Virtual reality
- VWCC:
-
Virtual whiteboard control console
- xEMU:
-
Exploration extravehicular mobility unit
References
Abdul Halim AZ (2018) Applications of augmented reality for inspection and maintenance process in automotive industry. J Fundam Appl Sci 10(3S):412–421
Abramovici M et al (2017) Context-aware maintenance support for augmented reality assistance and synchronous multi-user collaboration. Procedia CIRP 59:18–22. https://doi.org/10.1016/j.procir.2016.09.042
Anandapadmanaban E et al (2018) Holo-SEXTANT: an augmented reality planetary EVA navigation interface. In: 48th International conference on environmental systems, Albuquerque, New Mexico, 8–12 July, 2018.
“Artemis Plan (2020) NASA. www.nasa.gov/sites/default/files/atoms/files/artemis_plan-20200921.pdf
Boud AC, Haniff DJ, Baber C, Steiner SJ (1999) Virtual reality and augmented reality as a training tool for assembly tasks. Proc IEEE Int Conf Inf Visual 1999:32–36. https://doi.org/10.1109/IV.1999.781532
Byrne V, Mauldin J, Munson B (2021) Treadmill 2 augmented reality (T2 AR) ISS flight demonstration, system problem resolution team meeting
Dunbar B (n.d.) JSC human spaceflight capabilities—operations and training. Retrieved from April 29, 2019. www.nasa.gov/centers/johnson/capabilities/operationstraining-/index.html
Eppler D et al (2013) Desert research and technology studies (drats) 2010 science operations: operational approaches and lessons learned for managing science during human planetary surface missions. Acta Astronaut 90(2):224–241. https://doi.org/10.1016/j.actaastro.2012.03.009
“Headset-Head-Mounted Display.” Valve Index®—Upgrade Your Experience-Valve Corporation, www.valvesoftware.com/en/index/headset
Hsi S, Linn MC, Bell JE (1997) The role of spatial reasoning in engineering and the design of spatial instruction. J Eng Educ 86:151–158. https://doi.org/10.1002/j.2168-9830.1997.tb00278.x
Keller N et al (2021) Augmenting exercise protocols with interactive virtual reality environments. In: 2021 IEEE aerospace conference (50100), pp 1–13. https://doi.org/10.1109/AERO50100.2021.9438234.
Lim AK, Ryu J, Yoon HM et al (2021) Ergonomic effects of medical augmented reality glasses in video-assisted surgery. Surg Endosc. https://doi.org/10.1007/s00464-021-08363-8
Matsuo T et al (2021) High-intensity interval aerobic exercise training (HIAT) in occupational health. J Phys Fit Sports Med 10(3):145–150. https://doi.org/10.7600/jpfsm.10.145
McHenry N, Davis L, Gomez I, Coute N, Roehrs N, Villagran C, Chamitoff E, Diaz-Artiles A (2020) Design of an AR visor display system for extravehicular activity operations. In: 2020 IEEE aerospace conference, Big Sky, MT, USA, pp 1–11
McHenry N, Hunt T, Young W, Gardner A, Bhagavatula U, Bontz B, Chiu J, Chamitoff G, Diaz-Artiles A (2020) Evaluation of pre-flight and on orbit training methods utilizing VR, AIAA Scitech 2020 Forum. https://doi.org/10.2514/6.2020-0168
Meyer OA, Omdahl MK, Makransky G (2019) Investigating the effect of pre-training when learning through immersive virtual reality and video: a media and methods experiment. Comput Educ 140:103603. https://doi.org/10.1016/j.compedu.2019.103603.
Miller M et al (2014) Information flow model of human extravehicular activity operations. NASA Space Technology Research Fellowship
Moro C et al (2020) HoloLens and mobile augmented reality in medical and health science education: a randomised controlled trial. Br J Educ Technol 52(2):680–694. https://doi.org/10.1111/bjet.13049
Mourtzis D et al (2017) Augmented reality application to support remote maintenance as a service in the robotics industry. Procedia CIRP 63:46–51. https://doi.org/10.1016/j.procir.2017.03.154
“NASA Is Using Virtual Reality to Train Astronauts. Unimersiv, 14 Dec. 2017, unimersiv.com/how-nasa-is-using-virtual-and-augmented-reality-to-train-astronauts-37/
NASA SUITS Webinar, personal communication, Feb 19, 2020
Norheim J et al (2018) Architecture of a surface exploration traverse analysis and navigational tool. IEEE Aeros Conf 2018:1–11. https://doi.org/10.1109/AERO.2018.8396510
“OBS Studio.” Open Broadcaster Software, obsproject.com/
“PUN 2.” Photon Unity 3D Networking Framework SDKs and Game Backend | Photon Engine, www.photonengine.com/pun
Schlueter JA (2018) Remote maintenance assistance using real-time augmented reality authoring. Graduate theses and dissertations, 16457. https://doi.org/10.31274/etd-180810-6087
“SteamVR—Steam News Hub.” Welcome to Steam, SteamVR v1.14.16, store.steampowered.com/news/app/250820
Technologies, Unity. “Unity Game Engine.” Unity, unity.com/
Voorhis C, Morgan B (2007) Understanding power and rules of thumb for determining sample size. Tutor Quant Methods Psychol 3:21. https://doi.org/10.20982/tqmp.03.2.p043
Webel S, Bockholt U, Keil J (2011) Design criteria for AR-based training of maintenance and assembly tasks. In: Shumaker R (eds) Virtual and mixed reality—new trends. VMR 2011. Lecture Notes in Computer Science, vol 6773. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22021-0_15
Weeks JK et al (2020) Harnessing augmented reality and ct to teach first-year medical students head and neck anatomy. Acad Radiol. https://doi.org/10.1016/j.acra.2020.07.008
Woodruff R, Diaz-Artiles (2021) A multi-sensory virtual reality environment for the maintenance of long-term behavioral health. In: 72nd International astronautical congress, Dubai, United Arab Emirates, 25–29 Oct 2021
Zhuang Y et al (2021) Diagnosis of chronic kidney disease by three-dimensional contrast-enhanced ultrasound combined with augmented reality medical technology. J Healthc Eng 2021:1–12. https://doi.org/10.1155/2021/5542822
Acknowledgements
The results of this study contribute to the NASA Spacesuit User Interface Technologies for Students (SUITS) competition. This program is led by NASA Johnson Space Center and challenges students to design and create space suit information displays in AR, which will inform future capabilities of the heads-in display (HID) of the future Moon spacesuit, the exploration extravehicular mobility unit (xEMU). We would also like to thank the Texas A&M University Aerospace Engineering department for the support of this project, and our anonymous subjects for take part of this study. We especially appreciate the work and guidance of our mentors, Dr. Ana Diaz Artiles and Dr. Gregory Chamitoff, for assisting us in constructing this virtual study amidst a global pandemic. Thank you to the organizations that have funded the team and this research including Texas Space Grant, NASA. Lastly, we would like to thank the NASA SUITS competition organizers for bringing the team together.
Funding
This work was partially supported by the NASA Human Research Program (Grant Number 80NSSC19K0656) and the NASA Innovative Advanced Concepts (NIAC) program (Grant Number 80NSSC19K0969).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Availability of data, code, and materials
The data that support the findings of this study are available from the author, Neil McHenry, upon reasonable request. Sensitive information and participant names have been removed from the datasheets. All the source code for the software is available on GitHub, at this location: https://github.com/Henrynaut/UnityBase/tree/heatmap.
Ethics approval and consent to participate
Ethical approval for this research was obtained from the Texas A&M Institutional Review Board (Number: IRB2018-1472D), and written consent was given by all participants.
Consent for publication
All authors agreed to submit this research article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Davis, L., McHenry, N., Carrera, M. et al. Remote virtual whiteboard assistance for improving task performance during lunar surface operations. Virtual Reality 26, 559–570 (2022). https://doi.org/10.1007/s10055-021-00596-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-021-00596-1