Skip to main content
Book cover

International Conference on Human-Computer Interaction

HCII 2020: Virtual, Augmented and Mixed Reality. Design and Interaction pp 575–590Cite as

Virtual Reality for Immersive Human Machine Teaming with Vehicles

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12190)

Abstract

We present developments in constructing a 3D environment and integrating a virtual reality headset in our Project Aquaticus platform. We designed Project Aquaticus to examine the interactions between human-robot teammate trust, cognitive load, and perceived robot intelligence levels while they compete in games of capture the flag on the water. Further, this platform will allows us to study human learning of tactical judgment under a variety of robot capabilities. To enable human-machine teaming (HMT), we created a testbed where humans operate motorized kayaks while the robots are autonomous catamaran-style surface vehicles. MOOS-IvP provides autonomy for the robots. After receiving an order from a human, the autonomous teammates can perform tasks conducive to capturing the flag, such as defending or attacking a flag. In the Project Aquaticus simulation, the humans control their virtual vehicle with a joystick and communicate with their robots via radio. Our current simulation is not engaging or realistic for participants because it presents a top-down, omniscient view of the field. This fully observable representation of the world is well suited for managing operations from the shore and teaching new players game mechanics and strategies; however, it does not accurately reflect the limited and almost chaotic view of the world a participant experiences while in their motorized kayak on the water. We present creating a 3D visualization through Unity that users experience through a virtual reality headset. Such a system allows us to perform experiments without the need for a significant investment in on-water experiment resources while also permitting us to gather data year-round through the cold winter months.

Keywords

  • Underlying and supporting technologies
  • VAMR technologies and techniques for human-robot interaction
  • Virtual reality
  • Human-robot teaming
  • Simulation

The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Department of Defense or the U.S. Government.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-49695-1_39
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-49695-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

References

  1. Virtual Reality: State of Military Research and Applications in Member Countries (La realite virutelle: L’etat actuel des travaux de recherche et des applications militaires dans les pays membres de l’Alliance). Technical report, RTO-TR-018, NATO RESEARCH AND TECHNOLOGY ORGANIZATION NEUILLY-SUR-SEINE, FRANCE, February 2003. https://apps.dtic.mil/docs/citations/ADA411978

  2. Aguero, C., et al.: Inside the virtual robotics challenge: simulating real-time robotic disaster response. IEEE Trans. Autom. Sci. Eng. 12(2), 494–506 (2015). https://doi.org/10.1109/TASE.2014.2368997

    CrossRef  Google Scholar 

  3. Lee, E.A., Wong, K.W., Fung, C.C.: How does desktop virtual reality enhance learning outcomes? A structural equation modeling approach. Comput. Educ. 55(4), 1424–1442 (2010). https://doi.org/10.1016/j.compedu.2010.06.006. https://linkinghub.elsevier.com/retrieve/pii/S0360131510001661

    CrossRef  Google Scholar 

  4. Bailenson, J.: Experience on demand: what virtual reality is, how it works, and what it can do (2019). oCLC: 1037811122

    Google Scholar 

  5. Beal, S.A.: Using games for training dismounted light infantry leaders: emergent questions and lessons learned. Technical report, 1841, US Army Research Institude for the Behavioral and Social Sciences (2005). https://doi.org/10.1037/e423262005-001. http://doi.apa.org/get-pe-doi.cfm?doi=10.1037/e423262005-001, type: dataset

  6. Bonwell, C.C., Eison, J.A.: Active Learning: Creating Excitement in the Classroom. 1991 ASHE-ERIC Higher Education Reports. ERIC (1991)

    Google Scholar 

  7. Cabello, R.: Three.js, November 2018. https://threejs.org/. Accessed 02 Nov 2018

  8. Campbell, C.H., Knerr, B.W., Lampton, D.R.: Virtual environments for infantry soldiers: virtual environments for dismounted soldier simulation, training and mission rehearsal. Technical report, ARI-SP-59, ARMY RESEARCH INST FOR THE BEHAVIORAL AND SOCIAL SCIENCES ALEXANDRIA VA, May 2004. https://apps.dtic.mil/docs/citations/ADA425082

  9. Catanzariti, P.: Bringing VR to the web with google cardboard and three.js, November 2018. https://www.sitepoint.com/bringing-vr-to-web-google-cardboard-three-js/. Accessed 02 Dec 2018

  10. HTC Corporation: HTC VIVE. https://www.vive.com/. Accessed 14 Aug 2019

  11. Valve Corporation: Valve index. https://www.valvesoftware.com/en/index/headset/. Accessed 14 Aug 2019

  12. DARPA: MIT DARPA robotics challenge team, November 2018. http://drc.mit.edu/. Accessed 02 Nov 2018

  13. Deits, R.: Meshcat-Python, November 2018. https://github.com/rdeits/meshcat-python/. Accessed 02 Nov 2018

  14. Facebook Technologies, L.: Oculus rift. https://www.oculus.com/. Accessed 14 Aug 2019

  15. Foundation, O.S.R.: Gazebo, February 2018. http://gazebosim.org/. Accessed 02 Feb 2018

  16. Google: Google cardboard, November 2018. https://vr.google.com/cardboard/. Accessed 02 Dec 2018

  17. Kearns, K.: RFI: Autonomy for Loyal Wingman. Air Force Research Laboratory (AFRL), July 2015

    Google Scholar 

  18. Lipton, J.I., Fay, A.J., Rus, D.: Baxter’s homunculus: virtual reality spaces for teleoperation in manufacturing. IEEE Robot. Autom. Lett. 3(1), 179–186 (2018)

    CrossRef  Google Scholar 

  19. LLC., O.V.: Oculus rift, February 2018. https://www.oculus.com/. Accessed 02 Feb 2018

  20. Lynn, V.L., Cherry, P., Brady, E., Droulihet, P., Evers, W.: Army science board 1991 summer study - army simulation strategy. Technical report, Army Science Board, Washington, December 1991. https://apps.dtic.mil/docs/citations/ADA250382

  21. Novitzky, M., Benjamin, M.R., Robinette, P., Dougherty, H.R., Fitzgerald, C., Schmidt, H.: Virtual reality for immersive simulated experiments of human-robot interactions in the marine environment. In: Workshop on Virtual, Augmented and Mixed Reality for Human-Robot Interaction at HRI 2018, Chicago, March 2018

    Google Scholar 

  22. Novitzky, M., Benjamin, M.R., Robinette, P., Dougherty, H.R., Fitzgerald, C., Schmidt, H.: Virtual reality for immersive simulated experiments of human-robot interactions in the marine environment. In: ACM/IEEE International Conference on Human-Robot Interaction (HRI) 2018 Workshop on Virtual, Augmented, and Mixed Reality (VAM) for Human-Robot Interaction. ACM (2018)

    Google Scholar 

  23. Novitzky, M., Dougherty, H.R.R., Benjamin, M.R.: A human-robot speech interface for an autonomous marine teammate. In: Agah, A., Cabibihan, J.-J., Howard, A.M., Salichs, M.A., He, H. (eds.) ICSR 2016. LNCS (LNAI), vol. 9979, pp. 513–520. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47437-3_50

    CrossRef  Google Scholar 

  24. Novitzky, M., Fitzgerald, C., Gleason, D., Robinette, P., Benjamin, M.R., Schmidt, H.: Integrating a multi-modal interface in a marine human-robot teaming testbed. In: IEEE International Conference on Robotics and Automation (ICRA) workshop on Human-Robot Teaming Beyond Human Operational Speeds (RT-DUNE). IEEE, Montreal, May 2019

    Google Scholar 

  25. Novitzky, M., Fitzgerald, C., Robinette, P., Benjamin, M.R., Schmidt, H.: Updated: virtual reality for immersive simulated experiments of human-robot interactions in the marine environment. In: Proceedings of the Workshop Virtual, Augmented, and Mixed Reality for Human-Robot Interaction ACM/IEEE International Conference on Human-Robot Interaction. ACM/IEEE, Daegu, March 2019

    Google Scholar 

  26. Novitzky, M., Robinette, P., Benjamin, M.R., Fitzgerald, C., Schmidt, H.: Aquaticus: publicly available datasets from a marine human-robot teaming testbed. In: Companion of the 2019 ACM/IEEE International Conference on Human-Robot Interaction. Daegu, March 2019

    Google Scholar 

  27. Novitzky, M., Robinette, P., Benjamin, M.R., Gleason, D.K., Fitzgerald, C., Schmidt, H.: Preliminary interactions of human-robot trust, cognitive load, and robot intelligence levels in a competitive game. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 203–204 (2018)

    Google Scholar 

  28. Novitzky, M., Robinette, P., Fitzgerald, C., Dougherty, H.R.R., Benjamin, M., Schmidt, H.: Issues and mitigation strategies for deploying human-robot experiments on the water for competitive games in an academic environment. In: Proceedings of the Workshop Dangerous HRI: Testing Real-World Robots has Real-World Consequences ACM/IEEE International Conference on Human-Robot Interaction. ACM/IEEE, Daegu (2019)

    Google Scholar 

  29. Novitzky, M., Robinette, P., Gleason, D.K., Benjamin, M.R.: A platform for studying human-machine teaming on the water with physiological sensors. In: Workshop on Human-Centered Robotics: Interaction, Physiological Integration and Autonomy at RSS 2017, Cambridge, July 2017

    Google Scholar 

  30. Pleban, R.J., Vaughn, E.D., Sidman, J., Geyer, A., Semmens, R.: Training platoon leader adaptive thinking skills in a classroom setting. Research Report 1948, U.S. Army Research Institute for the Behavioral & Social Sciences, Arlington (2011). http://www.dtic.mil/dtic/tr/fulltext/u2/a544978.pdf

  31. Robinette, P., Novitzky, M., Benjamin, M.R.: Trusting a robot as a user versus as a teammate. In: Workshop on Morality and Social Trust in Autonomous Robots at RSS 2017, Cambridge, July 2017

    Google Scholar 

  32. Robinette, P., Novitzky, M., Benjamin, M.R.: Longitudinal interactions between human and robot teammates in a marine environment. In: In Workshop on Longitudinal Human-Robot Teaming at HRI 2018, Chicago, March 2018

    Google Scholar 

  33. Robinette, P., Novitzky, M., Benjamin, M.R., Fitzgerald, C., Schmidt, H.: Exploring human-robot trust during teaming in a real-world testbed. In: Companion of the 2019 ACM/IEEE International Conference on Human-Robot Interaction, March 2019

    Google Scholar 

  34. Ropelato, S., Zünd, F., Magnenat, S., Menozzi, M., Sumner, R.: Adaptive tutoring on a virtual reality driving simulator. In: 1st Workshop on Artificial Intelligence Meets Virtual and Augmented Worlds (AIVRAR) in Conjunction with SIGGRAPH Asia 2017, ETH Zurich (2017)

    Google Scholar 

  35. Semmens, R., Martelaro, N., Kaveti, P., Stent, S., Ju, W.: Is now a good time? An empirical study of vehicle-driver communication timing. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, pp. 1–12. Association for Computing Machinery, Glasgow, May 2019. https://doi.org/10.1145/3290605.3300867

  36. Semmens, R., Novitzky, M., Robinette, P., Lieberman, G.: Insights into Expertise and Tactics in Human-Robot Teaming

    Google Scholar 

  37. Taheri, S.M., Matsushita, K., Sasaki, M.: Development of a driving simulator with analyzing driver’ characteristics based on a virtual reality head mounted display. J. Transp. Technol. 7(03), 351 (2017)

    Google Scholar 

  38. Thorndike, E.L., Woodworth, R.S.: The influence of improvement in one mental functionupon the efficiency of other functions: III. Functions involving attention, observation and discrimination. Psychol. Rev. 8(6), 553–564 (1901). https://doi.org/10.1037/h0071363

    CrossRef  Google Scholar 

  39. Tucker, J., Semmens, R., Sidman, J., Geyer, A., Vaughn, E.: Training tactical-level planning skills: an investigation of problem-centered and direct instruction approaches. Technical report, U. S. Army Research Institute for the Behavioral & Social Sciences, Arlington (2011). http://www.dtic.mil/dtic/tr/fulltext/u2/a545362.pdf

  40. Whittle, R.: MUM-T is the word for AH-64E: helos fly, use drones. Breaking Defense, January 2015

    Google Scholar 

  41. Zaal, P.M.T., Sweet, B.T.: The challenges of measuring transfer of stall recovery training. In: 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3138–3143, October 2014. https://doi.org/10.1109/SMC.2014.6974410, ISSN 1062-922X

Download references

Acknowledgments

We thank Hugh R. R. Dougherty, Caileigh Fitzgerald, Paul Robinette, Michael R. Benjamin, and Henrik Schmidt for their contributions to this initial work with Gazebo and Meshcat-python. We also thank Clearpath Robotics for furnishing us with a model of their Heron M300 autonomous surface vehicle.

Author information

Authors and Affiliations

  1. United States Military Academy, West Point, NY, 10996, USA

    Michael Novitzky, Nicholas H. Franck, Christa M. Chewar & Christopher Korpela

  2. Naval Postgraduate School, Monterey, CA, 93943, USA

    Rob Semmens

Authors
  1. Michael Novitzky
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rob Semmens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicholas H. Franck
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christa M. Chewar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christopher Korpela
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Novitzky .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA

    Jessie Y. C. Chen

  2. U.S. Army Combat Capabilities Development Command, Orlando, FL, USA

    Gino Fragomeni

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Novitzky, M., Semmens, R., Franck, N.H., Chewar, C.M., Korpela, C. (2020). Virtual Reality for Immersive Human Machine Teaming with Vehicles. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality. Design and Interaction. HCII 2020. Lecture Notes in Computer Science(), vol 12190. Springer, Cham. https://doi.org/10.1007/978-3-030-49695-1_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49695-1_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49694-4

  • Online ISBN: 978-3-030-49695-1

  • eBook Packages: Computer ScienceComputer Science (R0)