Enabling Situational Awareness via Augmented Reality of Autonomous Robot-Based Environmental Change Detection

  • Christopher ReardonEmail author
  • Jason Gregory
  • Carlos Nieto-Granda
  • John G. Rogers
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12190)


Accurately detecting changes in one’s environment is an important ability for many application domains, but can be challenging for humans. Autonomous robots can easily be made to autonomously detect metric changes in the environment, but unlike humans, understanding context can be challenging for robots. We present a novel system that uses an autonomous robot performing point cloud-based change detection to facilitate information-gathering tasks and provides enhanced situational awareness. The robotic system communicates detected changes via augmented reality to a human teammate for evaluation. We present results from a fielded system using two differently-equipped robots to examine implementation questions of point cloud density and its effect on visualization of changes. Our results show that there are trade-offs between implementations that we believe will be constructive towards similar systems in the future.


Human-robot teaming Augmented reality Simultaneous Localization and Mapping (SLAM) Change detection Field robotics 


  1. 1.
    Dellaert, F.: Factor graphs and GTSAM: a hands-on introduction. Technical report, Georgia Institute of Technology (2012)Google Scholar
  2. 2.
    Dellaert, F., Kaess, M.: Square Root SAM: simultaneous localization and mapping via square root information smoothing. Int. J. Robot. Res. 25(12), 1181–1203 (2006)CrossRefGoogle Scholar
  3. 3.
    Durlach, P.J.: Change blindness and its implications for complex monitoring and control systems design and operator training. Hum.-Comput. Interact. 19(4), 423–451 (2004)CrossRefGoogle Scholar
  4. 4.
    Gregory, J., et al.: Application of multi-robot systems to disaster-relief scenarios with limited communication. In: Wettergreen, D.S., Barfoot, T.D. (eds.) Field and Service Robotics. STAR, vol. 113, pp. 639–653. Springer, Cham (2016). Scholar
  5. 5.
    Marsland, S., Nehmzow, U., Shapiro, J.: On-line novelty detection for autonomous mobile robots. Robot. Auton. Syst. 51(2–3), 191–206 (2005)CrossRefGoogle Scholar
  6. 6.
    Núñez, P., Drews, P., Rocha, R., Campos, M., Dias, J.: Novelty detection and 3D shape retrieval based on Gaussian mixture models for autonomous surveillance robotics. In: 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4724–4730. IEEE (2009)Google Scholar
  7. 7.
    Pimentel, M.A., Clifton, D.A., Clifton, L., Tarassenko, L.: A review of novelty detection. Sig. Process. 99, 215–249 (2014)CrossRefGoogle Scholar
  8. 8.
    Quigley, M., Faust, J., Foote, T., Leibs, J.: ROS: an open-source Robot Operating System. In: ICRA Workshop on Open Source Software (2009)Google Scholar
  9. 9.
    Reardon, C., Lee, K., Fink, J.: Come see this! Augmented reality to enable human-robot cooperative search. In: Proceedings of the 2018 IEEE Symposium on Safety, Security, and Rescue Robotics (2018)Google Scholar
  10. 10.
    Reardon, C., Lee, K., Rogers, J.G., Fink, J.: Augmented reality for human-robot teaming in field environments. In: Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2019. LNCS, vol. 11575, pp. 79–92. Springer, Cham (2019). Scholar
  11. 11.
    Rusu, R.B., Cousins, S.: 3D is here: point cloud library PCL. In: 2011 IEEE International Conference on Robotics and Automation, pp. 1–4. IEEE (2011)Google Scholar
  12. 12.
    Segal, A., Haehnel, D., Thrun, S.: Generalized-ICP. In: Robotics: Science and Systems, vol. 2, p. 435 (2009)Google Scholar
  13. 13.
    Simons, D.J., Chabris, C.F.: Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception 28(9), 1059–1074 (1999)CrossRefGoogle Scholar
  14. 14.
    Sofman, B., Neuman, B., Stentz, A., Bagnell, J.A.: Anytime online novelty and change detection for mobile robots. J. Field Robot. 28(4), 589–618 (2011)CrossRefGoogle Scholar
  15. 15.
    Sturari, M., Paolanti, M., Frontoni, E., Mancini, A., Zingaretti, P.: Robotic platform for deep change detection for rail safety and security. In: 2017 European Conference on Mobile Robots (ECMR), pp. 1–6. IEEE (2017)Google Scholar
  16. 16.
    Szafir, D.: Mediating human-robot interactions with virtual, augmented, and mixed reality. In: Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2019. LNCS, vol. 11575, pp. 124–149. Springer, Cham (2019). Scholar
  17. 17.
    Trevor, A.J.B., Rogers, J.G., Christensen, H.I.: OmniMapper: a modular multimodal mapping framework. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 1983–1990, May 2014.
  18. 18.
    Vieira, A.W., Drews, P.L., Campos, M.F.: Spatial density patterns for efficient change detection in 3D environment for autonomous surveillance robots. IEEE Trans. Autom. Sci. Eng. 11(3), 766–774 (2014)CrossRefGoogle Scholar

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2020

Authors and Affiliations

  • Christopher Reardon
    • 1
    Email author
  • Jason Gregory
    • 1
  • Carlos Nieto-Granda
    • 2
  • John G. Rogers
    • 1
  1. 1.U.S. Army Research LaboratoryAdelphiUSA
  2. 2.University of California, San DiegoSan DeigoUSA

Personalised recommendations