Multi-modal active perception for information gathering in science missions

  • Akash AroraEmail author
  • P. Michael Furlong
  • Robert Fitch
  • Salah Sukkarieh
  • Terrence Fong


Robotic science missions in remote environments, such as deep ocean and outer space, can involve studying phenomena that cannot directly be observed using on-board sensors but must be deduced by combining measurements of correlated variables with domain knowledge. Traditionally, in such missions, robots passively gather data along prescribed paths, while inference, path planning, and other high level decision making is largely performed by a supervisory science team located at a different location, often at a great distance. However, communication constraints hinder these processes, and hence the rate of scientific progress. This paper presents an active perception approach that aims to reduce robots’ reliance on human supervision and improve science productivity by encoding scientists’ domain knowledge and decision making process on-board. We present a Bayesian network architecture to compactly model critical aspects of scientific knowledge while remaining robust to observation and modeling uncertainty. We then formulate path planning and sensor scheduling as an information gain maximization problem, and propose a sampling-based solution based on Monte Carlo tree search to plan informative sensing actions which exploit the knowledge encoded in the network. The computational complexity of our framework does not grow with the number of observations taken and allows long horizon planning in an anytime manner, making it highly applicable to field robotics with constrained computing. Simulation results show statistically significant performance improvements over baseline methods, and we validate the practicality of our approach through both hardware experiments and simulated experiments with field data gathered during the NASA Mojave Volatiles Prospector science expedition.


Informative sensing planning Active perception Robotic exploration Space robotics 



This work was supported by the Australian Center for Field Robotics and NASA Mojave Volatiles Prospector (MVP) project in the NASA Moon and Mars Analog Missions Activities (MMAMA) Program. The authors would also like to thank Richard Elphic at NASA Ames Research Center for his help in understanding the Neutron Spectrometer System data.


  1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Süsstrunk, S. (2012). SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis, 34(11), 2274–2282.Google Scholar
  2. Andrews, D. R., Colaprete, A., Quinn, J., Chavers, D., & Picard, M. (2014). Introducing the resource prospector (RP) mission. In Proceedings of of AIAA SPACE, p. 4378.Google Scholar
  3. Apostolopoulos, D. S., Pedersen, L., Shamah, B. N., Shillcutt, K., Wagner, M. D., & Whittaker, W. L. (2001). Robotic antarctic meteorite search: Outcomes. In Proceedings of IEEE ICRA, pp. 4174–4179.Google Scholar
  4. Arora, A., Fitch, R., & Sukkarieh, S. (2017a). An approach to autonomous science by modeling geological knowledge in a Bayesian framework. In Proceedings of IEEE/RSJ IROS Google Scholar
  5. Arora, A., Furlong, M., Fitch, R., Fong, T., Sukkarieh, S., & Elphic, R. (2017b). Online multi-modal learning and adaptive informative trajectory planning for autonomous exploration. In Proceedings of FSR Google Scholar
  6. Arora, A., Furlong, M., Wong Uland, F. R., & Fong, T. (2018). Sampling-based descent trajectory planning and autonomous landing site selection for icy moon lander missions. In Proceedings of iSAIRAS.Google Scholar
  7. Azman, K. (2005). Incorporating prior knowledge into Gaussian process models. In Proceedings of international Ph.D. work. Systems and control, pp. 253–256.Google Scholar
  8. Bai, H., Cai, S., Ye, N., Hsu, D., & Lee, W. S. (2015). Intention-aware online POMDP planning for autonomous driving in a crowd. In Proceedings of IEEE ICRA. IEEE, pp. 454–460.Google Scholar
  9. Bajcsy, R. (1988). Active perception. Proceedings of IEEE, 76(8), 966–1005.Google Scholar
  10. Barnes, D., Pugh, S., & Tyler, L. (2009). Autonomous science target identification and acquisition (ASTIA) for planetary exploration. In Proceedings of IEEE/RSJ IROS, pp. 3329–3335.Google Scholar
  11. Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference. University of London United KingdomGoogle Scholar
  12. Bender, A., Williams, S. B., & Pizarro, O. (2013). Autonomous exploration of large-scale benthic environments. In Proceedings of IEEE ICRA, pp. 390–396.Google Scholar
  13. Best, G., Cliff, O. M., Patten, T., Mettu, R. R., & Fitch, R. (2016). Decentralised Monte Carlo tree search for active perception. In Proceedings of WAFR.Google Scholar
  14. Best, G., & Fitch, R. (2016) . Probabilistic maximum set cover with path constraints for informative path planning. In Proceedings of ARAA ACRA.Google Scholar
  15. Binney, J., & Sukhatme, G. S. (2012). Branch and bound for informative path planning. In Proceedings of IEEE ICRA, pp. 2147–2154.Google Scholar
  16. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.zbMATHGoogle Scholar
  17. Brachman, R. J., & Schmolze, J. G. (1985). An overview of the KL-ONE knowledge representation system. Cognitive Science, 9(2), 171–216.Google Scholar
  18. Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., et al. (2012). A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI, 4(1), 1–43.Google Scholar
  19. Cabrol, N. A., Wettergreen, D., Warren-Rhodes, K., Grin, E. A., Moersch, J., Diaz, G. C., et al. (2007). Life in the Atacama: Searching for life with rovers (science overview). Journal of Geophysical Research: Biogeosciences, 112(G4), G04S02.
  20. Candela, A., Thompson, D., Dobrea, E. N., & Wettergreen, D. (2017). Planetary robotic exploration driven by science hypotheses for geologic mapping. In Proceedings of IEEE/RSJ IROS.Google Scholar
  21. Cassandra, A. R. (1998). Exact and approximate algorithms for partially observable Markov decision processes. Brown University.Google Scholar
  22. Castano, R., Estlin, T., Anderson, R. C., Gaines, D. M., Castano, A., Bornstein, B., et al. (2007). OASIS: Onboard autonomous science investigation system for opportunistic rover science. Journal of Field Robotics, 24(5), 379–397.Google Scholar
  23. Castillo, E., Gutierrez, J. M., & Hadi, A. S. (2012). Expert systems and probabilistic network models. Berlin: Springer.zbMATHGoogle Scholar
  24. Chien, S., Sherwood, R., Tran, D., Cichy, B., Rabideau, G., Castano, R., & et al. (2004). The EO-1 autonomous science agent. In Proceedings of international joint conference on autonomous agents and multiagent systems. IEEE Computer Society, pp. 420–427.Google Scholar
  25. Choset, H. (2001). Coverage for robotics—A survey of recent results. Annals of Mathematics and Artificial Intelligence, 31(1), 113–126.zbMATHGoogle Scholar
  26. Das, J., Harvey, J., Py, F., Vathsangam, H., Graham, R., Rajan, K., & Sukhatme, G. S. (2013). Hierarchical probabilistic regression for AUV-based adaptive sampling of marine phenomena. In Proceedings of IEEE ICRA, pp. 5571–5578.Google Scholar
  27. Dunbabin, M., & Marques, L. (2012). Robots for environmental monitoring: Significant advancements and applications. IEEE Robotics and Automation Magazine, 19(1), 24–39.Google Scholar
  28. Dunlop, H., Thompson, D. R., & Wettergreen, D. (2007). Multi-scale features for detection and segmentation of rocks in Mars images. In Proceedings of IEEE CVPR Google Scholar
  29. Ellery, A. A. (2018). Robotic astrobiology - prospects for enhancing scientific productivity of Mars rover missions. International Journal of Astrobiology, 17(3), 203–217. Scholar
  30. Estlin, T. A., Bornstein, B. J., Gaines, D. M., Anderson, R. C., Thompson, D. R., Burl, M., et al. (2012). AEGIS automated science targeting for the MER opportunity rover. ACM Transactions on Intelligent Systems and Technology, 3(3), 50.Google Scholar
  31. Foil, G., Fong, T., Elphic, R. C., & Wettergreen, D. (2016). Physical process models for improved rover mapping. In Proceedings of iSAIRAS.Google Scholar
  32. Francis, R., Estlin, T., Doran, G., Johnstone, S., Gaines, D., Verma, V., et al. (2017). AEGIS autonomous targeting for ChemCam on Mars Science Laboratory: Deployment and results of initial science team use. Science Robototics, 2(7), eaan4582.Google Scholar
  33. Gallant, M. J., Ellery, A., & Marshall, J. A. (2013). Rover-based autonomous science by probabilistic identification and evaluation. Journal of Intelligent and Robotic Systems, 72(3–4), 591.Google Scholar
  34. Gelly, S., Kocsis, L., Schoenauer, M., Sebag, M., Silver, D., Szepesvári, C., et al. (2012). The grand challenge of computer Go: Monte Carlo tree search and extensions. Communications of ACM, 55(3), 106–113.Google Scholar
  35. Girdhar, Y., & Dudek, G. (2016). Modeling curiosity in a mobile robot for long-term autonomous exploration and monitoring. Autonomous Robots, 40(7), 1267–1278.Google Scholar
  36. Grotzinger, J. P., Crisp, J., Vasavada, A. R., Anderson, R. C., Baker, C. J., Barry, R., et al. (2012). Mars science laboratory mission and science investigation. Space Science Reviews, 170(1–4), 5–56.Google Scholar
  37. Hanheide, M., Gretton, C., Dearden, R., Hawes, N., Wyatt, J., Pronobis, A., & et al. (2011). Exploiting probabilistic knowledge under uncertain sensing for efficient robot behaviour. In Proceedings of IJCAI, pp. 2442–2449.Google Scholar
  38. Heckerman, D., Geiger, D., & Chickering, D. M. (1995). Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20(3), 197–243.zbMATHGoogle Scholar
  39. Heldmann, J., Colaprete, A., Cook, A., Roush, T., Deans, M., Elphic, R., & et al. (2015). Mojave volatiles prospector (MVP): Science and operations results from a lunar polar rover analog field campaign, Vol. 46. In Proceedings of lunar planetary science, p. 2165.Google Scholar
  40. Hollinger, G. A., & Sukhatme, G. S. (2014). Sampling-based robotic information gathering algorithms. International Journal of Robotics Research, 33(9), 1271–1287.Google Scholar
  41. Hunter, A., & Liu, W. (2010). A survey of formalisms for representing and reasoning with scientific knowledge. The Knowledge Engineering Review, 25(02), 199–222.Google Scholar
  42. Jawaid, S. T., & Smith, S. L. (2015). Informative path planning as a maximum traveling salesman problem with submodular rewards. Discrete Applied Mathematics, 186, 112–127.MathSciNetzbMATHGoogle Scholar
  43. Knight, S., Rabideau, G., Chien, S., Engelhardt, B., & Sherwood, R. (2001). CASPER: Space exploration through continuous planning. IEEE Intelligent Systems, 16(5), 70–75.Google Scholar
  44. Kocsis, L., & Szepesvári, C. (2006). Bandit-based Monte Carlo planning. In Proceedings of European conference machine learning, pp. 282–293.Google Scholar
  45. Kononenko, I. (1993). Inductive and Bayesian learning in medical diagnosis. Applied Artificial Intelligence, 7(4), 317–337.Google Scholar
  46. Krause, A., & Guestrin, C. (2007). Near-optimal observation selection using submodular functions. In Proceedings of AAAI, pp. 1650–1654.Google Scholar
  47. Lawrence, N. D. (2003). Gaussian process latent variable models for visualisation of high dimensional data. In Proceedings of the 16th international conference on neural information processing systems (pp. 329–336). MIT Press.Google Scholar
  48. Leshin, L., Mahaffy, P., Webster, C., Cabane, M., Coll, P., Conrad, P., et al. (2013). Volatile, isotope, and organic analysis of Martian fines with the Mars Curiosity rover. Science, 341(6153), 1238937.Google Scholar
  49. Liedloff, A., Woodward, E., Harrington, G., & Jackson, S. (2013). Integrating indigenous ecological and scientific hydro-geological knowledge using a Bayesian network in the context of water resource development. Journal of Hydrology, 499, 177–187.Google Scholar
  50. Ma, K.-C., Liu, L., Heidarsson, H. K., & Sukhatme, G. S. (2018). Data-driven learning and planning for environmental sampling. Journal of Field Robotics, 35(5), 643–661.Google Scholar
  51. Marchant, R., & Ramos, F. (2014). Bayesian optimization for informative continuous path planning. In Proceedings of IEEE ICRA, pp. 6136–6143.Google Scholar
  52. Murphy, K. P., Weiss, Y., & Jordan, M. I. (1999). Loopy belief propagation for approximate inference: An empirical study. In Proceedings of the uncertainty in artificial intelligence, Morgan Kaufmann Publishers Inc, pp. 467–475.Google Scholar
  53. Nguyen, J. L., Lawrance, N. R., Fitch, R., & Sukkarieh, S. (2016). Real-time path planning for long-term information gathering with an aerial glider. Autonomous Robots, 40(6), 1017–1039.Google Scholar
  54. Patten, T., Martens, W., & Fitch, R. (2018). Monte Carlo planning for active object classification. Autonomous Robots, 42(2), 391–421.Google Scholar
  55. Pearl, J. (2014). Probabilistic reasoning in intelligent systems: Networks of plausible inference. Burlington: Morgan Kaufmann.zbMATHGoogle Scholar
  56. Pedersen, L. (2001). Autonomous characterization of unknown environments. In Proceedings of IEEE ICRA, pp. 277–284.Google Scholar
  57. Peynot, T., Lui, S. T., McAllister, R., Fitch, R., & Sukkarieh, S. (2014). Learned stochastic mobility prediction for planning with control uncertainty on unstructured terrain. Journal of Field Robotics, 31(6), 969–995.Google Scholar
  58. Post, M. A., Li, J., & Quine, B. M. (2016). Planetary micro-rover operations on Mars using a Bayesian framework for inference and control. Acta Astronautica, 120, 295–314.Google Scholar
  59. Potiris, S., Tompkins, A., & Goktogan, A. (2014). Terrain-based path planning and following for an experimental Mars rover. In Proceedings of ARAA ACRA.Google Scholar
  60. Rasmussen, C. E. (2006). Gaussian processes for machine learning. Cambridge: MIT Press.zbMATHGoogle Scholar
  61. Roy, N., Gordon, G., & Thrun, S. (2005). Finding approximate pomdp solutions through belief compression. JAIR, 23, 1–40.zbMATHGoogle Scholar
  62. Sharif, H., Ralchenko, M., Samson, C., & Ellery, A. (2015). Autonomous rock classification using Bayesian image analysis for rover-based planetary exploration. Computers & Geosciences, 83, 153–167.Google Scholar
  63. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484–489.Google Scholar
  64. Silver, D., & Veness, J. (2010). Monte Carlo planning in large POMDPs. In Advances in neural information processing, pp. 2164–2172.Google Scholar
  65. Smith, T. (2007). Probabilistic planning for robotic exploration. Ph.D. thesis, Robotics Institute, Carnegie Mellon University.Google Scholar
  66. Song, Y., & Shan, J. (2008). Automated rock segmentation for Mars exploration rover imagery. In Proceedings of lunar and planetary science conference.Google Scholar
  67. Sowa, J. F. (2014). Principles of semantic networks: Explorations in the representation of knowledge. Burlington: Morgan Kaufmann.zbMATHGoogle Scholar
  68. Tabib, W., Whittaker, R., & Michael, N. (2016). Efficient multi-sensor exploration using dependent observations and conditional mutual information. In Proceedings of IEEE SSRR, pp. 42–47.Google Scholar
  69. Thompson, D. R., & Castano, R. (2007). Performance comparison of rock detection algorithms for autonomous planetary geology. In Proceedings of IEEE aerospace conference.Google Scholar
  70. Thompson, D. R., & Wettergreen, D. (2008). Intelligent maps for autonomous kilometer-scale science survey. In Proceedings of iSAIRAS.Google Scholar
  71. Thompson, D. R., Wettergreen, D. S., & Peralta, F. J. C. (2011). Autonomous science during large-scale robotic survey. Journal of Field Robotics, 28(4), 542–564.Google Scholar
  72. Van Winnendael, M., Baglioni, P., & Vago, J. (2005). Development of the ESA ExoMars rover. In Proceedings of iSAIRAS, pp. 5–8.Google Scholar
  73. Wettergreen, D., Foil, G., Furlong, M., & Thompson, D. R. (2014). Science autonomy for rover subsurface exploration of the Atacama desert. AI Magazine, 35(4), 47–60.Google Scholar
  74. Woods, M., Shaw, A., Barnes, D., Price, D., Long, D., & Pullan, D. (2009). Autonomous science for an ExoMars Rover-like mission. Journal of Field Robotics, 26(4), 358–390.Google Scholar
  75. Yedidia, J. S., Freeman, W. T., Weiss, Y., et al. (2000). Generalized belief propagation. Advances in Neural Information, 13, 689–695.Google Scholar
  76. Zhang, S., Sridharan, M., & Wyatt, J. L. (2015). Mixed logical inference and probabilistic planning for robots in unreliable worlds. IEEE Transactions on Robotics, 31(3), 699–713.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Australian Center for Field RoboticsThe University of SydneySydneyAustralia
  2. 2.Intelligent Robotics GroupNASA Ames Research CenterMountain ViewUSA
  3. 3.Center for Autonomous SystemsUniversity of Technology SydneySydneyAustralia

Personalised recommendations