Skip to main content
SpringerLink
Log in

MPDM: Multi-policy Decision-Making from Autonomous Driving to Social Robot Navigation

  • Chapter
  • First Online:
Control Strategies for Advanced Driver Assistance Systems and Autonomous Driving Functions

Part of the book series: Lecture Notes in Control and Information Sciences ((LNCIS,volume 476))

Abstract

This chapter presents multi-policy decision-making (MPDM): a novel approach to navigating in dynamic multi-agent environments. Rather than planning the trajectory of the robot explicitly, the planning process selects one of a set of closed-loop behaviors whose utility can be predicted through forward simulation that captures the complex interactions between the actions of these agents. These polices capture different high-level behavior and intentions, such as driving along a lane, turning at an intersection, or following pedestrians. We present two different scenarios where MPDM has been applied successfully: an autonomous driving environment models vehicle behavior for both our vehicle and nearby vehicles and a social environment, where multiple agents or pedestrians configure a dynamic environment for autonomous robot navigation. We present extensive validation for MPDM on both scenarios, using simulated and real-world experiments.

Alex G. Cunningham and Enric Galceran have contributed equally to this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In this paper, we use the term closed-loop policies to mean policies that react to the presence of other agents, in a coupled manner. The same concept applies to the term closed-loop forward simulation.

  2. 2.

    https://April.eecs.umich.edu/media/mehta2016iros.mp4.

References

  1. Galceran, E., Cunningham, A.G., Eustice, R.M., Olson, E.: Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction: theory and experiment. In: Autonomous Robots, pp. 1–16 (2017)

    Google Scholar 

  2. Mehta, D., Ferrer, G., Olson, E.: Autonomous navigation in dynamic social environments using multi-policy decision making. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1190–1197 (2016)

    Google Scholar 

  3. Cunningham, A.G., Galceran, E., Eustice, R.M., Olson, E.: MPDM: multipolicy decision-making in dynamic, uncertain environments for autonomous driving. In: Proceedings of the IEEE International Conference on Robotics and Automation. Seattle, WA, USA (2015)

    Google Scholar 

  4. Galceran, E., Cunningham, A.G., Eustice, R.M., Olson, E.: Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction. In: Proceedings of the Robotics: Science and Systems Conference. Rome, Italy (2015)

    Google Scholar 

  5. Choi, J., Eoh, G., Kim, J., Yoon, Y., Park, J., Lee, B.H.: Analytic collision anticipation technology considering agents’ future behavior. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1656–1661. Taipei, Taiwan (2010)

    Google Scholar 

  6. Ohki, T., Nagatani, K., Yoshida, K.: Collision avoidance method for mobile robot considering motion and personal spaces of evacuees. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1819–1824. Taipei, Taiwan (2010)

    Google Scholar 

  7. Petti, S., Fraichard, T.: Safe motion planning in dynamic environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2210–2215. Edmonton, AB, Canada (2005)

    Google Scholar 

  8. Du Toit, N., Burdick, J.: Robotic motion planning in dynamic, cluttered, uncertain environments. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 966–973. Anchorage, AK, USA (2010)

    Google Scholar 

  9. Fulgenzi, C., Tay, C., Spalanzani, A., Laugier, C.: Probabilistic navigation in dynamic environment using rapidly-exploring random trees and Gaussian processes. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1056–1062. Nice, France (2008)

    Google Scholar 

  10. Du Toit, N.E., Burdick, J.W.: Robot motion planning in dynamic, uncertain environments. IEEE Trans. Robot. 28(1), 101–115 (2012)

    Article  Google Scholar 

  11. Havlak, F., Campbell, M.: Discrete and continuous, probabilistic anticipation for autonomous robots in urban environments. IEEE Trans. Robot. 30(2), 461–474 (2014)

    Article  Google Scholar 

  12. Dagli, I., Brost, M., Breuel, G.: Agent technologies, infrastructures, tools, and applications for e-services. NODe 2002 Agent-Related Workshops. Chapter Action Recognition and Prediction for Driver Assistance Systems Using Dynamic Belief Networks, pp. 179–194. Springer, Berlin, Heidelberg (2003)

    Google Scholar 

  13. Gindele, T., Brechtel, S., Dillmann, R.: Learning driver behavior models from traffic observations for decision making and planning. In: IEEE Intelligent Transportation Systems Magazine, pp. 69–79 (2015)

    Article  Google Scholar 

  14. Broadhurst, A., Baker, S., Kanade, T.: Monte Carlo road safety reasoning. In: Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 319–324. Las Vegas, NV, USA (2005)

    Google Scholar 

  15. Ferguson, D., Darms, M., Urmson, C., Kolski, S.: Detection, prediction, and avoidance of dynamic obstacles in urban environments. In: Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 1149–1154. Eindhoven, Netherlands (2008)

    Google Scholar 

  16. Hardy, J., Campbell, M.: Contingency planning over probabilistic obstacle predictions for autonomous road vehicles. IEEE Trans. Robot. 29(4), 913–929 (2013)

    Article  Google Scholar 

  17. Joseph, J., Doshi-Velez, F., Huang, A.S., Roy, N.: A Bayesian nonparametric approach to modeling motion patterns. Auton. Robots 31(4), 383–400 (2011)

    Article  Google Scholar 

  18. Kim, K., Lee, D., Essa, I.: Gaussian process regression flow for analysis of motion trajectories. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1164–1171. Barcelona, Spain (2011)

    Google Scholar 

  19. Trautman, P., Krause, A.: Unfreezing the robot: Navigation in dense, interacting crowds. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 797–803. Taipei, Taiwan (2010)

    Google Scholar 

  20. Aoude, G.S., Luders, B.D., Joseph, J.M., Roy, N., How, J.P.: Probabilistically safe motion planning to avoid dynamic obstacles with uncertain motion patterns. Auton. Robots 35(1), 51–76 (2013)

    Article  Google Scholar 

  21. Tran, Q., Firl, J.: Modelling of traffic situations at urban intersections with probabilistic non-parametric regression. In: Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 334–339. Gold Coast City, Australia (2013)

    Google Scholar 

  22. Tran, Q., Firl, J.: Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression. In: Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 918–923. Dearborn, MI, USA (2014)

    Google Scholar 

  23. Kuderer, M., Gulati, S., Burgard, W.: Learning driving styles for autonomous vehicles from demonstration. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 2641–2646 (2015)

    Google Scholar 

  24. DARPA: DARPA Urban Challenge. http://archive.darpa.mil/grandchallenge/ (2007)

  25. Montemerlo, M., et al.: Junior: the Stanford entry in the urban challenge. J. Field Robot. 25(9), 569–597 (2008)

    Article  Google Scholar 

  26. Miller, I., et al.: Team Cornell’s skynet: robust perception and planning in an urban environment. J. Field Robot. 25(8), 493–527 (2008)

    Article  Google Scholar 

  27. Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M.N., Dolan, J., Duggins, D., Galatali, T., Geyer, C., Gittleman, M., Harbaugh, S., Hebert, M., Howard, T.M., Kolski, S., Kelly, A., Likhachev, M., McNaughton, M., Miller, N., Peterson, K., Pilnick, B., Rajkumar, R., Rybski, P., Salesky, B., Seo, Y.W., Singh, S., Snider, J., Stentz, A., Whittaker, W., Wolkowicki, Z., Ziglar, J., Bae, H., Brown, T., Demitrish, D., Litkouhi, B., Nickolaou, J., Sadekar, V., Zhang, W., Struble, J., Taylor, M., Darms, M., Ferguson, D.: Autonomous driving in urban environments: boss and the Urban Challenge. J. Field Robot. 25(8), 425–466 (2008)

    Article  Google Scholar 

  28. Ferguson, D., Howard, T.M., Likhachev, M.: Motion planning in urban environments. J. Field Robot. 25(11–12), 939–960 (2008)

    Article  Google Scholar 

  29. Werling, M., Ziegler, J., Kammel, S., Thrun, S.: Optimal trajectory generation for dynamic street scenarios in a frenet frame. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 987–993. Anchorage, AK, USA (2010)

    Google Scholar 

  30. Xu, W., Wei, J., Dolan, J., Zhao, H., Zha, H.: A real-time motion planner with trajectory optimization for autonomous vehicles. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 2061–2067. Saint Paul, MN, USA (2012)

    Google Scholar 

  31. Bai, H., Hsu, D., Lee, W.S.: Integrated perception and planning in the continuous space: a POMDP approach. Int. J. Robot. Res. 33(9), 1288–1302 (2014)

    Article  Google Scholar 

  32. Kurniawati, H., Hsu, D., Lee, W.: SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In: Proceedings of the Robotics: Science and Systems Conference. Zurich, Switzerland (2008)

    Google Scholar 

  33. Silver, D., Veness, J.: Monte-Carlo planning in large POMDPs. In: Lafferty J., Williams, C., Shawe-Taylor, J., Zemel, R., Culotta, A. (eds.) Advances in Neural Information Processing Systems, vol. 23, pp. 2164–2172. Curran Associates, Inc. (2010)

    Google Scholar 

  34. Madani, O., Hanks, S., Condon, A.: On the undecidability of probabilistic planning and related stochastic optimization problems. Artif. Intell. 147(1–2), 5–34 (2003)

    Article  MathSciNet  Google Scholar 

  35. Papadimitriou, C.H., Tsitsiklis, J.N.: The complexity of Markov decision processes. Math. Oper. Res. 12(3), 441–450 (1987)

    Article  MathSciNet  Google Scholar 

  36. Thrun, S.: Monte Carlo POMDPs. In: Proceedings of the Advances in Neural Information Processing Systems Conference pp. 1064–1070 (2000)

    Google Scholar 

  37. Candido, S., Davidson, J., Hutchinson, S.: Exploiting domain knowledge in planning for uncertain robot systems modeled as pomdps. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 3596–3603. Anchorage, AK, USA (2010)

    Google Scholar 

  38. Lee, T., Kim, Y.J.: Massively parallel motion planning algorithms under uncertainty using POMDP. Int. J. Robot. Res. 35(8), 928–942 (2016)

    Article  Google Scholar 

  39. Wei, J., Dolan, J.M., Snider, J.M., Litkouhi, B.: A point-based MDP for robust single-lane autonomous driving behavior under uncertainties. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 2586–2592. Shanghai, China (2011)

    Google Scholar 

  40. Ulbrich, S., Maurer, M.: Probabilistic online pomdp decision making for lane changes in fully automated driving. In: Proceedings of the IEEE Intelligent Transportation Systems Conference, pp. 2063–2067 (2013)

    Google Scholar 

  41. Brechtel, S., Gindele, T., Dillmann, R.: Probabilistic MDP-behavior planning for cars. In: Proceedings of the IEEE Intelligent Transportation Systems Conference, pp. 1537–1542 (2011)

    Google Scholar 

  42. Bandyopadhyay, T., Jie, C.Z., Hsu, D., Ang, M.H., Rus, D., Frazzoli, E.: In: Experimental Robotics: The 13th International Symposium on Experimental Robotics, Chapter Intention-Aware Pedestrian Avoidance, pp. 963–977. Springer (2013)

    Chapter  Google Scholar 

  43. Brechtel, S., Gindele, T., Dillmann, R.: Probabilistic decision-making under uncertainty for autonomous driving using continuous POMDPs. In: Proceedings of the IEEE Intelligent Transportation Systems Conference, pp. 392–399 (2014)

    Google Scholar 

  44. Bandyopadhyay, T., Won, K., Frazzoli, E., Hsu, D., Lee, W., Rus, D.: Intention-aware motion planning. In: Frazzoli, E., Lozano-Perez, T., Roy, N., Rus, D. (eds.) In: Proceedings of the International Workshop on the Algorithmic Foundations of Robotics. Springer Tracts in Advanced Robotics, vol. 86, pp. 475–491. Springer, Berlin, Heidelberg (2013)

    Google Scholar 

  45. He, R., Brunskill, E., Roy, N.: Efficient planning under uncertainty with macro-actions. J. Artif. Intell. Res. 40, 523–570 (2011)

    MATH  Google Scholar 

  46. Somani, A., Ye, N., Hsu, D., Lee, W.S.: DESPOT: Online POMDP planning with regularization. In: Burges, C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, vol. 26, pp. 1772–1780. Curran Associates, Inc. (2013)

    Google Scholar 

  47. van den Berg, J., Guy, S.J., Lin, M., Manocha, D.: Reciprocal n-body collision avoidance. Robotics Research, Springer Tracts in Advanced Robotics 70, 3–19 (2011)

    Article  Google Scholar 

  48. Guy, S.J., Chhugani, J., Kim, C., Satish, N., Lin, M., Manocha, D., Dubey, P.: Clearpath: highly parallel collision avoidance for multi-agent simulation. In: Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pp. 177–187. ACM (2009)

    Google Scholar 

  49. Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. Int. J. Robot. Res. 5(1), 90–98 (1986)

    Article  Google Scholar 

  50. Ferrer, G., Garrell, A., Sanfeliu, A.: Social-aware robot navigation in urban environments. In: European Conference on Mobile Robotics, pp. 331–336 (2013)

    Google Scholar 

  51. Sisbot, E.A., Marin-Urias, L.F., Alami, R., Simeon, T.: A human aware mobile robot motion planner. IEEE Trans. Robot. 23(5), 874–883 (2007)

    Article  Google Scholar 

  52. Svenstrup, M., Bak, T., Andersen, H.J.: Trajectory planning for robots in dynamic human environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4293–4298 (2010)

    Google Scholar 

  53. Huang, W.H., Fajen, B.R., Fink, J.R., Warren, W.H.: Visual navigation and obstacle avoidance using a steering potential function. Robot. Auton. Syst. 54(4), 288–299 (2006)

    Article  Google Scholar 

  54. Helbing, D., Molnár, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)

    Article  Google Scholar 

  55. Koren, Y., Borenstein, J.: Potential field methods and their inherent limitations for mobile robot navigation. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 1398–1404 (1991)

    Google Scholar 

  56. Brock, O., Khatib, O.: High-speed navigation using the global dynamic window approach. Proceedings of the IEEE International Conference on Robotics and Automation 1, 341–346 (1999)

    Article  Google Scholar 

  57. Kretzschmar, H., Spies, M., Sprunk, C., Burgard, W.: Socially compliant mobile robot navigation via inverse reinforcement learning. Int. J. Robot. Res. (2016)

    Google Scholar 

  58. Kuderer, M., Kretzschmar, H., Sprunk, C., Burgard, W.: Feature-based prediction of trajectories for socially compliant navigation. In: Proceedings of Robotics: Science and Systems (RSS) (2012)

    Google Scholar 

  59. Luber, M., Spinello, L., Silva, J., Arras, K.O.: Socially-aware robot navigation: a learning approach. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 902–907 (2012)

    Google Scholar 

  60. Ziebart, B.D., Ratliff, N., Gallagher, G., Mertz, C., Peterson, K., Bagnell, J.A., Hebert, M., Dey, A.K., Srinivasa, S.: Planning-based prediction for pedestrians. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3931–3936 (2009)

    Google Scholar 

  61. Ferrer, G., Garrell, A., Herrero, F., Sanfeliu, A.: Robot social-aware navigation framework to accompany people walking side-by-side. In: Autonomous Robots, pp. 1–19 (2016)

    Article  Google Scholar 

  62. Kuderer, M., Burgard, W.: An approach to socially compliant leader following for mobile robots. In: International Conference on Social Robotics, pp. 239–248. Springer (2014)

    Google Scholar 

  63. Stein, P., Spalanzani, A., Santos, V., Laugier, C.: Leader following: a study on classification and selection. Robot. Auton. Syst. 75(Part A), 79 – 95 (2016)

    Google Scholar 

  64. Ferrer, G.: Social robot navigation in urban dynamic environments. Ph.D. thesis, Universitat Politèctnica de Catalunya, Spain (October 2015)

    Google Scholar 

  65. Ferrer, G., Sanfeliu, A.: Multi-objective cost-to-go functions on robot navigation in dynamic environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3824–3829 (2015)

    Google Scholar 

  66. Fulgenzi, C., Spalanzani, A., Laugier, C.: Probabilistic motion planning among moving obstacles following typical motion patterns. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4027–4033. IEEE (2009)

    Google Scholar 

  67. Trautman, P., Ma, J., Murray, R.M., Krause, A.: Robot navigation in dense human crowds: Statistical models and experimental studies of human-robot cooperation. Int. J. Robot. Res. 34(3), 335–356 (2015)

    Article  Google Scholar 

  68. Foka, A., Trahanias, P.: Probabilistic Autonomous Robot Navigation in Dynamic Environments with Human Motion Prediction. Int. J. Soc. Robot. 2(1), 79–94 (2010). https://doi.org/10.1007/s12369-009-0037-z

    Article  Google Scholar 

  69. Galceran, E., Olson, E., Eustice, R.M.: Augmented vehicle tracking under occlusions for decision-making in autonomous driving. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3559–3565. Hamburg, Germany (2015)

    Google Scholar 

  70. Huang, A.S., Olson, E., Moore, D.C.: LCM: lightweight communications and marshalling. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4057–4062 (2010)

    Google Scholar 

  71. Olson, E., Strom, J., Morton, R., Richardson, A., Ranganathan, P., Goeddel, R., Bulic, M., Crossman, J., Marinier, B.: Progress toward multi-robot reconnaissance and the magic 2010 competition. J. Field Robot. 29(5), 762–792 (2012)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by a grant from Ford Motor Company via the Ford-UM Alliance under award N015392, DARPA YIP grant under award D13AP00059, CyberSEES grant award 1442773, and ARIA (TRI) grant award N021563.

Parts of this work have been previously published in [1] which is under Copyright by Springer, 2017. These parts are reused with the permission of Springer which is acknowledged with high appreciation.

Author information

Authors and Affiliations

  1. Toyota Research Institute, 2311 Green Rd, Ann Arbor, MI, 48105, USA

    Alex G. Cunningham

  2. Autonomous Systems Lab, Institute of Robotics and Intelligent Systems, ETH Zurich, Leonhardstrasse 21, 8092, Zurich, Switzerland

    Enric Galceran

  3. Department of Computer Science and Engineering, University of Michigan, 2260 Hayward St, Ann Arbor, MI, 48109, USA

    Dhanvin Mehta, Gonzalo Ferrer & Edwin Olson

  4. Department of Naval Architecture and Marine Engineering, University of Michigan, 2600 Draper Dr, Ann Arbor, MI, 48109, USA

    Ryan M. Eustice

Authors
  1. Alex G. Cunningham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Enric Galceran
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dhanvin Mehta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gonzalo Ferrer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ryan M. Eustice
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Edwin Olson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dhanvin Mehta .

Editor information

Editors and Affiliations

  1. Institute for Design and Control of Mechatronical Systems, Johannes Kepler University Linz, Linz, Austria

    Harald Waschl

  2. Department of Aerospace Engineering, University of Michigan, Ann Arbor, Michigan, USA

    Ilya Kolmanovsky

  3. Department of Mechanical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands

    Frank Willems

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cunningham, A.G., Galceran, E., Mehta, D., Ferrer, G., Eustice, R.M., Olson, E. (2019). MPDM: Multi-policy Decision-Making from Autonomous Driving to Social Robot Navigation. In: Waschl, H., Kolmanovsky, I., Willems, F. (eds) Control Strategies for Advanced Driver Assistance Systems and Autonomous Driving Functions . Lecture Notes in Control and Information Sciences, vol 476. Springer, Cham. https://doi.org/10.1007/978-3-319-91569-2_10

Download citation

Publish with us

Policies and ethics

Search

Navigation