Skip to main content
SpringerLink
Log in

Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction: Theory and experiment

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This paper reports on an integrated inference and decision-making approach for autonomous driving that models vehicle behavior for both our vehicle and nearby vehicles as a discrete set of closed-loop policies. Each policy captures a distinct high-level behavior and intention, such as driving along a lane or turning at an intersection. We first employ Bayesian changepoint detection on the observed history of nearby cars to estimate the distribution over potential policies that each nearby car might be executing. We then sample policy assignments from these distributions to obtain high-likelihood actions for each participating vehicle, and perform closed-loop forward simulation to predict the outcome for each sampled policy assignment. After evaluating these predicted outcomes, we execute the policy with the maximum expected reward value. We validate behavioral prediction and decision-making using simulated and real-world experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. In this paper, we use the term closed-loop policies to mean policies that react to the presence of other traffic participants, in a coupled manner. The same concept applies to the term closed-loop simulation.

References

  • Aoude, G. S., Luders, B. D., Joseph, J. M., Roy, N., & How, J. P. (2013). Probabilistically safe motion planning to avoid dynamic obstacles with uncertain motion patterns. Autonomous Robots, 35(1), 51–76.

    Article  Google Scholar 

  • Bai, H., Hsu, D., & Lee, W. S. (2014). Integrated perception and planning in the continuous space: A POMDP approach. International Journal of Robotics Research, 33(9), 1288–1302.

    Article  Google Scholar 

  • Bandyopadhyay, T., Jie, C. Z., Hsu, D., Ang, M. H., Rus, D., & Frazzoli, E. (2013a). In Experimental robotics: The 13th international symposium on experimental robotics (pp. 963–977). Springer International Publishing, chap Intention-Aware Pedestrian Avoidance.

  • Bandyopadhyay, T., Won, K., Frazzoli, E., Hsu, D., Lee, W., & Rus, D. (2013b). Intention-aware motion planning. In E. Frazzoli, T. Lozano-Perez, N. Roy, & D. Rus (Eds.), Proceedings of the international workshop on the algorithmic foundations of robotics, Springer tracts in advanced robotics (Vol. 86, pp. 475–491). Berlin: Springer.

    Google Scholar 

  • Bishop, C. M. (2007). Pattern recognition and machine learning. Information science and statistics. Berlin: Springer.

    Google Scholar 

  • Brechtel, S., Gindele, T., & Dillmann, R. (2011). Probabilistic MDP-behavior planning for cars. In Proceedings of the IEEE intelligent transportation systems conference (pp. 1537–1542).

  • Brechtel, S., Gindele, T., & Dillmann, R. (2014). Probabilistic decision-making under uncertainty for autonomous driving using continuous POMDPs. In Proceedings of the IEEE intelligent transportation systems conference (pp. 392–399).

  • Broadhurst, A., Baker, S., & Kanade, T. (2005). Monte carlo road safety reasoning. In Proceedings of the IEEE intelligent vehicles symposium (pp. 319–324). Las Vegas, NV: IEEE.

  • Candido, S., Davidson, J., & Hutchinson, S. (2010). Exploiting domain knowledge in planning for uncertain robot systems modeled as pomdps. In Proceedings of the IEEE international conference on robotics and automation (pp. 3596–3603). Anchorage, AK: IEEE.

  • Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 15.

    Article  Google Scholar 

  • Choi, J., Eoh, G., Kim, J., Yoon, Y., Park, J., & Lee, B. H. (2010). Analytic collision anticipation technology considering agents’ future behavior. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1656–1661). Taipei, Taiwan: IEEE.

  • Cunningham, A. G., Galceran, E., Eustice, R. M., & Olson, E. (2015). MPDM: Multipolicy decision-making in dynamic, uncertain environments for autonomous driving. In Proceedings of the IEEE international conference on robotics and automation, Seattle, WA.

  • Dagli, I., Brost, M., & Breuel, G. (2003). Agent technologies, infrastructures, tools, and applications for E-Services: NODe 2002 agent-related workshops. Springer Berlin Heidelberg, Chap Action Recognition and Prediction for Driver Assistance Systems Using Dynamic Belief Networks, pp. 179–194.

  • DARPA (2007) DARPA Urban Challenge. http://archive.darpa.mil/grandchallenge/

  • Du Toit, N., & Burdick, J. (2010). Robotic motion planning in dynamic, cluttered, uncertain environments. In Proceedings of the IEEE international conference on robotics and automation (pp. 966–973). Anchorage, AK: IEEE.

  • Du Toit, N. E., & Burdick, J. W. (2012). Robot motion planning in dynamic, uncertain environments. IEEE Transactions on Robotics, 28(1), 101–115.

    Article  Google Scholar 

  • Fearnhead, P., & Liu, Z. (2007). On-line inference for multiple changepoint problems. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 69(4), 589–605.

    Article  MathSciNet  Google Scholar 

  • Ferguson, D., Darms, M., Urmson, C., & Kolski, S. (2008a). Detection, prediction, and avoidance of dynamic obstacles in urban environments. In Proceedings of the IEEE intelligent vehicles symposium (pp. 1149–1154). Eindhoven, Netherlands: IEEE.

  • Ferguson, D., Howard, T. M., & Likhachev, M. (2008b). Motion planning in urban environments. Journal of Field Robotics, 25(11–12), 939–960.

    Article  Google Scholar 

  • Fulgenzi, C., Tay, C., Spalanzani, A., & Laugier, C. (2008). Probabilistic navigation in dynamic environment using rapidly-exploring random trees and gaussian processes. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1056–1062). Nice, France: IEEE.

  • Galceran, E., Cunningham, A.G., Eustice, R.M., & Olson, E. (2015a). Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction. In Proceedings of the robotics: science & systems conference. Rome, Italy: Robotics: Science and Systems Foundation.

  • Galceran, E., Olson, E., & Eustice, R. M. (2015b). Augmented vehicle tracking under occlusions for decision-making in autonomous driving. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 3559–3565). Hamburg, Germany: IEEE.

  • Gindele, T., Brechtel, S., & Dillmann, R. (2015). Learning driver behavior models from traffic observations for decision making and planning. IEEE Intelligent Transportation Systems Magazine, 7, 69–79.

    Article  Google Scholar 

  • Hardy, J., & Campbell, M. (2013). Contingency planning over probabilistic obstacle predictions for autonomous road vehicles. IEEE Transactions on Robotics, 29(4), 913–929.

    Article  Google Scholar 

  • Havlak, F., & Campbell, M. (2014). Discrete and continuous, probabilistic anticipation for autonomous robots in urban environments. IEEE Transactions on Robotics, 30(2), 461–474.

    Article  Google Scholar 

  • He, R., Brunskill, E., & Roy, N. (2011). Efficient planning under uncertainty with macro-actions. Journal of Artificial Intelligence Research, 40, 523–570.

    MATH  Google Scholar 

  • Joseph, J., Doshi-Velez, F., Huang, A. S., & Roy, N. (2011). A Bayesian nonparametric approach to modeling motion patterns. Autonomous Robots, 31(4), 383–400.

    Article  Google Scholar 

  • Kim, K., Lee, D., & Essa, I. (2011). Gaussian process regression flow for analysis of motion trajectories. In Proceedings of the IEEE international conference on computer vision (pp. 1164–1171). Barcelona, Spain: IEEE.

  • Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. In Proceedings of the IEEE international conference on robotics and automation (pp 2641–2646). Seattle, WA: IEEE.

  • Kurniawati, H., Hsu, D., & Lee, W. (2008). SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In Proceedings of the robotics: Science & systems conference. Zurich, Switzerland: IEEE.

  • Lee, T., & Kim, Y. J. (2016). Massively parallel motion planning algorithms under uncertainty using POMDP. International Journal of Robotics Research, 35(8), 928–942.

    Article  Google Scholar 

  • Madani, O., Hanks, S., & Condon, A. (2003). On the undecidability of probabilistic planning and related stochastic optimization problems. Artificial Intelligence, 147(1–2), 5–34.

    Article  MathSciNet  MATH  Google Scholar 

  • Miller, I., et al. (2008). Team Cornell’s Skynet: Robust perception and planning in an urban environment. Journal of Field Robotics, 25(8), 493–527.

    Article  Google Scholar 

  • Montemerlo, M., et al. (2008). Junior: The stanford entry in the urban challenge. Journal of Field Robotics, 25(9), 569–597.

    Article  Google Scholar 

  • Niekum, S., Osentoski, S., Atkeson, C. G., & Barto, A. G. (2014). CHAMP: Changepoint detection using approximate model parameters. Tech. Rep. CMU-RI-TR-14-10, Robotics Institute, Carnegie Mellon University.

  • Niekum, S., Osentoski, S., Atkeson, C. G., & Barto, A. G. (2015). Online bayesian changepoint detection for articulated motion models. In Proceedings of the IEEE international conference on robotics and automation. Seattle, WA: IEEE.

  • Ohki, T., Nagatani, K., & Yoshida, K. (2010). Collision avoidance method for mobile robot considering motion and personal spaces of evacuees. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 1819–1824). Taipei, Taiwan: IEEE.

  • Papadimitriou, C. H., & Tsitsiklis, J. N. (1987). The complexity of Markov decision processes. Mathematics of Operations Research, 12(3), 441–450.

    Article  MathSciNet  MATH  Google Scholar 

  • Petti, S., & Fraichard, T. (2005). Safe motion planning in dynamic environments. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2210–2215). Edmonton, AB: IEEE.

  • Piciarelli, C., & Foresti, G. (2006). On-line trajectory clustering for anomalous events detection. Pattern Recognition Letters, 27(15), 1835–1842.

    Article  Google Scholar 

  • Silver, D., & Veness, J. (2010). Monte-carlo planning in large POMDPs. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, & A. Culotta (Eds.), Advances in neural information processing systems 23 (pp. 2164–2172). Red Hook, NY: Curran Associates Inc.

  • Somani, A., Ye, N., Hsu, D., & Lee, W. S. (2013). DESPOT: Online POMDP planning with regularization. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Weinberger (Eds.), Advances in neural information processing systems 26 (pp. 1772–1780). Red Hook, NY: Curran Associates Inc.

  • Thrun, S. (2000). Monte Carlo POMDPs. In Proceedings of the advances in neural information processing systems Conference (pp 1064–1070).

  • Tran, Q., & Firl, J. (2013). Modelling of traffic situations at urban intersections with probabilistic non-parametric regression. In Proceedings of the IEEE intelligent vehicles symposium (pp. 334–339). Gold Coast City, Australia: IEEE.

  • Tran, Q., & Firl, J. (2014). Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression. In Proceedings of the IEEE intelligent vehicles symposium (pp. 918–923). Dearborn, MI: IEEE.

  • Trautman, P., & Krause, A. (2010). Unfreezing the robot: Navigation in dense, interacting crowds. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 797–803). Taipei, Taiwan: IEEE.

  • Ulbrich, S., & Maurer, M. (2013). Probabilistic online pomdp decision making for lane changes in fully automated driving. In Proceedings of the IEEE intelligent transportation systems conference (pp 2063–2067).

  • Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M. N., et al. (2008). Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics, 25(8), 425–466.

    Article  Google Scholar 

  • Wei, J., Dolan, J. M., Snider, J. M., & Litkouhi, B. (2011). A point-based MDP for robust single-lane autonomous driving behavior under uncertainties. In Proceedings of the IEEE international conference on robotics and automation (pp. 2586–2592). Shanghai, China: IEEE.

  • Werling, M., Ziegler, J., Kammel, S., & Thrun, S. (2010). Optimal trajectory generation for dynamic street scenarios in a frenet frame. In Proceedings of the IEEE international conference on robotics and automation (pp. 987–993). Anchorage, AK: IEEE.

  • Xu, W., Wei, J., Dolan, J., Zhao, H., & Zha, H. (2012). A real-time motion planner with trajectory optimization for autonomous vehicles. In Proceedings of the IEEE international conference on robotics and automation (pp. 2061–2067). Saint Paul, MN: IEEE.

Download references

Acknowledgements

This work was supported in part by a grant from Ford Motor Company via the Ford-UM Alliance under award N015392 and in part by DARPA under award D13AP00059. The authors are sincerely grateful to Patrick Carmody, Ryan Wolcott, Steve Vozar, Jeff Walls, Gonzalo Ferrer, and Igor Gilitschenski for help collecting experimental data and for valuable comments.

Author information

Author notes

    Authors and Affiliations

    1. Autonomous Systems Lab, Institute of Robotics and Intelligent Systems, ETH Zurich, Leonhardstrasse 21, Zurich, 8092, Switzerland

      Enric Galceran

    2. Toyota Research Institute, 2311 Green Rd, Ann Arbor, MI, 48105, USA

      Alexander G. Cunningham

    3. Department of Naval Architecture and Marine Engineering, University of Michigan, 2600 Draper Dr, Ann Arbor, MI, 48109, USA

      Ryan M. Eustice

    4. Department of Computer Science and Engineering, University of Michigan, 2260 Hayward St. BBB 3737, Ann Arbor, MI, 48109, USA

      Edwin Olson

    Authors
    1. Enric Galceran
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Alexander G. Cunningham
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Ryan M. Eustice
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Edwin Olson
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Enric Galceran.

    Additional information

    This is one of several papers published in Autonomous Robots comprising the “Special Issue on Robotics Science and Systems”.

    Enric Galceran and Alexander G. Cunningham have contributed equally to this work.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Galceran, E., Cunningham, A.G., Eustice, R.M. et al. Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction: Theory and experiment. Auton Robot 41, 1367–1382 (2017). https://doi.org/10.1007/s10514-017-9619-z

    Download citation

    • Received:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s10514-017-9619-z

    Keywords

    Search

    Navigation