Collision Avoidance Using Partially Controlled Markov Decision Processes

  • Mykel J. Kochenderfer
  • James P. Chryssanthacopoulos
Part of the Communications in Computer and Information Science book series (CCIS, volume 271)


Optimal collision avoidance in stochastic environments requires accounting for the likelihood and costs of future sequences of outcomes in response to different sequences of actions. Prior work has investigated formulating the problem as a Markov decision process, discretizing the state space, and solving for the optimal strategy using dynamic programming. Experiments have shown that such an approach can be very effective, but scaling to higher-dimensional problems can be challenging due to the exponential growth of the discrete state space. This paper presents an approach that can greatly reduce the complexity of computing the optimal strategy in problems where only some of the dimensions of the problem are controllable. The approach is applied to aircraft collision avoidance where the system must recommend maneuvers to an imperfect pilot.


Markov decision processes Dynamic programming Collision avoidance 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bellman, R.E.: Adaptive control processes: A guided tour. Princeton University Press (1961)Google Scholar
  2. 2.
    Bernstein, D.S., Zilberstein, S., Immerman, N.: The complexity of decentralized control of Markov decision processes. In: Conference on Uncertainty in Artificial Intelligence, pp. 32–37. Morgan Kaufmann (2000)Google Scholar
  3. 3.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn., vol. 1. Athena Scientific, Belmont (2005)zbMATHGoogle Scholar
  4. 4.
    Bilimoria, K.D.: A geometric optimization approach to aircraft conflict resolution. In: AIAA Guidance, Navigation, and Control Conference and Exhibit, Denver, Colo. (2000)Google Scholar
  5. 5.
    Carpenter, B.D., Kuchar, J.K.: Probability-based collision alerting logic for closely-spaced parallel approach. In: AIAA 35th Aerospace Sciences Meeting, Reno, NV (January 1997)Google Scholar
  6. 6.
    Chamlou, R.: Future airborne collision avoidance—design principles, analysis plan and algorithm development. In: Digital Avionics Systems Conference (2009)Google Scholar
  7. 7.
    Chryssanthacopoulos, J.P., Kochenderfer, M.J.: Accounting for state uncertainty in collision avoidance. Journal of Guidance, Control, and Dynamics 34(4), 951–960 (2011)CrossRefGoogle Scholar
  8. 8.
    Chryssanthacopoulos, J.P., Kochenderfer, M.J., Williams, R.E.: Improved Monte Carlo sampling for conflict probability estimation. In: AIAA Non-Deterministic Approaches Conference, Orlando, Florida (2010)Google Scholar
  9. 9.
    Davies, S.: Multidimensional triangulation and interpolation for reinforcement learning. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 1005–1011. MIT Press, Cambridge (1997)Google Scholar
  10. 10.
    Dowek, G., Geser, A., Muñoz, C.: Tactical conflict detection and resolution in a 3-D airspace. In: 4th USA/Europe Air Traffic Management R&D Seminar, Santa Fe, New Mexico (2001)Google Scholar
  11. 11.
    Duong, V.N., Zeghal, K.: Conflict resolution advisory for autonomous airborne separation in low-density airspace. In: IEEE Conference on Decision and Control, December 10-12, vol. 3, pp. 2429–2434 (1997)Google Scholar
  12. 12.
    Eby, M.S., Kelly, W.E.: Free flight separation assurance using distributed algorithms. In: IEEE Aerospace Conference, March 6-13, vol. 2, pp. 429–441 (1999)Google Scholar
  13. 13.
    Khatib, O., Maitre, J.F.L.: Dynamic control of manipulators operating in a complex environment. In: Symposium on Theory and Practice of Robots and Manipulators, pp. 267–282. Elsevier, Udine (1978)Google Scholar
  14. 14.
    Kochenderfer, M.J., Chryssanthacopoulos, J.P.: A decision-theoretic approach to developing robust collision avoidance logic. In: IEEE International Conference on Intelligent Transportation Systems, Madeira Island, Portugal (2010)Google Scholar
  15. 15.
    Kochenderfer, M.J., Chryssanthacopoulos, J.P., Kaelbling, L.P., Lozano-Perez, T.: Model-based optimization of airborne collision avoidance logic. Project Report ATC-360, Massachusetts Institute of Technology, Lincoln Laboratory (2010)Google Scholar
  16. 16.
    Kochenderfer, M.J., Edwards, M.W.M., Espindle, L.P., Kuchar, J.K., Griffith, J.D.: Airspace encounter models for estimating collision risk. Journal of Guidance, Control, and Dynamics 33(2), 487–499 (2010)CrossRefGoogle Scholar
  17. 17.
    Kurniawati, H., Hsu, D., Lee, W.: SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In: Robotics: Science and Systems (2008)Google Scholar
  18. 18.
    Kuwata, Y., Fiore, G.A., Teo, J., Frazzoli, E., How, J.P.: Motion planning for urban driving using RRT. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sepember 22-26, pp. 1681–1686 (2008)Google Scholar
  19. 19.
    LaValle, S.M.: Rapidly-exploring random trees: A new tool for path planning. Tech. Rep. 98-11, Computer Science Department, Iowa State University (October 1998)Google Scholar
  20. 20.
    Powell, W.B.: Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, Hoboken (2007)zbMATHCrossRefGoogle Scholar
  21. 21.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley series in probability and mathematical statistics. Wiley, New York (1994)Google Scholar
  22. 22.
    RTCA: Minimum operational performance standards for Traffic Alert and Collision Avoidance System II (TCAS II), DO-185b. RTCA, Inc., Washington, D.C. (June 2008)Google Scholar
  23. 23.
    Saunders, J., Beard, R., Byrne, J.: Vision-based reactive multiple obstacle avoidance for micro air vehicles. In: American Control Conference, June 10-12, pp. 5253–5258 (2009)Google Scholar
  24. 24.
    Smith, T., Simmons, R.G.: Point-based POMDP algorithms: Improved analysis and implementation. In: Uncertainty in Artificial Intelligence (2005)Google Scholar
  25. 25.
    Temizer, S., Kochenderfer, M.J., Kaelbling, L.P., Lozano-Pérez, T., Kuchar, J.K.: Collision avoidance for unmanned aircraft using Markov decision processes. In: AIAA Guidance, Navigation, and Control Conference, Toronto, Canada (2010)Google Scholar
  26. 26.
    Yang, L.C., Kuchar, J.K.: Prototype conflict alerting system for free flight. Journal of Guidance, Control, and Dynamics 20(4), 768–773 (1997)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Mykel J. Kochenderfer
    • 1
  • James P. Chryssanthacopoulos
    • 1
  1. 1.Lincoln LaboratoryMassachusetts Institute of TechnologyLexingtonU.S.A

Personalised recommendations