Abstract
Optimal collision avoidance in stochastic environments requires accounting for the likelihood and costs of future sequences of outcomes in response to different sequences of actions. Prior work has investigated formulating the problem as a Markov decision process, discretizing the state space, and solving for the optimal strategy using dynamic programming. Experiments have shown that such an approach can be very effective, but scaling to higher-dimensional problems can be challenging due to the exponential growth of the discrete state space. This paper presents an approach that can greatly reduce the complexity of computing the optimal strategy in problems where only some of the dimensions of the problem are controllable. The approach is applied to aircraft collision avoidance where the system must recommend maneuvers to an imperfect pilot.
This work is sponsored by the Federal Aviation Administration under Air Force Contract #FA8721-05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bellman, R.E.: Adaptive control processes: A guided tour. Princeton University Press (1961)
Bernstein, D.S., Zilberstein, S., Immerman, N.: The complexity of decentralized control of Markov decision processes. In: Conference on Uncertainty in Artificial Intelligence, pp. 32–37. Morgan Kaufmann (2000)
Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn., vol. 1. Athena Scientific, Belmont (2005)
Bilimoria, K.D.: A geometric optimization approach to aircraft conflict resolution. In: AIAA Guidance, Navigation, and Control Conference and Exhibit, Denver, Colo. (2000)
Carpenter, B.D., Kuchar, J.K.: Probability-based collision alerting logic for closely-spaced parallel approach. In: AIAA 35th Aerospace Sciences Meeting, Reno, NV (January 1997)
Chamlou, R.: Future airborne collision avoidance—design principles, analysis plan and algorithm development. In: Digital Avionics Systems Conference (2009)
Chryssanthacopoulos, J.P., Kochenderfer, M.J.: Accounting for state uncertainty in collision avoidance. Journal of Guidance, Control, and Dynamics 34(4), 951–960 (2011)
Chryssanthacopoulos, J.P., Kochenderfer, M.J., Williams, R.E.: Improved Monte Carlo sampling for conflict probability estimation. In: AIAA Non-Deterministic Approaches Conference, Orlando, Florida (2010)
Davies, S.: Multidimensional triangulation and interpolation for reinforcement learning. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 1005–1011. MIT Press, Cambridge (1997)
Dowek, G., Geser, A., Muñoz, C.: Tactical conflict detection and resolution in a 3-D airspace. In: 4th USA/Europe Air Traffic Management R&D Seminar, Santa Fe, New Mexico (2001)
Duong, V.N., Zeghal, K.: Conflict resolution advisory for autonomous airborne separation in low-density airspace. In: IEEE Conference on Decision and Control, December 10-12, vol. 3, pp. 2429–2434 (1997)
Eby, M.S., Kelly, W.E.: Free flight separation assurance using distributed algorithms. In: IEEE Aerospace Conference, March 6-13, vol. 2, pp. 429–441 (1999)
Khatib, O., Maitre, J.F.L.: Dynamic control of manipulators operating in a complex environment. In: Symposium on Theory and Practice of Robots and Manipulators, pp. 267–282. Elsevier, Udine (1978)
Kochenderfer, M.J., Chryssanthacopoulos, J.P.: A decision-theoretic approach to developing robust collision avoidance logic. In: IEEE International Conference on Intelligent Transportation Systems, Madeira Island, Portugal (2010)
Kochenderfer, M.J., Chryssanthacopoulos, J.P., Kaelbling, L.P., Lozano-Perez, T.: Model-based optimization of airborne collision avoidance logic. Project Report ATC-360, Massachusetts Institute of Technology, Lincoln Laboratory (2010)
Kochenderfer, M.J., Edwards, M.W.M., Espindle, L.P., Kuchar, J.K., Griffith, J.D.: Airspace encounter models for estimating collision risk. Journal of Guidance, Control, and Dynamics 33(2), 487–499 (2010)
Kurniawati, H., Hsu, D., Lee, W.: SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In: Robotics: Science and Systems (2008)
Kuwata, Y., Fiore, G.A., Teo, J., Frazzoli, E., How, J.P.: Motion planning for urban driving using RRT. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, Sepember 22-26, pp. 1681–1686 (2008)
LaValle, S.M.: Rapidly-exploring random trees: A new tool for path planning. Tech. Rep. 98-11, Computer Science Department, Iowa State University (October 1998)
Powell, W.B.: Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, Hoboken (2007)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley series in probability and mathematical statistics. Wiley, New York (1994)
RTCA: Minimum operational performance standards for Traffic Alert and Collision Avoidance System II (TCAS II), DO-185b. RTCA, Inc., Washington, D.C. (June 2008)
Saunders, J., Beard, R., Byrne, J.: Vision-based reactive multiple obstacle avoidance for micro air vehicles. In: American Control Conference, June 10-12, pp. 5253–5258 (2009)
Smith, T., Simmons, R.G.: Point-based POMDP algorithms: Improved analysis and implementation. In: Uncertainty in Artificial Intelligence (2005)
Temizer, S., Kochenderfer, M.J., Kaelbling, L.P., Lozano-Pérez, T., Kuchar, J.K.: Collision avoidance for unmanned aircraft using Markov decision processes. In: AIAA Guidance, Navigation, and Control Conference, Toronto, Canada (2010)
Yang, L.C., Kuchar, J.K.: Prototype conflict alerting system for free flight. Journal of Guidance, Control, and Dynamics 20(4), 768–773 (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kochenderfer, M.J., Chryssanthacopoulos, J.P. (2013). Collision Avoidance Using Partially Controlled Markov Decision Processes. In: Filipe, J., Fred, A. (eds) Agents and Artificial Intelligence. ICAART 2011. Communications in Computer and Information Science, vol 271. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29966-7_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-29966-7_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29965-0
Online ISBN: 978-3-642-29966-7
eBook Packages: Computer ScienceComputer Science (R0)