Advertisement

UAV Guidance Algorithms via Partially Observable Markov Decision Processes

  • Shankarachary Ragi
  • Edwin K. P. Chong
Reference work entry

Abstract

The goal here is to design a path-planning algorithm to guide unmanned aerial vehicles (UAVs) for tracking multiple ground targets based on the theory of partially observable Markov decision processes (POMDPs). This study shows how to exploit the generality and flexibility of the POMDP framework by incorporating a variety of features of interest naturally into the framework, which is accomplished by plugging in the appropriate models. Specifically, this study shows how to incorporate the following features by appropriately formulating the POMDP action space, state-transition law, and objective function: (1) control UAVs with both forward acceleration and bank angle subject to constraints, (2) account for the effect of wind disturbance on UAVs, (3) avoid collisions between UAVs and obstacles and among UAVs, (4) track targets while evading threats, (5) track evasive targets, and (6) mitigate track swaps.

Keywords

Belief State Partially Observable Markov Decision Process Bank Angle Observable Markov Decision Process Threat Index 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

This material is based upon work supported in part by Northrop Grumman Corporation through the Rocky Mountain Aerospace Technology Incubator (RMATI) program and by AFOSR contract FA9550-09-1-0518.

References

  1. W. Adiprawita, A.S. Ahmad, J. Sembiring, B.R. Trilaksono, Reinforcement learning with heuristic to solve POMDP problem in mobile robot path planning, in Proceedings of the International Conference on Electrical Engineering and Informatics, Bandung, July 2011Google Scholar
  2. H. Bai, D. Hsu, M. Kochenderfer, W.S. Lee, Unmanned aircraft collision avoidance using continuous-state POMDPs, in Proceedings of the Robotics: Science and Systems, Los Angeles, June 2011Google Scholar
  3. Y. Bar-Shalom, T.E. Fortmann, Tracking and Data Association (Academic, London, 1988)zbMATHGoogle Scholar
  4. Y. Bar-Shalom, X.R. Li, T. Kirubarajan, Estimation with Applications to Tracking and Navigation (Wiley, New York, 2001)CrossRefGoogle Scholar
  5. R. Bellman, Dynamic Programming (Princeton University Press, Princeton, 1957)zbMATHGoogle Scholar
  6. J. Berger, J. Happe, C. Gagne, M. Lau, Co-evolutionary information gathering for a cooperative unmanned aerial vehicle team, in Proceedings of the 12th International Conference on Information Fusion, Seattle, July 2009, pp. 347–354Google Scholar
  7. D.P. Bertsekas, Dynamic Programming and Optimal Control, vol. 2 (Athena Scientific, Belmont, 2007)Google Scholar
  8. D.P. Bertsekas, D.A. Castanon, Rollout algorithms for stochastic scheduling problems. J. Heuristics 5, 89–108 (1999)CrossRefzbMATHGoogle Scholar
  9. D.P. Bertsekas, J.N. Tsitsiklis, Neuro-Dynamic Programming (Athena Scientific, Belmont, 1996)zbMATHGoogle Scholar
  10. A. Bhattacharyya, On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc. 35, 99–109 (1943)zbMATHMathSciNetGoogle Scholar
  11. S. Blackman, R. Popoli, Design and Analysis of Modern Tracking Systems (Artech House, Boston, 1999)zbMATHGoogle Scholar
  12. J. Borenstein, Y. Koren, Real-time obstacle avoidance for fast mobile robots. IEEE Trans. Syst. Man Cybern. 19, 1179–1187 (1989)CrossRefGoogle Scholar
  13. J. Bruce, M. Veloso, Real-time randomized path planning for robot navigation, in Proceedings of the International Conference on Intelligent Robots and Systems, EPFL Lausanne, Oct 2002, pp. 2383–2388Google Scholar
  14. A. Bry, N. Roy, Rapidly-exploring random belief trees for motion planning under uncertainty, in Proceedings of the IEEE International Conference on Robotics and Automation, Shanghai, May 2011, pp. 723–730Google Scholar
  15. L.L. Cam, G.L. Yang, Asymptotics in Statistics (Springer, Berlin, 2000)CrossRefzbMATHGoogle Scholar
  16. S. Candido, S. Hutchinson, Minimum uncertainty robot path planning using a POMDP approach, in Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Taipei, Oct 2010, pp. 1408–1413Google Scholar
  17. J. Capitán, M.T.J. Spaan, L. Merino, A. Ollero, Decentralized multi-robot cooperation with auctioned POMDPs, in Proceedings of the Sixth Annual Workshop on Multiagent Sequential Decision-Making in Uncertain Domains, Taipei, May 2011Google Scholar
  18. C.G. Cassandras, W. Li, A receding horizon approach for dynamic UAV mission management, in Proceedings of SPIE 17th Annual International Symposium, Orlando, Apr 2003, pp. 284–293Google Scholar
  19. N. Ceccarelli, J.J. Enright, E. Frazzoli, S.J. Rasmussen, C.J. Schumacher, Micro UAV path planning for reconnaissance in wind, in Proceedings of the American Control Conference, New York, July 2007, pp. 5310–5315Google Scholar
  20. E.K.P. Chong, R.L. Givan, H.S. Chang, A framework for simulation-based network control via hindsight optimization, in Proceedings of the 39th IEEE Conference on Decision and Control, Sydney, Dec 2000, pp. 1433–1438Google Scholar
  21. E.K.P. Chong, C. Kreucher, A.O. Hero, Partially observable Markov decision process approximations for adaptive sensing. Discret. Event Dyn. Syst. 19, 377–422 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  22. R.V. Cowlagi, P. Tsiotras, Beyond quadtrees: cell decompositions for path planning using wavelet transforms, in Proceedings of the 46th IEEE Conference on Decision and Control, New Orleans, Dec 2007, pp. 1392–1397Google Scholar
  23. N. Dadkhah, B. Mettler, Survey of motion planning literature in the presence of uncertainty: considerations for UAV guidance. J. Intell. Robot. Syst. 65, 233–246 (2012)CrossRefGoogle Scholar
  24. T. Dong, X.H. Liao, R. Zhang, Z. Sun, Y.D. Song, Path tracking and obstacle avoidance of UAVs – fuzzy logic approach, in Proceedings of the IEEE International Conference on Fuzzy Systems, Reno, May 2005, pp. 43–48Google Scholar
  25. Y. Du, D. Hsu, H. Kurniawati, W.S. Lee, S.C.W. Ong, S.W. Png, A POMDP approach to robot motion planning under uncertainty, in Appeared in ICAPS 2010 POMDP Practitioners Workshop, Toronto, May 2010Google Scholar
  26. B.R. Geiger, J.F. Horn, A.M. DeLullo, L.N. Long, Optimal path planning of UAVs using direct collocation with nonlinear programming, in Proceedings of the AIAA Guidance, Navigation and Control Conference, Keystone, Aug 2006Google Scholar
  27. C. Geyer, Active target search from UAVs in urban environments, in Proceedings of the IEEE International Conference on Robotics and Automation, Pasadena, May 2008, pp. 2366–2371Google Scholar
  28. C. Goerzen, Z. Kong, B. Mettler, A survey of motion planning algorithms from the perspective of autonomous UAV guidance. J. Intell. Robot. Syst. 57, 65–100 (2010)CrossRefzbMATHGoogle Scholar
  29. Z. He, R.V. Iyer, P.R. Chandler, Vision-based UAV flight control and obstacle avoidance, in Proceedings of the American Control Conference, Minneapolis, June 2006, pp. 2166–2170Google Scholar
  30. R. He, A. Bachrach, M. Achtelik, A. Geramifard, D. Gurdan, S. Prentice, J. Stumpf, N. Roy, On the design and use of a micro air vehicle to track and avoid adversaries. Int. J. Robot. Res. 29, 529–546 (2010a).CrossRefGoogle Scholar
  31. R. He, A. Bachrach, N. Roy, Efficient planning under uncertainty for a target-tracking micro-aerial vehicle, in Proceedings of the IEEE International Conference on Robotics and Automation, Anchorage, May 2010bGoogle Scholar
  32. A.L. Jennings, R. Ordonez, N. Ceccarelli, Dynamic programming applied to UAV way point path planning in wind, in Proceedings of the IEEE Multi-conference on Systems and Control, San Antonio, Sept 2008, pp. 215–220Google Scholar
  33. A. Kaneshige, T. Akamatsu, K. Terashima, Real time path planning based on the potential method with 3-D diffusion process for an overhead traveling crane, in Proceedings of the 5th Asian Control Conference, Melbourne, July 2004, pp. 1715–1722Google Scholar
  34. L.E. Kavraki, P. Svestka, J. Latombe, M.H. Overmars, Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans. Robot. Autom. 12, 566–580 (1996)CrossRefGoogle Scholar
  35. L.E. Kavralu, P. Svestka, J. Latombe, M.H. Overmars, Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans. Robot. Autom. 12, 566–580 (1996)CrossRefGoogle Scholar
  36. O. Khatib, Real-time obstacle avoidance for manipulators and mobile robots, in Proceedings of the IEEE International Conference on Robotics and Automation, St. Louis, Mar 1985, pp. 500–505Google Scholar
  37. Y. Kim, D. Gub, I. Postlethwaite, Real-time path planning with limited information for autonomous unmanned air vehicles. J. Autom. 44, 696–712 (2008)CrossRefzbMATHGoogle Scholar
  38. Y. Koren, J. Borenstein, Potential field methods and their inherent limitations for mobile robot navigation, in Proceedings of the IEEE International Conference on Robotics and Automation, Sacramento, Apr 1991, pp. 1398–1404Google Scholar
  39. C. Kreucher, A.O. Hero, K. Kastella, D. Chang, Efficient methods of non-myopic sensor management for multitarget tracking, in Proceedings of the 43rd IEEE Conference on Decision and Control, Paradise Island, Dec 2004, pp. 722–727Google Scholar
  40. B.H. Krogh, C.E. Thorpe, Integrated path planning and dynamic steering control for autonomous vehicles, in Proceedings of the IEEE International Conference on Robotics and Automation, San Francisco, Apr 1986, pp. 1664–1669Google Scholar
  41. S. Kullback, R.A. Leibler, On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951)CrossRefzbMATHMathSciNetGoogle Scholar
  42. H. Kurniawati, Y. Du, D. Hsu, W.S. Lee, Motion planning under uncertainty for robotic tasks with long time horizons. Int. J. Robot. Res. 30, 308–323 (2011)CrossRefGoogle Scholar
  43. S.M. LaValle, Robot motion planning: a game-theoretic foundation. Algorithmica 26, 430–465 (2000)CrossRefzbMATHMathSciNetGoogle Scholar
  44. S.M. LaValle, Planning Algorithms (Cambridge University Press, Cambridge, 2006)CrossRefzbMATHGoogle Scholar
  45. N.R.J. Lawrance, S. Sukkarieh, Path planning for autonomous soaring flight in dynamic wind fields, in Proceedings of the IEEE International Conference on Robotics and Automation, Shanghai, May 2011, pp. 2499–2505Google Scholar
  46. P. Leven, S. Hutchinson, A framework for real-time path planning in changing environments. Int. J. Robot. Res. 21, 999–1030 (2002)CrossRefGoogle Scholar
  47. W. Li, C.G. Cassandras, A cooperative receding horizon controller for multivehicle uncertain environments. IEEE Trans. Autom. Control 51, 242–257 (2006)CrossRefMathSciNetGoogle Scholar
  48. Y. Lu, X. Huo, O. Arslan, P. Tsiotras, Incremental multi-scale search algorithm for dynamic path planning with low worst-case complexity. IEEE Trans. Syst. Man Cybern. B Cybern. 41, 1556– 1570 (2011)CrossRefGoogle Scholar
  49. X. Ma, D.A. Castanon, Receding horizon planning for dubins traveling salesman problems, in Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, Dec 2006, pp. 5453–5458CrossRefGoogle Scholar
  50. P.C. Mahalanobis, On the generalised distance in statistics, in Proceedings of the National Institute of Sciences of India, Calcutta, Apr 1936, pp. 49–55Google Scholar
  51. T.G. McGee, J.K. Hedrick, Path planning and control for multiple point surveillance by an unmanned aircraft in wind, in Proceedings of the American Control Conference, Minneapolis, June 2006, pp. 4261–4266Google Scholar
  52. T.G. McGee, S. Spry, J.K. Hedrick, Optimal path planning in a constant wind with a bounded turning rate, in Proceedings of the AIAA Guidance, Navigation and Control Conference and Exhibit, San Francisco, Aug 2005Google Scholar
  53. M.L. McIntyre, W.E. Dixon, D.M. Dawson, B. Xian, Adaptive tracking control of on-line path planners: velocity fields and navigation functions, in Proceedings of the American Control Conference, Portland, June 2005, pp. 3168–3173Google Scholar
  54. S.A. Miller, Z.A. Harris, E.K.P. Chong, A POMDP framework for coordinated guidance of autonomous UAVs for multitarget tracking. EURASIP J. Adv. Signal Process. 2009, 724597 (2009)CrossRefGoogle Scholar
  55. M. Naveed, A. Crampton, D. Kitchin, T.L. McCluskey, Real-time path planning using a simulation-based markov decision process, in Proceedings of the Thirty-First SGAI International Conference on Artificial Intelligence, Cambridge, Dec 2011Google Scholar
  56. I.K. Nikolos, K.P. Valavanis, N.C. Tsourveloudis, A.N. Kostaras, Evolutionary algorithm based offline/online path planner for UAV navigation. IEEE Trans. Syst. Man Cybern. B Cybern. 33, 898–912 (2003)CrossRefGoogle Scholar
  57. J. Ousingsawat, M.E. Campbell, On-line estimation and path planning for multiple vehicles in an uncertain environment. Int. J. Robust Nonlinear Control 14, 741–766 (2004)CrossRefzbMATHMathSciNetGoogle Scholar
  58. J. Park, H. Oh, M. Tahk, Uav collision avoidance based on geometric approach, in Proceedings of the SICE Annual Conference, Tokyo, Aug 2008, pp. 2122–2126Google Scholar
  59. S. Park, H. Choi, N. Roy, J.P. How, Learning covariance dynamics for path planning of UAV sensors in a large-scale dynamic environment, in Proceedings of the AIAA Guidance, Navigation, and Control Conference, Chicago, Aug 2009Google Scholar
  60. J. Redding, J.N. Amin, J.D. Boskovic, A real-time obstacle detection and reactive path planning system for autonomous small-scale helicopters, in Proceedings of the AIAA Guidance, Navigation and Control Conference and Exhibit, Hilton Head, Aug 2007, pp. 4261–4266Google Scholar
  61. W. Ren, J. Sun, R.W. Beard, T.W. McLain, Nonlinear tracking control for nonholonomic mobile robots with input constraints: an experimental study, in Proceedings of the American Control Conference, Portland, June 2005, pp. 4923–4928Google Scholar
  62. P.W. Sarunic, R.J. Evans, B. Moran, Control of unmanned aerial vehicles for passive detection and tracking of multiple emitters, in Proceedings of the IEEE Symposium on Computational Intelligence in Security and Defense Applications, Ottawa, July 2009Google Scholar
  63. J.B. Saunders, O. Call, A. Curtis, A.W. Beard, T.W. Mclain, Static and dynamic obstacle avoidance in miniature air vehicles, in Proceedings of the Infotech at Aerospace Conference, Arlington, Sep 2005, pp. 2005–6950Google Scholar
  64. A. Sud, E. Andersen, S. Curtis, M.C. Lin, D. Manocha, Real-time path planning in dynamic virtual environments using multiagent navigation graphs. IEEE Trans. Vis. Comput. Graph. 14, 526–538 (2008)CrossRefGoogle Scholar
  65. R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction (MIT, Cambridge, 1998)Google Scholar
  66. L. Techy, C.A. Woolsey, K.A. Morgansen, Planar path planning for flight vehicles in wind with turn rate and acceleration bounds, in Proceedings of the IEEE International Conference on Robotics and Automation, Anchorage, May 2010, pp. 3240–3245Google Scholar
  67. R.B. Tilove, Local obstacle avoidance for mobile robots based on the method of artificial potentials, in Proceedings of the IEEE International Conference on Robotics and Automation, Cincinnati, May 1990, pp. 566–571CrossRefGoogle Scholar
  68. J. Tisdale, H. Durrant-Whyte, J.K. Hedrick. Path planning for cooperative sensing using unmanned vehicles, in Proceedings of the ASME International Mechanical Engineering Conference and Exposition, Seattle, Nov 2007, pp. 715–723Google Scholar
  69. G. Wu, E.K.P. Chong, R. Givan, Burst-level congestion control using hindsight optimization. IEEE Trans. Autom. Control 47, 979–991 (2002)CrossRefMathSciNetGoogle Scholar
  70. C. Yong, E.J. Barth, Real-time dynamic path planning for dubins’ nonholonomic robot, in Proceedings of the 45th IEEE Conference on Decision and Control, San Diego, Dec 2006, pp. 2418–2423CrossRefGoogle Scholar
  71. D. Zu, J.D. Han, M. Campbell, Artificial potential guided evolutionary path plan for multi-vehicle multi-target pursuit, in Proceedings of the IEEE International Conference on Robotics and Biomimetics, Shenyang, Aug 2004, pp. 855–861Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringColorado State UniversityFort CollinsUSA

Personalised recommendations