Autonomous Robots

, Volume 43, Issue 3, pp 575–588 | Cite as

Scalable Markov chain approximation for a safe intercept navigation in the presence of multiple vehicles

  • Alexey A. Munishkin
  • Araz Hashemi
  • David W. Casbeer
  • Dejan MilutinovićEmail author


This paper studies a safe intercept navigation which accounts for the uncertainty of other vehicles’ trajectories, avoids collisions and any other positions in which vehicle safety is compromised. Since the number of vehicles can vary with time, it is important that the navigation strategy can quickly adjust to the current number of vehicles, i.e, that it scales well with the number of vehicles. The scalable strategy is based on a stochastic optimal control problem formulation of safe navigation in the presence of a single vehicle, denoted as the one-on-one vehicle problem. It is shown that safe navigation in the presence of multiple vehicles can be solved exactly as an auxiliary Markov decision problem. This allows us to approximate the solution based on the one-on-one vehicle optimal control solution and achieve scalable navigation. Our work is illustrated by a numerical example of safely navigating a vehicle in the presence of four other vehicles and by a robot experiment.


Autonomous navigation Dubins vehicles Stochastic optimal control 



Funding was provided by U.S. Department of Defense (Grant No. FA8650-15-D-2516).

Supplementary material

10514_2018_9739_MOESM1_ESM.mp4 (1.9 mb)
Supplementary material 1 (mp4 1917 KB)

Supplementary material 2 (mp4 6697 KB)


  1. Aigner, M., & Fromme, M. (1984). A game of cops and robbers. Discrete Applied Mathematics, 8(1), 1–12.MathSciNetCrossRefzbMATHGoogle Scholar
  2. Alonso-Mora, J., Breitenmoser, A., Rufli, M., Beardsley, P., & Siegwart, R. (2013). Optimal reciprocal collision avoidance for multiple non-holonomic robots (pp. 203–216). Berlin, Heidelberg: Springer.Google Scholar
  3. Anderson, R., & Milutinović, D. (2011). A stochastic approach to dubins feedback control for target tracking. In 2011 IEEE/RSJ international conference on intelligent robots and systems (pp. 3917–3922).
  4. Anderson, R. P., & Milutinović, D. (2014). A stochastic approach to dubins vehicle tracking problems. IEEE Transactions on Automatic Control, 59(10), 2801–2806. Scholar
  5. Ardema, M. D., Heymann, M., & Rajan, N. (1985). Combat games. Journal of Optimization Theory and Applications, 46(4), 391–398.MathSciNetCrossRefzbMATHGoogle Scholar
  6. Eklund, J., Sprinkle, J., Kim, H., & Sastry, S. (2005). Implementing and testing a nonlinear model predictive tracking controller for aerial pursuit/evasion games on a fixed wing aircraft. In 2005 American control conference (ACC) (Vol. 3, pp. 1509–1514).Google Scholar
  7. Festa, A., & Vinter, R. B. (2016). Decomposition of differential games with multiple targets. Journal of Optimization Theory and Applications, 169, 849–875.MathSciNetCrossRefzbMATHGoogle Scholar
  8. Fleming, W. H., & Rishel, R. W. (1975). Deterministic and stochastic optimal control. New York: Springer.CrossRefzbMATHGoogle Scholar
  9. Gardiner, C. (2009). Stochastic methods: A handbook for the natural and social sciences. Berlin, Heidelberg: Springer.zbMATHGoogle Scholar
  10. Getz, W. M., & Leitmann, G. (1979). Qualitative differential games with two targets. Journal of Mathematical Analysis and Applications, 68, 421–430.MathSciNetCrossRefzbMATHGoogle Scholar
  11. Getz, W. M., & Pachter, M. (1981). Capturability in a two-target “game of two cars”. Journal of Guidance and Control, 4(1), 15–22.CrossRefzbMATHGoogle Scholar
  12. Grimm, W., & Well, K. H. (1991). Modelling air combat as differential game recent approaches and future requirements. In R. P. Hämäläinen, & H. K. Ehtamo (Eds.), Differential games—Developments in modelling and computation. Lecture notes in control and information sciences (Vol. 156). Berlin, Heidelberg: Springer.Google Scholar
  13. Hashemi, A., Casbeer, D. W., & Milutinović, D. (2016). Scalable value approximation for multiple target tail-chase with collision avoidance. In 2016 IEEE 55th conference on decision and control (CDC) (pp. 2543–2548).
  14. Hoy, M., Matveev, A., & Savkin, A. (2015). Algorithms for collision-free navigation of mobile robots in complex cluttered environments: A survey. Robotica, 33(3), 463–497.CrossRefGoogle Scholar
  15. Huang, H., Ding, J., Zhang, W., & Tomlin, C. J. (2015). Automation-assisted capture-the-flag: A differential game approach. IEEE Transactions on Control Systems Technology, 23(3), 1014–1028.CrossRefGoogle Scholar
  16. Isaacs, R. (1965). Differential games. New York, NY: Wiley.zbMATHGoogle Scholar
  17. Israelsen, B. W., Ahmed, N., Center, K., Green, R., & Bennett Jr., W. (2017). Adaptive simulation-based training of ai decision-makers using bayesian optimization. arxiv:1703.09310.
  18. Kushner, H. J., & Dupuis, P. (2001). Numerical methods for stochastic control problems in continuous time, stochastic modelling and applied probability (Vol. 24). New York, NY: Springer.CrossRefzbMATHGoogle Scholar
  19. Li, D., Cruz, J. B., & Schumacher, C. J. (2008). Stochastic multi-player pursuit-evasion differential games. International Journal of Robust and Nonlinear Control, 18(6), 218–247.MathSciNetCrossRefzbMATHGoogle Scholar
  20. McGrew, J. S., How, J. P., Williams, B., & Roy, N. (2010). Air-combat strategy using approximate dynamic programming. Journal of Guidance, Control, and Dynamics, 33(5), 1509–1514.CrossRefGoogle Scholar
  21. Milutinović, D., Casbeer, D. W., Kingston, D., & Rasmussen, S. A. (2017). Stochastic approach to small uav feedback control for target tracking and blind spot avoidance. In Proceedings of the 1st IEEE conference on control technology and applications.Google Scholar
  22. Munishkin, A. A., Milutinović, D., & Casbeer, D. W. (2016). Stochastic optimal control navigation with the avoidance of unsafe configurations. In 2016 international conference on unmanned aircraft systems (ICUAS) (pp. 211–218).
  23. Panagou, D., Stipanović, D. M., & Voulgaris, P. G. (2016). Distributed coordination control for multi-robot networks using Lyapunov-like barrier functions. IEEE Transactions on Automatic Control, 61(3), 617–632.MathSciNetCrossRefzbMATHGoogle Scholar
  24. Powell, W. B. (2009). What you should know about approximate dynamic programming. Naval Research Logistics (NRL), 56(3), 239–249.MathSciNetCrossRefzbMATHGoogle Scholar
  25. Song, Q., & Yin, G. G. (2010). Convergence rates of Markov chain approximation methods for controlled diffusions with stopping. Journal of Systems Science and Complexity, 23(3), 600–621.MathSciNetCrossRefzbMATHGoogle Scholar
  26. Vidal, R., Shakernia, O., Kim, H. J., Shim, D. H., & Sastry, S. (2002). Probabilistic pursuit-evasion games: Theory, implementation, and experimental evaluation. IEEE Transactions on Robotics and Automation, 18(5), 662–669.CrossRefGoogle Scholar
  27. Vieira, M. A. M., Govindan, R., & Sukhatme, G. S. (2009). Scalable and practical pursuit-evasion with networked robots. Intelligent Service Robotics, 2(4), 247.CrossRefGoogle Scholar
  28. Virtanen, K., Karelahti, J., & Raivio, T. (2006). Modeling air combat by a moving horizon influence diagram game. Journal of Guidance, Control, and Dynamics, 29(5), 1509–1514.CrossRefGoogle Scholar
  29. Wang, L., Ames, A. D., & Egerstedt, M. (2017). Safety barrier certificates for collisions-free multirobot systems. IEEE Transactions on Robotics, 33(3), 661–674. Scholar
  30. Yavin, Y. (1988). Stochastic two-target pursuit-evasion differential games in the plane. Journal of Optimization Theory and Applications, 56(3), 325–343.MathSciNetCrossRefzbMATHGoogle Scholar
  31. Yavin, Y., & Villers, R. D. (1988). Stochastic pursuit-evasion differential games in 3D. Journal of Optimization Theory and Applications, 56(3), 345–357.MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Computer EngineeringUniversity of CaliforniaSanta CruzUSA
  2. 2.Control Science Center of Excellence, Air Force Research LaboratoryWright-Patterson AFBDaytonUSA

Personalised recommendations