Scalable Markov chain approximation for a safe intercept navigation in the presence of multiple vehicles

Abstract

This paper studies a safe intercept navigation which accounts for the uncertainty of other vehicles’ trajectories, avoids collisions and any other positions in which vehicle safety is compromised. Since the number of vehicles can vary with time, it is important that the navigation strategy can quickly adjust to the current number of vehicles, i.e, that it scales well with the number of vehicles. The scalable strategy is based on a stochastic optimal control problem formulation of safe navigation in the presence of a single vehicle, denoted as the one-on-one vehicle problem. It is shown that safe navigation in the presence of multiple vehicles can be solved exactly as an auxiliary Markov decision problem. This allows us to approximate the solution based on the one-on-one vehicle optimal control solution and achieve scalable navigation. Our work is illustrated by a numerical example of safely navigating a vehicle in the presence of four other vehicles and by a robot experiment.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Aigner, M., & Fromme, M. (1984). A game of cops and robbers. Discrete Applied Mathematics, 8(1), 1–12.

    MathSciNet  Article  MATH  Google Scholar 

  2. Alonso-Mora, J., Breitenmoser, A., Rufli, M., Beardsley, P., & Siegwart, R. (2013). Optimal reciprocal collision avoidance for multiple non-holonomic robots (pp. 203–216). Berlin, Heidelberg: Springer.

    Google Scholar 

  3. Anderson, R., & Milutinović, D. (2011). A stochastic approach to dubins feedback control for target tracking. In 2011 IEEE/RSJ international conference on intelligent robots and systems (pp. 3917–3922). https://doi.org/10.1109/IROS.2011.6094760.

  4. Anderson, R. P., & Milutinović, D. (2014). A stochastic approach to dubins vehicle tracking problems. IEEE Transactions on Automatic Control, 59(10), 2801–2806. https://doi.org/10.1109/TAC.2014.2314224.

    MathSciNet  Article  MATH  Google Scholar 

  5. Ardema, M. D., Heymann, M., & Rajan, N. (1985). Combat games. Journal of Optimization Theory and Applications, 46(4), 391–398.

    MathSciNet  Article  MATH  Google Scholar 

  6. Eklund, J., Sprinkle, J., Kim, H., & Sastry, S. (2005). Implementing and testing a nonlinear model predictive tracking controller for aerial pursuit/evasion games on a fixed wing aircraft. In 2005 American control conference (ACC) (Vol. 3, pp. 1509–1514).

  7. Festa, A., & Vinter, R. B. (2016). Decomposition of differential games with multiple targets. Journal of Optimization Theory and Applications, 169, 849–875.

    MathSciNet  Article  MATH  Google Scholar 

  8. Fleming, W. H., & Rishel, R. W. (1975). Deterministic and stochastic optimal control. New York: Springer.

    Google Scholar 

  9. Gardiner, C. (2009). Stochastic methods: A handbook for the natural and social sciences. Berlin, Heidelberg: Springer.

    Google Scholar 

  10. Getz, W. M., & Leitmann, G. (1979). Qualitative differential games with two targets. Journal of Mathematical Analysis and Applications, 68, 421–430.

    MathSciNet  Article  MATH  Google Scholar 

  11. Getz, W. M., & Pachter, M. (1981). Capturability in a two-target “game of two cars”. Journal of Guidance and Control, 4(1), 15–22.

    Article  MATH  Google Scholar 

  12. Grimm, W., & Well, K. H. (1991). Modelling air combat as differential game recent approaches and future requirements. In R. P. Hämäläinen, & H. K. Ehtamo (Eds.), Differential games—Developments in modelling and computation. Lecture notes in control and information sciences (Vol. 156). Berlin, Heidelberg: Springer.

  13. Hashemi, A., Casbeer, D. W., & Milutinović, D. (2016). Scalable value approximation for multiple target tail-chase with collision avoidance. In 2016 IEEE 55th conference on decision and control (CDC) (pp. 2543–2548). https://doi.org/10.1109/CDC.2016.7798645.

  14. Hoy, M., Matveev, A., & Savkin, A. (2015). Algorithms for collision-free navigation of mobile robots in complex cluttered environments: A survey. Robotica, 33(3), 463–497.

    Article  Google Scholar 

  15. Huang, H., Ding, J., Zhang, W., & Tomlin, C. J. (2015). Automation-assisted capture-the-flag: A differential game approach. IEEE Transactions on Control Systems Technology, 23(3), 1014–1028.

    Article  Google Scholar 

  16. Isaacs, R. (1965). Differential games. New York, NY: Wiley.

    Google Scholar 

  17. Israelsen, B. W., Ahmed, N., Center, K., Green, R., & Bennett Jr., W. (2017). Adaptive simulation-based training of ai decision-makers using bayesian optimization. arxiv:1703.09310.

  18. Kushner, H. J., & Dupuis, P. (2001). Numerical methods for stochastic control problems in continuous time, stochastic modelling and applied probability (Vol. 24). New York, NY: Springer.

    Google Scholar 

  19. Li, D., Cruz, J. B., & Schumacher, C. J. (2008). Stochastic multi-player pursuit-evasion differential games. International Journal of Robust and Nonlinear Control, 18(6), 218–247.

    MathSciNet  Article  MATH  Google Scholar 

  20. McGrew, J. S., How, J. P., Williams, B., & Roy, N. (2010). Air-combat strategy using approximate dynamic programming. Journal of Guidance, Control, and Dynamics, 33(5), 1509–1514.

    Article  Google Scholar 

  21. Milutinović, D., Casbeer, D. W., Kingston, D., & Rasmussen, S. A. (2017). Stochastic approach to small uav feedback control for target tracking and blind spot avoidance. In Proceedings of the 1st IEEE conference on control technology and applications.

  22. Munishkin, A. A., Milutinović, D., & Casbeer, D. W. (2016). Stochastic optimal control navigation with the avoidance of unsafe configurations. In 2016 international conference on unmanned aircraft systems (ICUAS) (pp. 211–218). https://doi.org/10.1109/ICUAS.2016.7502568.

  23. Panagou, D., Stipanović, D. M., & Voulgaris, P. G. (2016). Distributed coordination control for multi-robot networks using Lyapunov-like barrier functions. IEEE Transactions on Automatic Control, 61(3), 617–632.

    MathSciNet  Article  MATH  Google Scholar 

  24. Powell, W. B. (2009). What you should know about approximate dynamic programming. Naval Research Logistics (NRL), 56(3), 239–249.

    MathSciNet  Article  MATH  Google Scholar 

  25. Song, Q., & Yin, G. G. (2010). Convergence rates of Markov chain approximation methods for controlled diffusions with stopping. Journal of Systems Science and Complexity, 23(3), 600–621.

    MathSciNet  Article  MATH  Google Scholar 

  26. Vidal, R., Shakernia, O., Kim, H. J., Shim, D. H., & Sastry, S. (2002). Probabilistic pursuit-evasion games: Theory, implementation, and experimental evaluation. IEEE Transactions on Robotics and Automation, 18(5), 662–669.

    Article  Google Scholar 

  27. Vieira, M. A. M., Govindan, R., & Sukhatme, G. S. (2009). Scalable and practical pursuit-evasion with networked robots. Intelligent Service Robotics, 2(4), 247.

    Article  Google Scholar 

  28. Virtanen, K., Karelahti, J., & Raivio, T. (2006). Modeling air combat by a moving horizon influence diagram game. Journal of Guidance, Control, and Dynamics, 29(5), 1509–1514.

    Article  Google Scholar 

  29. Wang, L., Ames, A. D., & Egerstedt, M. (2017). Safety barrier certificates for collisions-free multirobot systems. IEEE Transactions on Robotics, 33(3), 661–674. https://doi.org/10.1109/TRO.2017.2659727.

    Article  Google Scholar 

  30. Yavin, Y. (1988). Stochastic two-target pursuit-evasion differential games in the plane. Journal of Optimization Theory and Applications, 56(3), 325–343.

    MathSciNet  Article  MATH  Google Scholar 

  31. Yavin, Y., & Villers, R. D. (1988). Stochastic pursuit-evasion differential games in 3D. Journal of Optimization Theory and Applications, 56(3), 345–357.

    MathSciNet  Article  MATH  Google Scholar 

Download references

Acknowledgements

Funding was provided by U.S. Department of Defense (Grant No. FA8650-15-D-2516).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Dejan Milutinović.

Electronic supplementary material

Below is the link to the electronic supplementary material.

10514_2018_9739_MOESM1_ESM.mp4

Supplementary material 1 (mp4 1917 KB)

Supplementary material 2 (mp4 6697 KB)

Supplementary material 1 (mp4 1917 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Munishkin, A.A., Hashemi, A., Casbeer, D.W. et al. Scalable Markov chain approximation for a safe intercept navigation in the presence of multiple vehicles. Auton Robot 43, 575–588 (2019). https://doi.org/10.1007/s10514-018-9739-0

Download citation

Keywords

  • Autonomous navigation
  • Dubins vehicles
  • Stochastic optimal control