Journal of Intelligent & Robotic Systems

, Volume 82, Issue 1, pp 135–162 | Cite as

Stochastic Optimal Coordination of Small UAVs for Target Tracking using Regression-based Dynamic Programming

  • Steven A. P. QuinteroEmail author
  • Michael Ludkovski
  • João P. Hespanha


We study the problem of optimally coordinating multiple fixed-wing UAVs to perform vision-based target tracking, which entails that the UAVs are tasked with gathering the best joint vision-based measurements of an unpredictable ground target. We utilize an analytic expression for the error covariance associated with the fused measurements of the target’s position, and we employ stochastic fourth-order models for all vehicles, thereby incorporating a high degree of realism into the problem formulation. While dynamic programming can generate an optimal control policy that minimizes the expected value of the fused geolocation error covariance over time, it is accompanied by significant computational challenges due to the curse of dimensionality. In order to circumvent this challenge, we present a novel policy generation technique that combines simulation-based policy iteration with a robust regression scheme. The resulting control policy offers a significant advantage over alternative approaches and shows that the optimal control strategy involves coordinating the UAVs’ distances to the target rather than their viewing angles, which had been a common practice in target tracking.


Target tracking Unmanned aerial vehicle Autonomous vehicle Regression Monte Carlo Motion planning Probabilistic planning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mallick, M.: Geolocation using video sensor measurements. In: IEEE Int. Conf. Information Fusion. Quebec, Canada (2007)Google Scholar
  2. 2.
    Collins, G.E., Stankevitz, C.R., Liese, J.: Implementation of a sensor guided flight algorithm for target tracking by small UAS. In: Ground/Air Multi-Sens. Interoperability, Integration, Netw. Persistent ISR II, vol. 8047, SPIE (2011)Google Scholar
  3. 3.
    Kingston, D.B.: Decentralized Control of Multiple UAVs for perimeter and target surveillance. PhD thesis, Brigham Young University (2007)Google Scholar
  4. 4.
    Gu, G., Chandler, P.R., Schumacher, C.J., Sparks, A., Pachter, M.: Optimum cooperative UAV sensing using a team of UAVs. IEEE Trans. Aerosp. Electron. Syst. 42, 1446–1458 (2006)CrossRefGoogle Scholar
  5. 5.
    Rysdyk, R.: UAV path following for constant line-of-sight. In: Proc. 2nd AIAA Unmanned Unltd. Syst. Technol. Operations Aerosp, Land Sea Conf (2003)Google Scholar
  6. 6.
    Frew, E.W.: Lyapunov guidance vector fields for unmanned aircraft applications. In: Am. Control Conf. (2007)Google Scholar
  7. 7.
    Kim, S., Oh, H., Tsourdos, A.: Nonlinear model predictive coordinated standoff tracking of a moving ground vehicle. J. Guid. Control Dyn. 36(2), 557–566 (2013)CrossRefGoogle Scholar
  8. 8.
    Ma, L., Hovakimyan, N.: Cooperative target tracking in balanced circular formation: Multiple UAVs tracking a ground vehicle. In: Am. Control Conf., pp 5386–5391. IEEE (2013)Google Scholar
  9. 9.
    Summers, T.H.: Cooperative Shape and Orientation Control of Autonomous Vehicle Formations. PhD thesis, University of Texas at Austin (2010)Google Scholar
  10. 10.
    Oh, H., Kim, S., Tsourdos, A., White, B.A.: Decentralised standoff tracking of moving targets using adaptive sliding mode control for UAVs. J. Intell. Robot. Syst., 1– 15 (2013)Google Scholar
  11. 11.
    Peterson, C., Paley, D.A.: Multivehicle coordination in an estimated time-varying flowfield. J. Guid. Control Dyn. 34(1), 177–191 (2011)CrossRefGoogle Scholar
  12. 12.
    Anderson, R., Milutinović, D.: A stochastic approach to Dubins feedback control for target tracking. In: IEEE / RSJ Conf. Intell. Robots Syst., pp 3917–3922 (2011)Google Scholar
  13. 13.
    Quintero, S.A.P., Hespanha, J.P.: Vision-based target tracking with a small UAV: Optimization-based control strategies. Control Eng. Pract. 32, 28–42 (2014)CrossRefGoogle Scholar
  14. 14.
    Miller, S.A., Harris, Z.A., Chong, E.K.P.: A POMDP framework for coordinated guidance of autonomous UAVs for multitarget tracking, EURASIP. J. Adv. Signal Process., 1–17 (2009)Google Scholar
  15. 15.
    Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics. The MIT Press (2005)Google Scholar
  16. 16.
    Stachura, M., Carfang, A., Frew, E.W.. In: Workshop Robotic Wirel. Sens. Netw., (Marina Del Ray, CA) Cooperative target tracking with a communication limited active sensor network (2009)Google Scholar
  17. 17.
    Ding, C., Morye, A.A., Farrell, J.A., Roy-Chowdhury, A. K.: Coordinated sensing and tracking for mobile camera platforms. In: Am. Control Conf., pp. 5114–5119, IEEE (2012)Google Scholar
  18. 18.
    Quintero, S.A.P., Papi, F., Klein, D.J., Chisci, L., Hespanha, J.P.: Optimal UAV coordination for target tracking using dynamic programming. In: IEEE Conf. Decis. Control, (Atlanta, GA) (2010)Google Scholar
  19. 19.
    Lalish, E., Morgansen, K., Tsukamaki, T.: Oscillatory control for constant-speed unicycle-type vehicles. In: IEEE Conf. Decis. Control (2007)Google Scholar
  20. 20.
    Regina, N., Zanzi, M.: UAV guidance law for ground-based target trajectory tracking and loitering. In: Aerosp. Conf., IEEE (2011)Google Scholar
  21. 21.
    Quintero, S.A.P., Collins, G.E., Hespanha, J.P.: Flocking with fixed-wing UAVs for distributed sensing: A stochastic optimal control approach. In: Am. Control Conf., (Washington, D.C.) (2013)Google Scholar
  22. 22.
    Bouchard, B., Warin, X.: Monte-carlo valorisation of American options: facts and new algorithms to improve existing methods. In: R. Carmona, P. Del Moral, P. Hu, N. Oudjane (eds.) Numer. Methods Finance,Springer Proc. Math. (2011)Google Scholar
  23. 23.
    Lewis, F. L., Vrabie, D., Syrmos, V.L.: 3rd. John Wiley, Hoboken, New Jersey (2012)Google Scholar
  24. 24.
    Guestrin, C., Hauskrecht, M., Kveton, B.: Solving factored mdps with continuous and discrete variables. In: Proc. 20th Conf. on Uncertain. in Artif. Intell., 235–242, AUAI Press (2004)Google Scholar
  25. 25.
    Wiegerinck, W., Broek, B.v.d., Kappen, H.: Stochastic optimal control in continuous space-time multi-agent systems. In: Proc. 22nd Conf. on Uncertain. in Artif. Intell. (2006)Google Scholar
  26. 26.
    Longstaff, F.A., Schwartz, E.S.: Valuing American options by simulation: A simple least-squares approach. Rev. Financial Stud. 14(1), 113–147 (2001)CrossRefGoogle Scholar
  27. 27.
    Egloff, D.: Monte carlo algorithms for optimal stopping and statistical learning. Ann. Appl. Probab. 15(2), 1396–1432 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Ludkovski, M., Niemi, J.: Optimal dynamic policies for influenza management, Statistical Commun. Infect. Dis. (2010)Google Scholar
  29. 29.
    Bertsekas, D.P., Dynamic Programming and Optimal Control, vol. 2. Belmont, MA: Athena Scientific (2012)Google Scholar
  30. 30.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning, 2nd edn. Springer (2009)Google Scholar
  31. 31.
    Belomestny, D., Kolodko, A., Schoenmakers, J.: Regression methods for stochastic control problems and their convergence analysis. SIAM J. on Control and Optim. 48(5), 3562–3588 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Vrabie, D., Vamvoudakis, K.G., Lewis, F.L.: Optimal adaptive control and differential games by reinforcement learning principles, vol. 81. IET (2013)Google Scholar
  33. 33.
    Oh, H., Kim, S., Shin, H.-S., Tsourdos, A., White, B.: Coordinated standoff tracking of groups of moving targets using multiple UAVs. In: Control & Automation (MED), 2013 21st Mediterranean Conf. on, 969–977, IEEE (2013)Google Scholar
  34. 34.
    Gramacy, R.B., Ludkovski, M.: Sequential design for optimal stopping problems,” SIAM J. on Financial Math. Note Accepted subject to minor revision (2015)Google Scholar
  35. 35.
    Carrillo, L., Russell, W., Hespanha, J., Collins, G.: State estimation of multiagent systems under impulsive noise and disturbances. IEEE Trans. Control Syst. Technol. 23, 13–26 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Steven A. P. Quintero
    • 1
    Email author
  • Michael Ludkovski
    • 2
  • João P. Hespanha
    • 1
  1. 1.Department of Electrical and Computer EngineeringUniversity of CaliforniaSanta BarbaraUSA
  2. 2.Department of Statistics and Applied ProbabilityUniversity of CaliforniaSanta BarbaraUSA

Personalised recommendations