Autonomous Robots

, Volume 39, Issue 4, pp 555–571 | Cite as

Chance-constrained dynamic programming with application to risk-aware robotic space exploration

  • Masahiro OnoEmail author
  • Marco Pavone
  • Yoshiaki Kuwata
  • J. Balaram


Existing approaches to constrained dynamic programming are limited to formulations where the constraints share the same additive structure of the objective function (that is, they can be represented as an expectation of the summation of one-stage costs). As such, these formulations cannot handle joint probabilistic (chance) constraints, whose structure is not additive. To bridge this gap, this paper presents a novel algorithmic approach for joint chance-constrained dynamic programming problems, where the probability of failure to satisfy given state constraints is explicitly bounded. Our approach is to (conservatively) reformulate a joint chance constraint as a constraint on the expectation of a summation of indicator random variables, which can be incorporated into the cost function by considering a dual formulation of the optimization problem. As a result, the primal variables can be optimized by standard dynamic programming, while the dual variable is optimized by a root-finding algorithm that converges exponentially. Error bounds on the primal and dual objective values are rigorously derived. We demonstrate algorithm effectiveness on three optimal control problems, namely a path planning problem, a Mars entry, descent and landing problem, and a Lunar landing problem. All Mars simulations are conducted using real terrain data of Mars, with four million discrete states at each time step. The numerical experiments are used to validate our theoretical and heuristic arguments that the proposed algorithm is both (i) computationally efficient, i.e., capable of handling real-world problems, and (ii) near-optimal, i.e., its degree of conservatism is very low.


Dynamic programming Constrained stochastic optimal control Chance-constrained optimization Markov decision processes Path planning 



The research described in this paper was conducted at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. Marco Pavone was supported in part by NASA under the Space Technology Research Grants Program, Grant NNX12AQ43G.


  1. Acikmese, B., & Ploen, S. (2007). Convex programming approach to powered descent guidance for mars landing. AIAA Journal of Guidance, Control, and Dynamics, 30(5), 1353–1366.CrossRefGoogle Scholar
  2. Acikmese, B., Carson, J. M., & Blackmore, L. (2013). Lossless convexification of nonconvex control bound and pointing constraints of the soft landing optimal control problem. IEEE Transactions on Control System Technology, 21, 2104–2113.Google Scholar
  3. Altman, E. (1999). Constrained Markov decision processes. Stochastic modeling. Boca Raton: Chapman & Hall/CRC.Google Scholar
  4. Askew, A. J. (1974). Chance-constraind dynamic programming and the optimization of water resource systems. Water Resources Research, 10, 1099–1106.CrossRefGoogle Scholar
  5. Askew, A. J. (1974). Optimum reservoir operating policies and the imposition of a reliability constraint. Water Resources Research, 10, 51–56.CrossRefGoogle Scholar
  6. Atkinson, K. E. (1989). An introduction to numerical analysis (2nd ed.). New York: wiley.zbMATHGoogle Scholar
  7. Bertsekas, D. P. (1999). Nonlinear programming. Belmont: Athena Scientific.zbMATHGoogle Scholar
  8. Bertsekas, D. P. (1999). Nonlinear Programming (2nd ed.). Belmont: Athena Scientific.zbMATHGoogle Scholar
  9. Beutler, F. J., & Ross, K. W. (1985). Optimal policies for controlled markov chains with a constraint. Journal of Mathematical Analysis and Applications, 112, 236–252.zbMATHMathSciNetCrossRefGoogle Scholar
  10. Biesiadecki, J. J., & Maimone, M. (1985). The mars exploration rover surface mobility flight software: Driving ambition. In IEEE Aerospace Conference Proceedings 2006.Google Scholar
  11. Blackmore, L., Acikmese, B., & Scharf, D. P. (2010). Minimum-landing-error powered-descent guidance for mars landing using convex optimization. AIAA Journal of Guidance Control and Dynamics, 33, 1161–1171.CrossRefGoogle Scholar
  12. Blackmore, L., Li, H., & Williams, B. C. (2006). A probabilistic approach to optimal robust path planning with obstacles. In Proceedings of American Control Conference.Google Scholar
  13. Blackmore, L., & Ono, M. (2009). Convex chance constrained predictive control without sampling. In Proceedings of the AIAA Guidance, Navigation and Control Conference.Google Scholar
  14. Blackmore, L., Ono, M., Bektassov, A., & Williams, B. C. (2010). A probabilistic particle-control approximation of chance-constrained stochastic predictive control. IEEE Transactions on Robotics, 26(3), 502–517.CrossRefGoogle Scholar
  15. Charnes, A., & Cooper, W. W. (1959). Chance-constrained programming. Management Science, 6, 73–79.zbMATHMathSciNetCrossRefGoogle Scholar
  16. Chow, Y. L., & Pavone, M. (2014). A framework for time-consistent, risk-averse model predictive control: Theory and algorithms. In American Control Conference, (pp. 4204–4211).Google Scholar
  17. Couchman, P., Cannon, M., & Kouvaritakis, B. (2006). Stochastic MPC with inequality stability constraints. Automatica, 42(12), 2169–2174.zbMATHMathSciNetCrossRefGoogle Scholar
  18. Estlin, T. A., Bornstein, B. J., Gaines, D. M., Anderson, R. C., Thompson, D. R., Burl, M., et al. (2012). Aegis: Automated science targeting for the mer opportunity rover. ACM Transactions on Intelligent Systems and Technology, 3(3), 50.CrossRefGoogle Scholar
  19. Fleming, W., & McEneaney, W. (1995). Risk-sensitive control on an infinite time horizon. SIAM Journal on Control and Optimization, 33(6), 1881–1915.zbMATHMathSciNetCrossRefGoogle Scholar
  20. Harris, M., & Akmee, B. (2014). Maximum divert for planetary landing using convex optimization. Journal of Optimization Theory and Applications, 162(3), 975–995. doi: 10.1007/s10957-013-0501-7.zbMATHMathSciNetCrossRefGoogle Scholar
  21. Hokayem, P., Chatterjee, D., & Lygeros, J. (2013). Chance-constrained LQG with bounded contr ol policies. In IEEE Conference on Decision and Control (pp. 2471–2476). Italy: Florence.Google Scholar
  22. Johnson, A., Huertas, A., Werner, R., & Montgomery, J. (2008). Analysis of On-Board Hazard Detection and Avoidance for Safe Lunar Landing. In IEEE Aerospace Conference.Google Scholar
  23. Knocke, P. C., Wawrzyniak, G. G., Kennedy, B. M., Desai, P. N., Parker, T. J., Golombek, M. P., Duxbury, T. C., & Kass, D. M. (2004) Mars exploration rovers landing dispersion analysis. In Proceedings of AIAA/AAS Astrodynamics Specialist Conferencec and Exhibit.Google Scholar
  24. Kuwata, Y., & Balaram, J. B. (2011). Combined EDL-mobility planning for planetary missions. In Proceedings of Infotech@Aerospace.Google Scholar
  25. Kuwata, Y., Pavone, M., & Balaram, J. B. (2012). A risk-constrained multi-stage decision making approach to the architectural analysis of planetary missions. In Proceedings of the IEEE Conference on Decision and Control.Google Scholar
  26. Li, P., Wendt, M., & Wozny, G. (2002). A probabilistically constrained model predictive controller. Automatica, 38, 1171–1176.zbMATHMathSciNetCrossRefGoogle Scholar
  27. McGhan, C. L. R., Serra, R., Ingham, M. D., Ono, M., Estlin, T., Murray, R. M., & Williams, B. C. (2015). A risk-aware architecture for resilient spacecraft operations. In Accepted for publication in the Proceedings of IEEE aerospace conference.Google Scholar
  28. Murray, R. M., Day, J. C., Ingham, M. D., Reder, L. J., & Williams, B. C. (2013). Engineering resilient space systems: Final report.Google Scholar
  29. National Aeronautics and Space Administration: Gravity Recovery and Interior Laboratory (GRAIL) Launch Press Kit.Google Scholar
  30. National Research Council of the National Academies: Vision and voyages for planetary science in the decade 2013-2022 (2010).Google Scholar
  31. Nemirovski, A., & Shapiro, A. (2006). Convex approximations of chance constrained programs. SIAM Journal on Optimization, 17, 969–996.zbMATHMathSciNetCrossRefGoogle Scholar
  32. Oldewurtel, F., Jones, C. N., & Morari, M. (2008). A tractable approximation of chance constrained stochastic MPC based on affine disturbance feedback. In Proceedings of Conference on Decision and Control.Google Scholar
  33. Oldewurtel, F., Ulbig, A., Parisio, A., Andersson, G., & Morari, M. (2010). Reducing peak electricity demand in building climate control using real-time pricing and model predictive control. In Proceedings of Conference on Decision and Control.Google Scholar
  34. Ono, M. (2012). Joint chance-constrained model predictive control with probabilistic resolvability. In Proceedings of American Control Conference.Google Scholar
  35. Ono, M. (2012). Robust, goal-directed plan execution with bounded risk. Ph.D. thesis, Massachusetts Institute of Technology.Google Scholar
  36. Ono, M., Topcu, U., Yo, M., & Adachi, S. (2013). Risk-limiting power grid control with an arma-based prediction model. In IEEE 52nd Annual Conference on Decision and Control (CDC), 2013, (pp. 4949–4956).Google Scholar
  37. Ono, M., & Williams, B. C. (2008). Iterative risk allocation: A new approach to robust model predictive control with a joint chance constraint. In Proceedings of 47th IEEE Conference on Decision and Control.Google Scholar
  38. Parker, J. S., & Anderson, R. L. (2014). Low-energy lunar trajectory design. New York: Wiley.CrossRefGoogle Scholar
  39. Prakash, R., Burkhart, P., Chen, A., Comeaux, K., Guernsey, C., Kipp, D., Lorenzoni, L., Mendeck, G., Powell, R., Rivellini, T., San Martin, A., Sell, S., Steltzner, A., & Way, D. (2008). Mars science laboratory entry, descent, and landing system overview. In IEEE Conference on Aerospace (pp. 1–18).Google Scholar
  40. Rossman, L. A. (1977). Reliability-constrained dynamic programing and randomized release rules in reservoir management. Water Resources Research, 13, 247–255.CrossRefGoogle Scholar
  41. Schildbach, G., Goulart, P., & Morari, M. (2013). The linear quadratic regulator with chance constraints. In European Control Conference (ECC) (pp. 2746–2751).Google Scholar
  42. Schwarm, A. T., & Nikolaou, M. (1999). Chance-constrained model predictive control. AIChE Journal, 45(8), 1743–1752.CrossRefGoogle Scholar
  43. Sjödin, A. E., Gayme, D. F., & Topcu, U. (2012). Risk-mitigated optimal power flow with high wind penetration. In Proceedings of American Control Conference (2012).Google Scholar
  44. Summers, T., Warrington, J., Morari, M., & Lygeros, J. (2014). Stochastic optimal power flow based on convex approximations of chance constraints. In Power Systems Computation Conference. Wroclaw, Poland.Google Scholar
  45. van Hessem, D. H. (2004). Stochastic inequality constrained closed-loop model predictive control with application to chemical process operation. Ph.D. thesis, Delft University of Technology.Google Scholar
  46. Vitus, M. P., & Tomlin, C. J. (2011). On feedback design and risk allocation in chance constrained control. In Proceedings of IEEE Conference on Decision and Control (pp. 734–739).Google Scholar
  47. Williams, J. L., Fisher, J. W., & Willsky, A. S. (2007). Approximate dynamic programming for communication-constrained sensor network management. IEEE Transactions on Signal Processing, 55, 4300–4311.MathSciNetCrossRefGoogle Scholar
  48. Yo, M., Ono, M., Williams, B., & Adachi, S. (2013). Risk-limiting, market-based power dispatch and pricing. In European Control Conference (pp. 3038–3045).Google Scholar
  49. Zhang, X., Grammatico, S., Schildbach, G., Goulart, P., & Lygeros, J. (2014). On the sample size of randomized MPC for chance-constrained systems with application to building climate control. In European Control Conference (pp. 478–483). Strasbourg, France.Google Scholar

Copyright information

© Springer Science+Business Media New York (outside the USA) 2015

Authors and Affiliations

  1. 1.Jet Propulsion LaboratoryCalifornia Institute of TechnologyPasadenaUSA
  2. 2.Department of Aeronautics and AstronauticsStanford UniversityStanfordUSA
  3. 3.Guidance, Control, and Navigation Division, Space Exploration Technologies Corporation (SpaceX)HawthorneUSA

Personalised recommendations