Skip to main content
Log in

Event-based optimization approach for solving stochastic decision problems with probabilistic constraint

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

In many practical control systems or management systems, the manager of systems may allow that the statistic probability of system error or parameter deviation occurs within a certain range. The problem of decision optimization under probabilistic constraint is thus an issue needs to be addressed urgently. In this paper, we consider to develop an event-based approach which can solve the probabilistic constrained decision problems in discrete events dynamic systems. The framework of the event-based optimization is first introduced, and then with the methodology of the performance sensitivity analysis, we present an online event-based policy iteration algorithm based on the derived performance gradient formula. We apply the event-based idea and propose the concept of “risk state”, “risk event” and “risk index” which can be used to better describe the nature of the probabilistic constrained problem. Furthermore, by taking the Lagrangian approach, the constrained decision problem can be solved with two steps. Finally, numerical experiments are designed to verify the efficiency of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Puterman, M.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, New York (1994)

    Book  Google Scholar 

  2. Aldhaheri, R., Khalil, H.: Aggregation of the policy iteration method for nearly completely decomposable Markov chains. IEEE Trans. Autom. Control 36(2), 178–187 (1991)

    Article  MathSciNet  Google Scholar 

  3. Ren, Z., Krogh, B.: State aggregation in Markov decision processes. In: Conference on Decision and Control. IEEE, pp. 3819–3824. Pittsburgh, USA (2002)

  4. Cao, X., Ren, Z., Bhatnagar, S., et al.: A time aggregation approach to Markov decision processes. Automatica 38(6), 929–943 (2002)

    Article  MathSciNet  Google Scholar 

  5. Sun, T., Zhao, Q., Luh, P.: Incremental value iteration for time-aggregated Markov-decision processes. IEEE Trans. Autom. Control 52(11), 2177–2182 (2007)

    Article  MathSciNet  Google Scholar 

  6. Powell, W.: Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, New York (2007)

    Book  Google Scholar 

  7. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT press, Cambridge (1998)

    MATH  Google Scholar 

  8. Wan, J., Liu, L., Guo, J.: Dynamic request routing for online video-on-demand service: a Markov decision process approach. Math. Probl. Eng. (2014). https://doi.org/10.1155/2014/920829

    Article  MathSciNet  MATH  Google Scholar 

  9. Zhang, S., Yang, J., Shi, Y., et al.: Dynamic energy storage control for reducing electricity cost in data centers. Math. Probl. Eng. (2015). https://doi.org/10.1155/2015/380926

    Article  Google Scholar 

  10. Jiang, X., Xi, H., Wang, X., et al.: Finding optimal observation-based policies for constrained POMDPs under the expected average reward criterion. IEEE Trans. Autom. Control 61(10), 3070–3075 (2016)

    Article  MathSciNet  Google Scholar 

  11. Cao, X., Chen, H.: Perturbation realization, potentials and sensitivity analysis of Markov processes. IEEE Trans. Autom. Control 42(10), 1382–1393 (1997)

    Article  MathSciNet  Google Scholar 

  12. Cao, X.: Basic ideas for event-based optimization of Markov systems. Discrete Event Dyn. Syst. Theory Appl. 15(2), 169–197 (2005)

    Article  MathSciNet  Google Scholar 

  13. Cao, X.: Stochastic Learning and Optimization: A Sensitivity-Based Approach. Springer, New York (2007)

    Book  Google Scholar 

  14. Cao, X., Zhang, J.: Event-based optimization of Markov systems. IEEE Trans. Autom. Control 53(4), 1076–1082 (2008)

    Article  MathSciNet  Google Scholar 

  15. Xu, C., Yang, J., Xi, H., et al.: Event-related optimization for a class of resource location with admission control. In: International Joint Conference on Neural Networks. IEEE, Hefei, China, pp. 1092–1097 (2008)

  16. Zhao, Y., Zhao, Q., Jia, Q., et al.: Event-related optimization for a class of resource location with admission control. In: Conference on Decision and Control. IEEE, Beijing, China, pp. 2173–2178 (2008)

  17. Sun, B., Luh, P.B., Jia, Q.S., et al.: Event-based optimization within the Lagrangian relaxation framework for energy savings in HVAC systems. IEEE Trans. Autom. Sci. Eng. 12(4), 1396–1406 (2015)

    Article  Google Scholar 

  18. Parlar, M., Rodrigues, B., Sharafali, M.: Event-based allocation of airline check-in counters: a simple dynamic optimization method supported by empirical data. Int. Trans. Oper. Res. 25(5), 1553–1582 (2018)

    Article  MathSciNet  Google Scholar 

  19. Jia, Q.: On solving optimal policies for event-based dynamic programming. In: Chinese Control Conference. IEEE, Beijing. China, pp. 1511–1516 (2010)

  20. Jia, Q.: On solving event-based optimization with average reward over infinite stages. IEEE Trans. Autom. Control 56(12), 2912–2917 (2011)

    Article  Google Scholar 

  21. Jia, Q.: Event-based optimization with lagged state information. In: Chinese Control Conference. IEEE, Hefei. China, pp. 2055–2060 (2012)

  22. Xia, L., Jia, Q., Cao, X.: A tutorial on event-based optimization-a new optimization framework. Discrete Event Dyn. Syst. Theory Appl. 24(2), 103–132 (2014)

    Article  MathSciNet  Google Scholar 

  23. Xia, L.: Event-based optimization of admission control in open queueing networks. Discrete Event Dyn. Syst. Theory Appl. 24(2), 133–151 (2014)

    Article  MathSciNet  Google Scholar 

  24. Xia, L.: Policy gradient approach of event-based optimization and its online implementation. Asian J. Control 16(6), 1735–1743 (2014)

    Article  MathSciNet  Google Scholar 

  25. Pietrabissa, A.: Admission control in UMTS networks based on approximate dynamic programming. Eur. J. Control 14(1), 62–75 (2008)

    Article  MathSciNet  Google Scholar 

  26. Bhatnagar, S., Lakshmanan, K.: An online actor-critic algorithm with function approximation for constrained Markov decision processes. J. Optim. Theory Appl. 153(3), 688–708 (2012)

    Article  MathSciNet  Google Scholar 

  27. Djonin, D., Krishnamurthy, V.: Q-learning algorithms for constrained Markov decision processes with randomized monotone policies: application to MIMO transmission control. IEEE Trans. Signal Process. 55(5), 2170–2181 (2007)

    Article  MathSciNet  Google Scholar 

  28. Sun, C., Stevens, E., Shah, V., et al.: A constrained MDP-based vertical handoff decision algorithm for 4G heterogeneous wireless networks. Wirel. Netw. 17(4), 1063–1081 (2011)

    Article  Google Scholar 

  29. Calafiore, G., Dabbene, F.: Probabilistic and Randomized Methods for Design Under Uncertainty. Springer, London (2006)

    Book  Google Scholar 

  30. Cannon, M., Kouvaritakis, B., Rakovic, V., et al.: Stochastic tubes in model predictive control with probabilistic constraints. IEEE Trans. Autom. Control 56(1), 194–200 (2011)

    Article  MathSciNet  Google Scholar 

  31. Chung, J., Du, H., Gondzio, J.: A probabilistic constraint approach for robust transmit beamforming with imperfect channel information. IEEE Trans. Signal Process. 59(6), 2773–2782 (2011)

    Article  MathSciNet  Google Scholar 

  32. Uryasev, S.: Probabilistic Constrained Optimization: Methodology and Applications. Springer, Gainesville (2013)

    MATH  Google Scholar 

  33. Li, Y., Cao, F.: A basic formula for performance gradient estimation of semi-Markov decision processes. Eur. J. Oper. Res. 224(2), 333–339 (2013)

    Article  MathSciNet  Google Scholar 

  34. Mundur, P., Sood, A.K., Simon, R.: Class-based access control for distributed video-on-demand systems. IEEE Trans. Circuits Syst. Video Technol. 15(7), 844–853 (2005)

    Article  Google Scholar 

  35. Yin, B., Lu, S., Guo, D.: Analysis of admission control in P2P-based media delivery network based on POMDP. Int. J. Innov. Comput. Inf. Control 7(7B), 4411–4422 (2011)

    Google Scholar 

Download references

Acknowledgements

This work is supported by ‘the Natural Science Foundation of Anhui Province’ (No. 1808085QG220, 1708085QG164), ‘the National Natural Science Foundation of China’ (No. 71601066, 71501055, 71690230, 71690235), ‘the Humanities and Social Science Foundation of Ministry of Education in China’ (No. 16YJC630093).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhanglin Peng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, X., Peng, Z., Zhang, Q. et al. Event-based optimization approach for solving stochastic decision problems with probabilistic constraint. Optim Lett 15, 569–590 (2021). https://doi.org/10.1007/s11590-019-01403-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-019-01403-2

Keywords

Navigation