Markov Decision Processes with a Constraint on the Asymptotic Failure Rate

  • M. Boussemart
  • T. Bickard
  • N. Limnios


In this paper, we introduce a Markov decision model with absorbing states and a constraint on the asymptotic failure rate. The objective is to find a stationary policy which minimizes the infinite horizon expected average cost, given that the system never fails. Using Perron-Frobenius theory of non-negative matrices and spectral analysis, we show that the problem can be reduced to a linear programming problem. Finally, we apply this method to a real problem for an aeronautical system.

constrained Markov decision process asymptotic failure rate primitive matrices linear programming 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. E. Altman and A. Shwartz, “Markov decision problems and state-action frequencies,” SIAM Journal and Optimization vol. 29 pp. 786-809, 1991.Google Scholar
  2. R. E. Bellman, “A Markovian decision process,” Journal of Mathematics and Mechanics vol. 6 pp. 679-684, 1957.Google Scholar
  3. A. Berman and R. J. Plemmons, Nonnegative matrices in the mathematical sciences, Academic Press, 1979.Google Scholar
  4. F. J. Beutler and K. W. Ross, “Optimal policies for controlled Markov chains with a constraint,” Journal of Mathematical Analysis and Applications vol. 111 pp. 236-252, 1985.Google Scholar
  5. T. Bickard, N. Hubart, and J. L. Lanet, “Future jet engine control systems,” Proc. of the 1997 European Propulsion Forum, Design of Aero Engines for Safety, Reliability, Maintainability and Cost, Berlin, Germany, pp. 149-153, March 1997.Google Scholar
  6. J. F. Bonnans and J. C. Gilbert, Optimisation Numérique, Springer-Verlag, 1997.Google Scholar
  7. M. Boussemart, Modélisation de la maintenance des calculateurs à architecture duale et répartie, utilisés pour la régulation des turboréacteurs, DEA report, Control of Systems UTC, 1998.Google Scholar
  8. P. G. Ciarlet, Introduction à l'analyse numérique matricielle et à l' optimisation, Masson, 1982.Google Scholar
  9. C. Derman, Finite State Markovian Decision Processes, Academic Press: New York and London, 1970.Google Scholar
  10. J. Filar and K. Vrieze, Competitive Markov Decision Proceses, Springer-Verlag: New York, 1997.Google Scholar
  11. F. R. Gantmacher, The Theory of Matrices, Tome I: New York, 1959.Google Scholar
  12. I. B. Gertsbakh, Models of preventive maintenance, Studies in mathematical and managerial economics, North-Holland Publishing Company, 1977.Google Scholar
  13. R. A. Howard, Dynamic Programming and Markov Process, Wiley and Sons, 1960.Google Scholar
  14. V. G. Kulkarni, Modeling and analysis of stochastic systems, Chapman &; Hall, 1995.Google Scholar
  15. N. Limnios and G. Oprisan, Semi-Markov Processes and Reliability, Birkhäuser: Boston, 2001.Google Scholar
  16. B. H. Lindqvist and H. Amundrustad, “Markov model for periodically tested components,” ESREL'98, Trondheim, Norway, June 1998.Google Scholar
  17. M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley: New York, 1994.Google Scholar
  18. E. Seneta, Non-negative Matrices and Markov Chains, Springer-Verlag, 1981.Google Scholar
  19. D. J. White, Markov Decision Process, Wiley and Sons, 1993.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • M. Boussemart
    • 1
    • 2
  • T. Bickard
    • 2
  • N. Limnios
    • 1
  1. 1.Equipe de Mathématiques AppliquéesUniversité de Technologie de CompiègneCompiègneFrance
  2. 2.Snecma Control SystemsVillarocheFrance

Personalised recommendations