Markov Decision Processes with Multiple Objectives

  • Krishnendu Chatterjee
  • Rupak Majumdar
  • Thomas A. Henzinger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3884)


We consider Markov decision processes (MDPs) with multiple discounted reward objectives. Such MDPs occur in design problems where one wishes to simultaneously optimize several criteria, for example, latency and power. The possible trade-offs between the different objectives are characterized by the Pareto curve. We show that every Pareto-optimal point can be achieved by a memoryless strategy; however, unlike in the single-objective case, the memoryless strategy may require randomization. Moreover, we show that the Pareto curve can be approximated in polynomial time in the size of the MDP. Additionally, we study the problem if a given value vector is realizable by any strategy, and show that it can be decided in polynomial time; but the question whether it is realizable by a deterministic memoryless strategy is NP-complete. These results provide efficient algorithms for design exploration in MDP models with multiple objectives.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Etzioni, O., Hanks, S., Jiang, T., Karp, R.M., Madari, O., Waarts, O.: Efficient information gathering on the internet. In: FOCS 1996, pp. 234–243. IEEE, Los Alamitos (1996)Google Scholar
  2. 2.
    Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer, Heidelberg (1997)zbMATHGoogle Scholar
  3. 3.
    Garey, M.R., Johnson, D.S.: Computers and Intractability. W.H. Freeman, New York (1979)zbMATHGoogle Scholar
  4. 4.
    Hartley, R.: Finite Discounted Vector Markov Decision Processes. Technical Report, Department of Decision Theory, Manchester University (1979)Google Scholar
  5. 5.
    Koski, J.: Multicriteria truss optimization. In: Multicriteria Optimization in Engineering and in the Sciences (1988)Google Scholar
  6. 6.
    Owen, G.: Game Theory. Academic Press, London (1995)zbMATHGoogle Scholar
  7. 7.
    Papadimitriou, C.H., Yannakakis, M.: On the approximability of trade-offs and optimal access of web sources. In: FOCS 2000, pp. 86–92. IEEE, Los Alamitos (2000)Google Scholar
  8. 8.
    Puterman, M.L.: Markov Decision Processes. Wiley, Chichester (1994)CrossRefzbMATHGoogle Scholar
  9. 9.
    Szymanek, R., Catthoor, F., Kuchcinski, K.: Time-energy design space exploration for multi-layer memory architectures. In: DATE 2004, IEEE, Los Alamitos (2004)Google Scholar
  10. 10.
    White, D.J.: Multi-objective infinite-horizon discounted Markov decision processes. Journal of Mathematical Analysis and Applications 89, 639–647 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Yang, P., Catthoor, F.: Pareto-optimization based run time task scheduling for embedded systems. In: CODES-ISSS 2003, pp. 120–125. ACM, New York (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Krishnendu Chatterjee
    • 1
  • Rupak Majumdar
    • 2
  • Thomas A. Henzinger
    • 1
    • 3
  1. 1.UC BerkeleyUSA
  2. 2.UC Los AngelesUSA
  3. 3.EPFLSwitzerland

Personalised recommendations