Abstract
In this paper, we propose an exact solution method to generate fair policies in Multiobjective Markov Decision Processes (MMDPs). MMDPs consider n immediate reward functions, representing either individual payoffs in a multiagent problem or rewards with respect to different objectives. In this context, we focus on the determination of a policy that fairly shares regrets among agents or objectives, the regret being defined on each dimension as the opportunity loss with respect to optimal expected rewards. To this end, we propose to minimize the ordered weighted average of regrets (OWR). The OWR criterion indeed extends the minimax regret, relaxing egalitarianism for a milder notion of fairness. After showing that OWR-optimality is state-dependent and that the Bellman principle does not hold for OWR-optimal policies, we propose a linear programming reformulation of the problem. We also provide experimental results showing the efficiency of our approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Altman, E.: Constrained Markov Decision Processes. CRC Press, Boca Raton (1999)
Boutilier, C.: Sequential optimality and coordination in multiagent systems. In: Proc. IJCAI (1999)
Chatterjee, K., Majumdar, R., Henzinger, T.: Markov decision processes with multiple objectives. In: Durand, B., Thomas, W. (eds.) STACS 2006. LNCS, vol. 3884, pp. 325–336. Springer, Heidelberg (2006)
Desrosiers, J., Luebbecke, M.: A primer in column generation. In: Desaulniers, G., Desrosier, J., Solomon, M. (eds.) column generation, pp. 1–32. Springer, Heidelberg (2005)
Furukawa, N.: Vector-valued Markovian decision processes with countable state space. In: Recent Developments in MDPs, vol. 36, pp. 205–223 (1980)
Geoffrion, A.: Proper efficiency and the theory of vector maximization. J. Math. Anal. Appls. 22, 618–630 (1968)
Guestrin, C., Koller, D., Parr, R.: Multiagent planning with factored MDPs. In: NIPS (2001)
Hansen, P.: Bicriterion Path Problems. In: Multiple Criteria Decision Making Theory and Application, pp. 109–127. Springer, Heidelberg (1979)
Kostreva, M., Ogryczak, W., Wierzbicki, A.: Equitable aggregations and multiple criteria analysis. Eur. J. Operational Research 158, 362–367 (2004)
Littman, M.L., Dean, T.L., Kaelbling, L.P.: On the complexity of solving Markov decision problems. In: UAI, pp. 394–402 (1995)
Llamazares, B.: Simple and absolute special majorities generated by OWA operators. Eur. J. Operational Research 158, 707–720 (2004)
Marshall, A., Olkin, I.: Inequalities: Theory of Majorization and its Applications. Academic Press, London (1979)
Mouaddib, A.: Multi-objective decision-theoretic path planning. IEEE Int. Conf. Robotics and Automation 3, 2814–2819 (2004)
Ogryczak, W., Sliwinski, T.: On solving linear programs with the ordered weighted averaging objective. Eur. J. Operational Research 148, 80–91 (2003)
Puterman, M.: Markov decision processes: discrete stochastic dynamic programming. Wiley, Chichester (1994)
Steuer, R.: Multiple criteria optimization. John Wiley, Chichester (1986)
Viswanathan, B., Aggarwal, V., Nair, K.: Multiple criteria Markov decision processes. TIMS Studies in the Management Sciences 6, 263–272 (1977)
White, D.: Multi-objective infinite-horizon discounted Markov decision processes. J. Math. Anal. Appls. 89, 639–647 (1982)
Yager, R.: On ordered weighted averaging aggregation operators in multi-criteria decision making. IEEE Trans. on Syst., Man and Cyb. 18, 183–190 (1988)
Yager, R.: Decision making using minimization of regret. Int. J. of Approximate Reasoning 36, 109–128 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ogryczak, W., Perny, P., Weng, P. (2011). On Minimizing Ordered Weighted Regrets in Multiobjective Markov Decision Processes. In: Brafman, R.I., Roberts, F.S., Tsoukià s, A. (eds) Algorithmic Decision Theory. ADT 2011. Lecture Notes in Computer Science(), vol 6992. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24873-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-642-24873-3_15
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-24872-6
Online ISBN: 978-3-642-24873-3
eBook Packages: Computer ScienceComputer Science (R0)