Markov Decision Processes in Practice

  • Richard J. Boucherie
  • Nico M. van Dijk

Part of the International Series in Operations Research & Management Science book series (ISOR, volume 248)

Table of contents

  1. Front Matter
    Pages i-xxxii
  2. General Theory

    1. Front Matter
      Pages 1-1
    2. Martijn R. K. Mes, Arturo Pérez Rivera
      Pages 63-101
    3. András Mészáros, Miklós Telek
      Pages 103-129
  3. Healthcare

    1. Front Matter
      Pages 187-187
    2. Lauren N. Steimle, Brian T. Denton
      Pages 189-222
    3. J. W. M. Otten, A. Witteveen, I. M. H. Vliegen, S. Siesling, J. B. Timmer, M. J. IJzerman
      Pages 223-244
    4. Antoine Sauré, Martin L. Puterman
      Pages 245-268
    5. C. J. Jagtenberg, S. Bhulai, R. D. van der Mei
      Pages 269-291
    6. Rene Haijema, Nico M. van Dijk, Jan van der Wal
      Pages 293-317
  4. Transportation

    1. Front Matter
      Pages 319-319
    2. T. R. Meerburg, Richard J. Boucherie, M. J. A. L. van Kraaij
      Pages 321-335
    3. M. Fackrell, P. Taylor
      Pages 337-370
    4. Rene Haijema, Eligius M. T. Hendrix, Jan van der Wal
      Pages 371-386
    5. Pia L. Kempker, Nico M. van Dijk, Werner Scheinhardt, Hans van den Berg, Johann Hurink
      Pages 387-404
  5. Production

    1. Front Matter
      Pages 405-405
    2. Nicky D. van Foreest, Jacob Wijngaard
      Pages 407-423
    3. Eligius M. T. Hendrix, Rene Haijema, Diana van Dijk
      Pages 425-438
    4. Daphne van Leeuwen, Rudesindo Núñez-Queija
      Pages 439-459
  6. Communications

    1. Front Matter
      Pages 461-461
    2. Julia Kuhn, Yoni Nazarathy
      Pages 463-485
    3. Alex Roubos, Sandjai Bhulai, Ger Koole
      Pages 487-503
    4. Mihaela Mitici
      Pages 505-519
  7. Financial Modeling

    1. Front Matter
      Pages 521-521
  8. Back Matter
    Pages 547-550

About this book


This book presents classical Markov Decision Processes (MDP) for real-life applications and optimization. MDP allows users to develop and formally support approximate and simple decision rules, and this book showcases state-of-the-art applications in which MDP was key to the solution approach.  The book is divided into six parts. Part 1 is devoted to the state-of-the-art theoretical foundation of MDP, including approximate methods such as policy improvement, successive approximation and infinite state spaces as well as an instructive chapter on Approximate Dynamic Programming. It then continues with five parts of specific and non-exhaustive application areas. Part 2 covers MDP healthcare applications, which includes different screening procedures, appointment scheduling, ambulance scheduling and blood management. Part 3 explores MDP modeling within transportation.  This ranges from public to private transportation, from airports and traffic lights to car parking or charging your electric car. Part 4 contains three chapters that illustrates the structure of approximate policies for  production or manufacturing structures. In Part 5, communications is highlighted as an important application area for MDP. It includes Gittins indices, down-to-earth call centers and wireless sensor networks. Finally Part 6 is dedicated to financial modeling, offering an instructive review to account for financial portfolios and derivatives under proportional transactional costs. The MDP applications in this book illustrate a variety of both standard and non-standard aspects of MDP modeling and its practical use. This book should appeal to readers for practitioning, academic research and educational purposes, with a background in, among others, operations research, mathematics, computer science, and industrial engineering.


Approximate Dynamic Programming Computational Research Markov Decision Processes Neuro Dynamic Programming Probabilistic Modeling Stochastic Operations Research

Editors and affiliations

  • Richard J. Boucherie
    • 1
  • Nico M. van Dijk
    • 2
  1. 1.Stochastic Operations ResearchUniversity of TwenteEnschedeThe Netherlands
  2. 2.Stochastic Operations ResearchUniversity of TwenteEnschedeThe Netherlands

Bibliographic information