A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes

  • Nicolás Wolovick
  • Sven Johr
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4202)


Continuous-time Markov decision process are an important variant of labelled transition systems having nondeterminism through labels and stochasticity through exponential fire-time distributions. Nondeterministic choices are resolved using the notion of a scheduler. In this paper we characterize the class of measurable schedulers, which is the most general one, and show how a measurable scheduler induces a unique probability measure on the sigma-algebra of infinite paths. We then give evidence that for particular reachability properties it is sufficient to consider a subset of measurable schedulers. Having analyzed schedulers and their induced probability measures we finally show that each probability measure on the sigma-algebra of infinite paths is indeed induced by a measurable scheduler which proves that this class is complete.


Probability Measure Sojourn Time Markov Decision Process Combine Transition Negative Exponential Distribution 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. II. Athena Scientific (1995)Google Scholar
  2. 2.
    Feinberg, E.A.: Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach. Mathematics of Operations Research 29, 492–524 (2004)MATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Chichester (1994)MATHGoogle Scholar
  4. 4.
    Sennot, L.: Stochastic Dynamic Programming and the Control of Queueing Systems. John Wiley & Sons, Chichester (1999)Google Scholar
  5. 5.
    Abdeddaïm, Y., Asarin, E., Maler, O.: On Optimal Scheduling under Uncertainty. In: Garavel, H., Hatcliff, J. (eds.) ETAPS 2003 and TACAS 2003. LNCS, vol. 2619, pp. 240–253. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  6. 6.
    Bruno, J., Downey, P., Frederickson, G.N.: Sequencing Tasks with Exponential Service Times to Minimize the Expected Flow Time or Makespan. J. ACM 28, 100–113 (1981)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Qiu, Q., Pedram, M.: Dynamic power managment based on continuous-time Markov decision processes. In: Proceedings DAC, pp. 555–561 (1999)Google Scholar
  8. 8.
    Baier, C., Haverkort, B., Hermanns, H., Katoen, J.P.: Efficient Computation of Time-Bounded Reachability Probabilities in Uniform Continuous-Time Markov Decision Processes. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 61–76. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Cattani, S., Segala, R., Kwiatkowska, M.Z., Norman, G.: Stochastic Transition Systems for Continuous State Spaces and Non-determinism. In: Sassone, V. (ed.) FOSSACS 2005. LNCS, vol. 3441, pp. 125–139. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Kwiatkowska, M., Norman, G., Segala, R., Sproston, J.: Verifying quantitative properties of continuous probabilistic timed automata. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 123–137. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  11. 11.
    Hernndez-Lerma, O., Lassere, J.B.: Discrete-time Markov control processes: Basic optimality criteria. Appl. Math., vol. 30. Springer, Heidelberg (1996)Google Scholar
  12. 12.
    Grabiszewski, K.: Type space with disintegrability. Draft (2005)Google Scholar
  13. 13.
    Valadier, M.: Désintégration d’une mesure sur un produit. C.R. Acad. Sc. Paris 276, 33–35 (1973) Serie AGoogle Scholar
  14. 14.
    Ash, R.B., Doléans-Dade, C.A.: Probability & Measure Theory, 2nd edn. Academic Press, London (2000)MATHGoogle Scholar
  15. 15.
    Shiryaev, A.N.: Probability, 2nd edn. Springer, Heidelberg (1995)MATHGoogle Scholar
  16. 16.
    Panangaden, P.: Measure and probability for concurrency theorists. TCS: Theoretical Computer Science 253 (2001)Google Scholar
  17. 17.
    Panangaden, P.: Stochastic Techniques in Concurrency. Lecture Notes from a course given at BRICS (unpublished, 1997)Google Scholar
  18. 18.
    Billingsley, P.: Probability and Measure, 2nd edn. Wiley, New York (1986)MATHGoogle Scholar
  19. 19.
    Leao Jr., D., Fragoso, M., Ruffino, P.: Regular conditional probability, disintegration of probability and radon spaces. Proyecciones 23, 15–29 (2004)MathSciNetGoogle Scholar
  20. 20.
    Edalat, A.: Semi-pullbacks and bisimulation in categories of markov processes. Mathematical Structures in Computer Science 9, 523–543 (1999)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Nicolás Wolovick
    • 1
  • Sven Johr
    • 2
  1. 1.Fa.M.A.F.Universidad Nacional de Córdoba, Ciudad UniversitariaCórdobaArgentina
  2. 2.FR 6.2 InformatikUniversität des SaarlandesSaarbrückenGermany

Personalised recommendations