A Characterization of Meaningful Schedulers for Continuous-Time Markov Decision Processes

  • Nicolás Wolovick
  • Sven Johr
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4202)


Continuous-time Markov decision process are an important variant of labelled transition systems having nondeterminism through labels and stochasticity through exponential fire-time distributions. Nondeterministic choices are resolved using the notion of a scheduler. In this paper we characterize the class of measurable schedulers, which is the most general one, and show how a measurable scheduler induces a unique probability measure on the sigma-algebra of infinite paths. We then give evidence that for particular reachability properties it is sufficient to consider a subset of measurable schedulers. Having analyzed schedulers and their induced probability measures we finally show that each probability measure on the sigma-algebra of infinite paths is indeed induced by a measurable scheduler which proves that this class is complete.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. II. Athena Scientific (1995)Google Scholar
  2. 2.
    Feinberg, E.A.: Continuous Time Discounted Jump Markov Decision Processes: A Discrete-Event Approach. Mathematics of Operations Research 29, 492–524 (2004)MATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Chichester (1994)MATHGoogle Scholar
  4. 4.
    Sennot, L.: Stochastic Dynamic Programming and the Control of Queueing Systems. John Wiley & Sons, Chichester (1999)Google Scholar
  5. 5.
    Abdeddaïm, Y., Asarin, E., Maler, O.: On Optimal Scheduling under Uncertainty. In: Garavel, H., Hatcliff, J. (eds.) ETAPS 2003 and TACAS 2003. LNCS, vol. 2619, pp. 240–253. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  6. 6.
    Bruno, J., Downey, P., Frederickson, G.N.: Sequencing Tasks with Exponential Service Times to Minimize the Expected Flow Time or Makespan. J. ACM 28, 100–113 (1981)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Qiu, Q., Pedram, M.: Dynamic power managment based on continuous-time Markov decision processes. In: Proceedings DAC, pp. 555–561 (1999)Google Scholar
  8. 8.
    Baier, C., Haverkort, B., Hermanns, H., Katoen, J.P.: Efficient Computation of Time-Bounded Reachability Probabilities in Uniform Continuous-Time Markov Decision Processes. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 61–76. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Cattani, S., Segala, R., Kwiatkowska, M.Z., Norman, G.: Stochastic Transition Systems for Continuous State Spaces and Non-determinism. In: Sassone, V. (ed.) FOSSACS 2005. LNCS, vol. 3441, pp. 125–139. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Kwiatkowska, M., Norman, G., Segala, R., Sproston, J.: Verifying quantitative properties of continuous probabilistic timed automata. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 123–137. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  11. 11.
    Hernndez-Lerma, O., Lassere, J.B.: Discrete-time Markov control processes: Basic optimality criteria. Appl. Math., vol. 30. Springer, Heidelberg (1996)Google Scholar
  12. 12.
    Grabiszewski, K.: Type space with disintegrability. Draft (2005)Google Scholar
  13. 13.
    Valadier, M.: Désintégration d’une mesure sur un produit. C.R. Acad. Sc. Paris 276, 33–35 (1973) Serie AGoogle Scholar
  14. 14.
    Ash, R.B., Doléans-Dade, C.A.: Probability & Measure Theory, 2nd edn. Academic Press, London (2000)MATHGoogle Scholar
  15. 15.
    Shiryaev, A.N.: Probability, 2nd edn. Springer, Heidelberg (1995)MATHGoogle Scholar
  16. 16.
    Panangaden, P.: Measure and probability for concurrency theorists. TCS: Theoretical Computer Science 253 (2001)Google Scholar
  17. 17.
    Panangaden, P.: Stochastic Techniques in Concurrency. Lecture Notes from a course given at BRICS (unpublished, 1997)Google Scholar
  18. 18.
    Billingsley, P.: Probability and Measure, 2nd edn. Wiley, New York (1986)MATHGoogle Scholar
  19. 19.
    Leao Jr., D., Fragoso, M., Ruffino, P.: Regular conditional probability, disintegration of probability and radon spaces. Proyecciones 23, 15–29 (2004)MathSciNetGoogle Scholar
  20. 20.
    Edalat, A.: Semi-pullbacks and bisimulation in categories of markov processes. Mathematical Structures in Computer Science 9, 523–543 (1999)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Nicolás Wolovick
    • 1
  • Sven Johr
    • 2
  1. 1.Fa.M.A.F.Universidad Nacional de Córdoba, Ciudad UniversitariaCórdobaArgentina
  2. 2.FR 6.2 InformatikUniversität des SaarlandesSaarbrückenGermany

Personalised recommendations