Skip to main content

Computing Minimum and Maximum Reachability Times in Probabilistic Systems

  • Conference paper
  • First Online:
CONCUR’99 Concurrency Theory (CONCUR 1999)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 1664))

Included in the following conference series:


A Markov decision process is a generalization of a Markov chain in which both probabilistic and nondeterministic choice coexist. Given a Markov decision process with costs associated with the transitions and a set of target states, the stochastic shortest path problem consists in computing the minimum expected cost of a control strategy that guarantees to reach the target. In this paper, we consider the classes of stochastic shortest path problems in which the costs are all non-negative, or all non-positive. Previously, these two classes of problems could be solved only under the assumption that the policies that minimize or maximize the expected cost also lead to the target with probability 1. This assumption does not necessarily hold for Markov decision processes that arise as model for distributed probabilistic systems. We present efficient methods for solving these two classes of problems without relying on additional assumptions. The methods are based on algorithms to transform the original problems into problems that satisfy the required assumptions. The methods lead to the efficient solution of two basic problems in the analysis of the reliability and performance of partially-specified systems: the computation of the minimum (or maximum) probability of reaching a target set, and the computation of the minimum (or maximum) expected time to reach the set.

This research was supported in part by the NSF CAREER award CCR-9501708, by the DARPA (NASA Ames) grant NAG2-1214, by the DARPA (Wright-Patterson AFB) grant F33615-98-C-3614, by the ARO MURI grant DAAH-04-96-1-0341, and by the Gigascale Silicon Research Center.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others


  1. A. Bianco and L. de Alfaro. Model checking of probabilistic and nondeterministic systems. In Found. of Software Tech. and Theor. Comp. Sci., volume 1026 of Lect. Notes in Comp. Sci., pages 499–513. Springer-Verlag, 1995.

    Google Scholar 

  2. R. E. Bellman. Dynamic Programming. Princeton University Press, 1957.

    Google Scholar 

  3. D. P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 1995. Volumes I and II.

    Google Scholar 

  4. D. P. Bertsekas and J. N. Tsitsiklis. An analysis of stochastic shortest path problems. Math. of Op. Res., 16(3):580–595, 1991.

    Article  MATH  MathSciNet  Google Scholar 

  5. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996.

    Google Scholar 

  6. C. Courcoubetis and M. Yannakakis. Markov decision processes and regular events. In Proc. 17th Int. Colloq. Aut. Lang. Prog., volume 443 of Lect. Notes in Comp. Sci., pages 336–349. Springer-Verlag, 1990.

    Google Scholar 

  7. C. Courcoubetis and M. Yannakakis. The complexity of probabilistic verification. J. ACM, 42(4):857–907, 1995.

    Article  MATH  MathSciNet  Google Scholar 

  8. L. de Alfaro. Formal Verification of Probabilistic Systems. PhD thesis, Stanford University, 1997. Technical Report STAN-CS-TR-98-1601.

    Google Scholar 

  9. L. de Alfaro. Stochastic transition systems. In CONCUR’98: Concurrency Theory. 9th Int. Conf., Lect. Notes in Comp. Sci. Springer-Verlag, 1998.

    Google Scholar 

  10. L. de Alfaro, T. A. Henzinger, and O. Kupferman. Concurrent reachability games. In Proc. 39th IEEE Symp. Found. of Comp. Sci., 1998.

    Google Scholar 

  11. C. Derman. Finite State MarkovianDecision Processes. Academic Press, 1970. cut?

    Google Scholar 

  12. J. H. Eaton and L. A. Zadeh. Optimal pursuit strategies in discrete-state probabilistic systems. J. of Basic Engineering, pages 23–29, 1962.

    Google Scholar 

  13. H. Hansson. Time and Probabilities in Formal Design of Distributed Systems. Real-Time Safety Critical Systems Series. Elsevier, 1994.

    Google Scholar 

  14. B. Jonsson and K. G. Larsen. Specification and refinement of probabilistic processes. In Proc. 6th IEEE Symp. Logic in Comp. Sci., pages 266–277, 1991.

    Google Scholar 

  15. C. Baier and M. Kwiatkowska. Model checking for a probabilistic branching time logic with fairness. Distributed Computing, vol. 11, May 1998.

    Google Scholar 

  16. M. Z. Kwiatkowska, G. Norman, D. Parker, and R. Segala. Symbolic model checking of concurrent probabilistic systems using MTBDDS and simplex. Technical Report CSR-99-1, University of Birmingham, 1999.

    Google Scholar 

  17. J. G. Kemeny, J. L. Snell, and A. W. Knapp. Denumerable Markov Chains. D. Van Nostrand Company, 1966.

    Google Scholar 

  18. K. G. Larsen and A. Skou. Bisimulation through probabilistic testing (preliminary report). In Proc. 16th ACM Symp. Princ. of Prog. Lang., pages 344–352, 1989.

    Google Scholar 

  19. A. Pnueli and L. Zuck. Probabilistic verification by tableaux. In Proc. First IEEE Symp. Logic in Comp. Sci.., pages 511–528, 1994.

    Google Scholar 

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 1999 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

de Alfaro, L. (1999). Computing Minimum and Maximum Reachability Times in Probabilistic Systems. In: Baeten, J.C.M., Mauw, S. (eds) CONCUR’99 Concurrency Theory. CONCUR 1999. Lecture Notes in Computer Science, vol 1664. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-66425-3

  • Online ISBN: 978-3-540-48320-5

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics