Acta Informatica

, Volume 53, Issue 4, pp 425–457 | Cite as

Synthesizing efficient systems in probabilistic environments

  • Christian von Essen
  • Barbara Jobstmann
  • David Parker
  • Rahul Varshneya
Original Article


We present a formalism, algorithms and tools to synthesise reactive systems that behave efficiently, i.e., which achieve an optimal trade-off between a given cost and reward model. Synthesis aims to automatically generate a program from a specification. Most research in this area focuses on qualitative specifications, i.e., those that define a system as either correct or incorrect. The result can be a system that is correct, but still shows undesired behaviour, e.g., because it is too slow, inefficient or resource-intensive. Quantitative synthesis aims to use additional information to guide the synthesizer towards a desired implementation. Trade-offs between costs and rewards provide a natural source of information in order to guarantee efficiency. The systems we want to synthesize are open, i.e., they react to input signals from their environment. So, we have to specify how to combine the trade-offs the system decides to make for each input. There are several possible ways, e.g., worst or best case, or average case. In this paper we focus on the average case, i.e., we focus on the expected trade-off achieved by a system. We define the problem of finding the system with the best expected behaviour according to a quantitative specification. This specification associates costs and rewards with each decision the system makes and defines a probabilistic environment that the system operates in. We analyze the feasibility of this task (i.e., prove that such systems exist and are computable) and present three algorithms to compute an optimal system for a given specification. We compare a prototypical implementation of these algorithms against each other and, based on the best-performing algorithm, develop a novel symbolic implementation and integrate it into the probabilistic model checker PRISM. We report on experiments showing that our algorithm can analyze models with several million states.


  1. 1.
    Bahar, I., Frohm, E., Gaona, C., Hachtel, G., Macii, E., Pardo, A., Somenzi, F.: Algebraic decision diagrams and their applications. Form. Methods Syst. Des. 10(2/3), 171–206 (1997)CrossRefGoogle Scholar
  2. 2.
    Baier, C., Clarke, E., Hartonas-Garmhausen, V., Kwiatkowska, M., Ryan, M.: Symbolic model checking for probabilistic processes. In: Degano, P., Gorrieri, R., Marchetti-Spaccamela, A. (eds.) Proceedings of 24th International Colloquium on Automata, Languages and Programming (ICALP’97), volume 1256 of LNCS, pp. 430–440. Springer, New York (1997)Google Scholar
  3. 3.
    Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Berlin (2008)MATHGoogle Scholar
  4. 4.
    Bloem, R., Chatterjee, K., Henzinger, T.A., Jobstmann, B.: Better quality in synthesis through quantitative objectives. In: Bouajjani, A., Maler, O. (eds.) CAV, volume 5643 of LNCS, pp. 140–156. Springer, Berlin (2009)Google Scholar
  5. 5.
    Bloem, R., Gabow, H., Somenzi, F.: An algorithm for strongly connected component analysis in n log n symbolic steps. In: Proceedings of 3rd International Conference on Formal Methods in Computer-Aided Design (FMCAD’00), pp. 37–54 (2000)Google Scholar
  6. 6.
    Bloem, R., Greimel, K., Henzinger, T. A., Jobstmann, B.: Synthesizing robust systems. In: FMCAD, pp. 85–92. IEEE (2009)Google Scholar
  7. 7.
    Brázdil, T., Brožek, V., Chatterjee, K., Forejt, V.Kučera, A.: Two views on multiple mean-payoff objectives inMarkov decision processes. In: LICS, pp. 33–42. IEEE ComputerSociety (2011)Google Scholar
  8. 8.
    Bryant, R.E.: Graph-based algorithms for boolean function manipulation. IEEE Trans. Comput. 35(8), 677–691 (1986)CrossRefMATHGoogle Scholar
  9. 9.
    Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: \(10^{20}\) states and beyond. Inf. Comput. 98(2), 142–170 (1992)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Chatterjee, K., Henzinger, M.: Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification. In: Proceedings of 22nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA’11), pp. 1318–1336 (2011)Google Scholar
  11. 11.
    Chatterjee, K., Henzinger, T.A., Jobstmann, B., Singh, R.: Measuring and synthesizing systems in probabilistic environments. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV, volume 6174 of LNCS, pp. 380–395. Springer, Berlin (2010)Google Scholar
  12. 12.
    Chatterjee, K., Majumdar, R., Henzinger, T.: Markov decision processes with multiple objectives. In: Proceedings of 23rd International Symposium on Theoretical Aspects of Computer Science (STACS’06), volume 3884 of LNCS, pp. 325–336. Springer (2006)Google Scholar
  13. 13.
    de Alfaro, L.: Formal Verification of Probabilistic Systems. PhD thesis, Stanford University (1997)Google Scholar
  14. 14.
    Derman, C.: On sequential decisions and Markov chains. Manage. Sci. 9(1), 16–24 (1962)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Etessami, K., Kwiatkowska, M., Vardi, M., Yannakakis, M.: Multi-objective model checking of Markov decision processes. In: Proceedings of 13th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’07), volume 4424 of LNCS, pp. 50–65. Springer (2007)Google Scholar
  16. 16.
    Forejt, V., Kwiatkowska, M., Norman, G., Parker, D., Qu, H.: Quantitative multi-objective verification for probabilistic systems. In: Abdulla, P.A., Leino, K.R.M. (eds.) TACAS, volume 6605 of LNCS, pp. 112–127. Springer, Berlin (2011)Google Scholar
  17. 17.
    Fujita, M., Mcgeer, P.C., Yang, J.C.Y.: Multi-terminal binary decision diagrams: an efficient data structure for matrix representation. Form. Methods Syst. Des. V 10(2/3), 149–169 (1997)CrossRefGoogle Scholar
  18. 18.
    Gimbert, H.: Pure stationary optimal strategies in Markov decision processes. In: STACS’07, pp. 200–211. Springer (2007)Google Scholar
  19. 19.
    Hachtel, G., Macii, E., Pardo, A., Somenzi, F.: Markovian analysis of large finite state machines. Comput. Aided Des. Integr. Circuits Syst. IEEE Trans. 15(12), 1479–1493 (1996)CrossRefGoogle Scholar
  20. 20.
    Haverkort, B.R.: Performance of Computer Communication Systems—A Model-Based Approach. Wiley, New York (1998)CrossRefGoogle Scholar
  21. 21.
    Isbell, J.R., Marlow, W.H.: Attrition games. Nav. Res. Logist. Q. 3, 71–94 (1956)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Kemeny, J., Snell, J., Knapp, A.: Denumerable Markov Chains, 2nd edn. Springer, New York (1976)CrossRefMATHGoogle Scholar
  23. 23.
    Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic symbolic model checking with PRISM: a hybrid approach. In: Katoen, J.-P., Stevens, P. (eds.) TACAS, volume 2280 of LNCS, pp. 52–66. Springer, Berlin (2002)Google Scholar
  24. 24.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: Proceedings of 23rd International Conference on Computer Aided Verification (CAV’11), volume 6806 of LNCS, pp. 585–591. Springer (2011)Google Scholar
  25. 25.
    Kwiatkowska, M., Norman, G., Parker. D.: The PRISM benchmark suite. In: Proceedings of 9th International Conference on Quantitative Evaluation of Systems (QEST’12), pp. 203–204. IEEE CS Press (2012)Google Scholar
  26. 26.
    Kwiatkowska, M., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Form. Methods Syst. Des. 29(1), 33–78 (2006)CrossRefMATHGoogle Scholar
  27. 27.
    Lehmann, D.J., Rabin, M. O.: On the advantages of free choice: a symmetric and fully distributed solution to the dining philosophers problem. In: POPL (1981)Google Scholar
  28. 28.
    Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems—Safety. Springer, Berlin (1995)CrossRefMATHGoogle Scholar
  29. 29.
    Norman, G., Parker, D., Kwiatkowska, M., Shukla, S.K., Gupta, R.: Using probabilistic model checking for dynamic power management. Formal Asp. Comput. 17(2), 160–176 (2005)CrossRefMATHGoogle Scholar
  30. 30.
    Norris, J.: Markov Chains. Cambridge University Press, Cambridge (2003)MATHGoogle Scholar
  31. 31.
    Parker, D.: Implementation of Symbolic Model Checking for Probabilistic Systems. PhD thesis, University of Birmingham (2002)Google Scholar
  32. 32.
    Parr, R., Russell, S.J.: Reinforcement learning with hierarchies of machines. In: Jordan, M.I., Kearns, M.J., Solla, S.A. (eds.) NIPS. The MIT Press, Cambridge (1997)Google Scholar
  33. 33.
    Pnueli, A.: The temporal logic of programs. In: FOCS, pp. 46–57. IEEE Computer Society (1977)Google Scholar
  34. 34.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, Chichester (1994)CrossRefMATHGoogle Scholar
  35. 35.
    Rabin, M.O.: N-process mutual exclusion with bounded waiting by 4 \(log_2 n\)-valued shared variable. J. Comput. Syst. Sci. 25(1), 66–75 (1982)MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Tijms, H.C.: A First Course in Stochastic Models. Wiley, Chichester (2003)CrossRefMATHGoogle Scholar
  37. 37.
    von Essen, C., Jobstmann, B.: Synthesizing systems with optimal average-case behavior for ratio objectives. In: Reich, J., Finkbeiner, B. (eds.) Proceedings International Workshop on Interactions, Games and Protocols, iWIGP 2011, Saarbrücke, Germany, 27th March 2011. EPTCS, vol. 50, pp. 17–32 (2011)Google Scholar
  38. 38.
    von Essen, C., Jobstmann, B.: Synthesizing efficient controllers. In: International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI), pp. 428–444 (2012)Google Scholar
  39. 39.
    Wimmer, R., Braitling, B., Becker, B., Hahn, E.M., Crouzen, P., Hermanns, H., Dhama, A., Theel, O.: Symblicit calculation of long-run averages for concurrent probabilistic systems. In: QEST, pp. 27–36. IEEE Computer Society (2010)Google Scholar
  40. 40.
    Wimmer, R., Derisavi, S., Hermanns, H.: Symbolic partition refinement with dynamic balancing of time and space. In: QEST, pp. 65–74. IEEE Computer Society (2008)Google Scholar
  41. 41.
    Yue, H., Bohnenkamp, H.C., Katoen, J.-P.: Analyzing energy consumption in a gossiping MAC protocol. In: Müller-Clostermann, B., Echtle, K., Rathgeb, E.P. (eds.) MMB/DFT, volume 5987 of LNCS, pp. 107–119. Springer (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Christian von Essen
    • 1
  • Barbara Jobstmann
    • 2
  • David Parker
    • 3
  • Rahul Varshneya
    • 4
  1. 1.Google ZürichZurichSwitzerland
  2. 2.École Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  3. 3.University of BirminghamBirminghamUK
  4. 4.IIT BombayMumbaiIndia

Personalised recommendations