Software & Systems Modeling

, Volume 13, Issue 4, pp 1395–1415 | Cite as

Evaluating probabilistic models with uncertain model parameters

  • Indika Meedeniya
  • Irene Moser
  • Aldeida Aleti
  • Lars Grunske
Theme Section Paper

Abstract

Probabilistic models are commonly used to evaluate quality attributes, such as reliability, availability, safety and performance of software-intensive systems. The accuracy of the evaluation results depends on a number of system properties which have to be estimated, such as environmental factors or system usage. Researchers have tackled this problem by including uncertainties in the probabilistic models and solving them analytically or with simulations. The input parameters are commonly assumed to be normally distributed. Accordingly, reporting the mean and variances of the resulting attributes is usually considered sufficient. However, many of the uncertain factors do not follow normal distributions, and analytical methods to derive objective uncertainties become impractical with increasing complexity of the probabilistic models. In this work, we introduce a simulation-based approach which uses Discrete Time Markov Chains and probabilistic model checking to accommodate a diverse set of parameter range distributions. The number of simulation runs automatically regulates to the desired significance level and reports the desired percentiles of the values which ultimately characterises a specific quality attribute of the system. We include a case study which illustrates the flexibility of this approach using the evaluation of several probabilistic properties.

Keywords

Software architecture evaluation  Parameter uncertainty Probabilistic quality models  Monte-Carlo simulation 

Notes

Acknowledgments

This original research was proudly supported by the Commonwealth of Australia, through the Cooperative Research Center for Advanced Automotive Technology (projects C4-501: Safe and Reliable Integration and Deployment Architectures for Automotive Software Systems). Furthermore, the research was supported by the Center for Mathematical and Computational Modelling \(\text{(CM)}^2\) at the University of Kaiserslautern.

References

  1. 1.
    Alur, R., Courcoubetis, C., Dill, D.: Model-checking in dense real-time. Inf. Comput. 104(1), 2–34 (1993)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Ardagna, D., Ghezzi, C., Mirandola, R.: Rethinking the use of models in software architecture. In: Quality of Software Architectures. Models and Architectures, pp. 1–27. Springer, Berlin (2008)Google Scholar
  3. 3.
    Assayad, I., Girault, A., Kalla, H.: A bi-criteria scheduling heuristic for distributed embedded systems under reliability and real-time constraints. In: Dependable Systems and Networks, pp. 347–356. IEEE (2004)Google Scholar
  4. 4.
    Axelsson, J.: Cost models with explicit uncertainties for electronic architecture trade-off and risk analysis. In: Current Practice (2006)Google Scholar
  5. 5.
    Aziz, A., Singhal, V., Balarin, F.: It usually works: The temporal logic of stochastic systems. In: Wolper, P. (ed.) Proceedings of 7th International Conference on Computer Aided Verification, CAV 95. LNCS, vol 939, pp. 155–165. Springer, Berlin (1995)Google Scholar
  6. 6.
    Baier, C., Katoen, J.-P., Hermanns, H.: Approximate symbolic model checking of continuous-time markov chains. In: Baeten, J.C.M., Mauw, S. (eds.) Proceedings of 10th International Conference on Concurrency Theory, CONCUR 99. LNCS, vol. 1664, pp. 146–161. Springer, Berlin (1999)Google Scholar
  7. 7.
    Balsamo, S., Di Marco, A., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: a survey. IEEE Trans. Softw. Eng. 30(5), 295–310 (2004)CrossRefGoogle Scholar
  8. 8.
    Basseur, M., Zitzler, E.: A preliminary study on handling uncertainty in indicator-based multiobjective optimization. In: Appl. of Evol. Computing, pp. 727–739. Springer, Berlin (2006)Google Scholar
  9. 9.
    Becker, S., Grunske, L., Mirandola, R., Overhage, S.: Performance prediction of component-based systems—a survey from an engineering perspective. In: Architecting Systems with Trustworthy Components. LNCS, vol. 3938, pp. 169–192. Springer, Berlin (2006)Google Scholar
  10. 10.
    Beyer, H., Sendhoff, B.: Robust optimization: a comprehensive survey. Comput. Methods Appl. Mech. Eng. 196(33–34), 3190–3218 (2007)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Bhunia, A., Sahoo, L., Roy, D.: Reliability stochastic optimization for a series system with interval component reliability via genetic algorithm. Appl. Math. Comput. 216(3), 929–939 (2010)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Birolini, A.: Reliability Engineering: Theory and Practice. Springer, Berlin (2010)CrossRefGoogle Scholar
  13. 13.
    Bolch, G., Greiner, S., de Meer, H., Trivedi, K.S.: Queuing Network and Markov Chains. Wiley, New York (1998)Google Scholar
  14. 14.
    Cheung, R.: A user-oriented software reliability model. IEEE Trans. Softw. Eng. 6(2), 118–125 (1980)CrossRefMATHGoogle Scholar
  15. 15.
    Ciesinski, F., Größer, M.: On probabilistic computation tree logic. In: Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P., Siegle, M. (eds.) Validation of Stochastic Systems: A Guide to Current Research. LNCS, vol. 2925, pp. 147–188. Springer, Berlin (2004)CrossRefGoogle Scholar
  16. 16.
    Coit, D., Jin, T., Wattanapongsakorn, N.: System optimization with component reliability estimation uncertainty: a multi-criteria approach. IEEE Trans. Reliab. 53(3), 369–380 (2004)CrossRefGoogle Scholar
  17. 17.
    Coit, D.W., Smith, A.E.: Genetic algorithm to maximize a lower-bound for system time-to-failure with uncertain component Weibull parameters. Comput. Ind. Eng. 41 (2002)Google Scholar
  18. 18.
    Cortellessa, V., Grassi, V.: A modeling approach to analyze the impact of error propagation on reliability of component-based systems. In: Component-Based Software Engineering, pp. 140–156. Springer, Berlin (2007)Google Scholar
  19. 19.
    Filieri, A., Ghezzi, C., Tamburrelli, G.: Run-time efficient probabilistic model checking. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21–28, 2011, pp. 341–350. ACM, New York (2011)Google Scholar
  20. 20.
    Fiondella, L., Gokhale, S.S.: Software reliability with architectural uncertainties. In: Parallel and Distributed Processing, pp. 1–5. IEEE (2008)Google Scholar
  21. 21.
    Förster, M., Trapp, M.: Fault tree analysis of software-controlled component systems based on second-order probabilities. In: International Symposium on Software Reliability Engineering, pp. 146–154. IEEE, Nov 2009Google Scholar
  22. 22.
    Frolund, S., Koistinen, J.: Quality-of-service specification in distributed object systems. Distrib. Syst. Eng. J. 5(4), 179–202 (1998)CrossRefGoogle Scholar
  23. 23.
    Goeva-Popstojanova, K., Trivedi, K.: Architecture-based approach to reliability assessment of software systems. Perform. Eval. 45(2–3), 179–204 (2001)Google Scholar
  24. 24.
    Goseva-Popstojanova, K., Hamill, M.: Architecture-based software reliability: why only a few parameters matter?. In: Computer Software and Applications Conference, 2007, vol. 1, pp. 423–430. IEEE (2007)Google Scholar
  25. 25.
    Goseva-Popstojanova, K., Hamill, M., Perugupalli, R.: Large empirical case study of architecture-based software reliability. In: International Symposium on Software Reliability Engineering, vol. 54, pp. 10–52. IEEE (2005)Google Scholar
  26. 26.
    Goseva-Popstojanova, K., Hamill, M., Wang, X.: Adequacy, accuracy, scalability, and uncertainty of architecture-based software reliability: lessons learned from large empirical case studies. In: International Symposium on Software Reliability Engineering, pp. 197–203. IEEE (2006)Google Scholar
  27. 27.
    Goseva-Popstojanova, K., Kamavaram, S.: Assessing uncertainty in reliability of component-based software systems. In: ISSRE 2003, pp. 307–320. IEEE (2003)Google Scholar
  28. 28.
    Goseva-Popstojanova, K., Kamavaram, S.: Software reliability estimation under uncertainty:generalization of the method of moments. High Assur. Syst. Eng. 2004, 209–218 (2004)Google Scholar
  29. 29.
    Grunske, L.: Specification patterns for probabilistic quality properties. In: Proceedings of the 13th International Conference on Software Engineering, ICSE ’08 61(0), 31 (2008)Google Scholar
  30. 30.
    Grunske, L., Han, J.: A comparative study into architecture-based safety evaluation methodologies using AADL’s error annex and failure propagation models. High Assurance Systems Engineering, Symposium, pp. 283–292 (2008)Google Scholar
  31. 31.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects Comput. 6(5), 512–535 (1994)CrossRefMATHGoogle Scholar
  32. 32.
    Jhumka, A., Hiller, M., Suri, N.: Component-based synthesis of dependable embedded software. Lect. Notes Comput. Sci. 2469, 111–128 (2002)CrossRefGoogle Scholar
  33. 33.
    Koymans, R.: Specifying real-time properties with metric temporal logic. Real-Time Syst. 2(4), 255–299 (1990)CrossRefGoogle Scholar
  34. 34.
    Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic symbolic model checking with PRISM: a hybrid approach. Int. J. Softw. Tools Technol. Transfer 6(2), 128–142 (2004)CrossRefGoogle Scholar
  35. 35.
    Kwiatkowska, M., Norman, G., Parker, D.: Probabilistic model checking in practice: case studies with PRISM. ACM SIGMETRICS Perform. Eval. Rev. 32(4), 16–21 (2005)CrossRefGoogle Scholar
  36. 36.
    Kwiatkowska, M., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Formal Methods Syst. Des. 29(1), 33–78 (2006)CrossRefMATHGoogle Scholar
  37. 37.
    Limbourg, P.: Multi-objective optimization of generalized reliability design problems using feature models: a concept for early design stages. Reliab. Eng. Syst. Saf. 93(6), 815–828 (2008)CrossRefGoogle Scholar
  38. 38.
    Marseguerra, M., Zio, E., Podofillini, L., Coit, D.: Optimal design of reliable network systems in presence of uncertainty. IEEE Trans. Reliab. 54(2), 243–253 (2005)CrossRefGoogle Scholar
  39. 39.
    Meedeniya, I., Buhnova, B., Aleti, A., Grunske, L.: Architecture-driven reliability and energy optimization for complex embedded systems. In: Quality of Software Architectures, QoSA 2010. LNCS, vol. 6093, pp. 52–67. Springer, Berlin (2010)Google Scholar
  40. 40.
    Meedeniya, I., Bühnova, B., Aleti, A., Grunske, L.: Reliability-driven deployment optimization for embedded systems. J. Syst. Softw. 84(5), 835–846 (2011)CrossRefGoogle Scholar
  41. 41.
    Meedeniya, I., Moser, I., Aleti, A., Grunske, L.: Architecture-based reliability evaluation under uncertainty. In: 7th International Conference on the Quality of Software Architectures, QoSA 2011 and 2nd International Symposium on Architecting Critical Systems, ISARCS 2011, pp. 85–94 (2011)Google Scholar
  42. 42.
    Montgomery, D., Runger, G.: Applied Statistics and Probability for Engineers. Wiley, India (2007)MATHGoogle Scholar
  43. 43.
    Oberkampf, W., Helton, J., Joslyn, C., Wojtkiewicz, S., Ferson, S.: Challenge problems: uncertainty in system response given uncertain parameters. Reliab. Eng. Syst. Saf. 85(1–3), 11–19 (2004)CrossRefGoogle Scholar
  44. 44.
    Puterman, M.L.: Markov Decision Processes. Wiley, New York (1994)CrossRefMATHGoogle Scholar
  45. 45.
    Roshandel, R., Banerjee, S., Cheung, L., Medvidovic, N., Golubchik, L.: Estimating software component reliability by leveraging architectural models. In: International Conference on Software Engineering, p. 853. ACM, New York (2006)Google Scholar
  46. 46.
    Roshandel, R., Medvidovic, N., Golubchik, L.: A Bayesian model for predicting reliability of software systems at the architectural level. LNCS 4880, 108–126 (2007)Google Scholar
  47. 47.
    Rubinstein, R., Kroese, D.: Simulation and the Monte Carlo Method. Wiley-Interscience, New York (2008)MATHGoogle Scholar
  48. 48.
    Sanchez, A., Carlos, S., Martorell, S., Villanueva, J.: Addressing imperfect maintenance modelling uncertainty in unavailability and cost based optimization. Reliab. Eng. Syst. Saf. 94(1), 22–32 (2009)CrossRefGoogle Scholar
  49. 49.
    Trivedi, K.: Probability& Statistics with Reliability. Queuing and Computer Science Applications. Wiley, India (2009)Google Scholar
  50. 50.
    Wang, W., Wu, Y., Chen, M.: An architecture-based software reliability model. In: Proceedings of 1999 Pacific Rim International Symposium on Dependable Computing, pp. 143–150. IEEE (2002)Google Scholar
  51. 51.
    Wattanapongskorn, N., Coit, D.W.: Fault-tolerant embedded system design and optimization considering reliability estimation uncertainty. Rel. Eng. 92, 395–407 (2007)Google Scholar
  52. 52.
    Yin, L., Smith, M., Trivedi, K.: Uncertainty analysis in reliability modeling. In: Symposium on Reliability and Maintainability, pp. 229–234 (2001)Google Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  • Indika Meedeniya
    • 1
  • Irene Moser
    • 1
  • Aldeida Aleti
    • 2
  • Lars Grunske
    • 3
  1. 1.Faculty of Information and Communication TechnologiesSwinburne University of TechnologyHawthornAustralia
  2. 2.Faculty of Information TechnologyMonash UniversityClaytonAustralia
  3. 3.Faculty of Computer Science and Center for Mathematical and Computational Modelling (CM)University of KaiserslauternKaiserslauternGermany

Personalised recommendations