ICTSS 2017: Testing Software and Systems pp 293-310 | Cite as
Checking Response-Time Properties of Web-Service Applications Under Stochastic User Profiles
Conference paper
First Online:
Abstract
Performance evaluation of critical software is important but also computationally expensive. It usually involves sophisticated load-testing tools and demands a large amount of computing resources. Analysing different user populations requires even more effort, becoming infeasible in most realistic cases. Therefore, we propose a model-based approach. We apply model-based test-case generation to generate log-data and learn the associated distributions of response times. These distributions are added to the behavioural models on which we perform statistical model checking (SMC) in order to assess the probabilities of the required response times. Then, we apply classical hypothesis testing to evaluate if an implementation of the behavioural model conforms to these timing requirements. This is the first model-based approach for performance evaluation combining automated test-case generation, cost learning and SMC for real applications. We realised this method with a property-based testing tool, extended with SMC functionality, and evaluate it on an industrial web-service application.
Keywords
Statistical model checking Property-based testing Model-based testing FsCheck User profiles Response time Cost learningNotes
Acknowledgments
This work was funded by the Austrian Research Promotion Agency (FFG), project TRUCONF, No. 845582. We are grateful to Martin Tappler, the team at AVL, especially Elisabeth Jöbstl, and the anonymous reviewers for their valuable inputs.
References
- 1.Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016)Google Scholar
- 2.Aichernig, B.K., Schumi, R.: Towards integrating statistical model checking into property-based testing. In: MEMOCODE, pp. 71–76. IEEE (2016)Google Scholar
- 3.Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017)Google Scholar
- 4.Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994)MathSciNetCrossRefMATHGoogle Scholar
- 5.Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014)Google Scholar
- 6.Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40196-1_30 CrossRefGoogle Scholar
- 7.Becker, S., Koziolek, H., Reussner, R.H.: The palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)CrossRefGoogle Scholar
- 8.Blair, L., Jones, T., Blair, G.: Stochastically enhanced timed automata. In: Smith, S.F., Talcott, C.L. (eds.) FMOODS 2000. IAICT, vol. 49, pp. 327–347. Springer, Boston, MA (2000). doi: 10.1007/978-0-387-35520-7_17 CrossRefGoogle Scholar
- 9.Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005)Google Scholar
- 10.Bulychev, P.E., David, A., Larsen, K.G., Mikucionis, M., Poulsen, D.B., Legay, A., Wang, Z.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL. EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012). doi: 10.4204/EPTCS.85.1
- 11.Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001)Google Scholar
- 12.Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000)Google Scholar
- 13.Claessen, K., Palka, M.H., Smallbone, N., Hughes, J., Svensson, H., Arts, T., Wiger, U.T.: Finding race conditions in Erlang with QuickCheck and PULSE. In: ICFP, pp. 149–160. ACM (2009)Google Scholar
- 14.Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006)Google Scholar
- 15.Govindarajulu, Z.: Sequential Statistics. World Scientific, Singapore (2004)CrossRefMATHGoogle Scholar
- 16.Grinchtein, O.: Learning of Timed Systems. Ph.D. thesis, Uppsala Univ. (2008)Google Scholar
- 17.Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics, 2nd edn. Springer, New York (2009). doi: 10.1007/978-0-387-84858-7 CrossRefMATHGoogle Scholar
- 18.Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Statist. Assoc. 58(301), 13–30 (1963)MathSciNetCrossRefMATHGoogle Scholar
- 19.Hughes, J.: QuickCheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2006). doi: 10.1007/978-3-540-69611-7_1 CrossRefGoogle Scholar
- 20.Hughes, J., Pierce, B.C., Arts, T., Norell, U.: Mysteries of dropbox: property-based testing of a distributed synchronization service. In: ICST, pp. 135–145. IEEE (2016)Google Scholar
- 21.Jaccard, J., Turrisi, R.: Interaction Effects in Multiple Regression. SAGE, Thousand Oaks (2003)CrossRefGoogle Scholar
- 22.Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009)Google Scholar
- 23.Kwiatkowska, M., Norman, G., Segala, R., Sproston, J.: Verifying quantitative properties of continuous probabilistic timed automata. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 123–137. Springer, Heidelberg (2000). doi: 10.1007/3-540-44618-4_11 CrossRefGoogle Scholar
- 24.Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Barringer, H., Falcone, Y., Finkbeiner, B., Havelund, K., Lee, I., Pace, G., Roşu, G., Sokolsky, O., Tillmann, N. (eds.) RV 2010. LNCS, vol. 6418, pp. 122–135. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-16612-9_11 CrossRefGoogle Scholar
- 25.Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014)Google Scholar
- 26.Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012)Google Scholar
- 27.Nilsson, R.: ScalaCheck: The Definitive Guide. IT Pro, Artima Incorporated (2014)Google Scholar
- 28.Norell, U., Svensson, H., Arts, T.: Testing blocking operations with QuickCheck’s component library. In: Erlang 2013, pp. 87–92. ACM (2013)Google Scholar
- 29.Papadakis, M., Sagonas, K.: A proper integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011)Google Scholar
- 30.Rencher, A., Christensen, W.: Methods of Multivariate Analysis. Wiley, New York (2012)CrossRefMATHGoogle Scholar
- 31.Rina, T.S.: A comparative study of performance testing tools. Intern. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013)Google Scholar
- 32.Tang, J., Alelyani, S., Liu, H.: Feature selection for classification: a review. In: Data Classification: Algorithms and Applications, pp. 37–64. CRC Press (2014)Google Scholar
- 33.Verwer, S., Weerdt, M., Witteveen, C.: A likelihood-ratio test for identifying probabilistic deterministic real-time automata from positive data. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS (LNAI), vol. 6339, pp. 203–216. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15488-1_17 CrossRefGoogle Scholar
- 34.Wald, A.: Sequential Analysis. Courier Corporation, New York City (1973)MATHGoogle Scholar
- 35.West, B., Welch, K., Galecki, A.: Linear Mixed Models. CRC Press, Boca Raton (2006)MATHGoogle Scholar
Copyright information
© IFIP International Federation for Information Processing 2017