A Methodology for Generating Tests for Evaluating User-Centric Performance of Mobile Streaming Applications

  • Mustafa Al-tekreetiEmail author
  • Kshirasagar Naik
  • Atef Abdrabou
  • Marzia Zaman
  • Pradeep Srivastava
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 991)


Compared to other platforms, mobile apps’ quality assurance is more challenging, since their functionality is affected by the surrounding environment. In literature, a considerable volume of research has been devoted to develop frameworks that facilitate conducting performance analysis during the development life cycle. However, less attention has been given to test generation and test selection criteria for performance evaluation. In this work, a model based test generation methodology is proposed to evaluate the impact of the interaction of the environment, the wireless network, and the app configurations on the performance of a mobile streaming app and thereby on the experience of the end user. The methodology steps, inputs, and outputs are explained using an app example. The methodology assumes that the app has a network access through a WiFi access point. We evaluate the effectiveness of the methodology by comparing the time cost to design a test suite with random testing. The obtained results are very promising.


Performance Testing Software Coverage criteria 


  1. 1.
    Advanced combinatorial testing system (acts) (2016).
  2. 2.
    Abbors, F., Ahmad, T., Truscan, D., Porres, I.: Model-based performance testing in the cloud using the mbpet tool. In: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering, pp. 423–424. ACM (2013)Google Scholar
  3. 3.
    Abbors, F., Can, D.T.: Approaching performance testing from a model-based testing perspective. In: 2010 Second International Conference on Advances in System Testing and Validation Lifecycle (VALID), pp. 125–128. IEEE (2010)Google Scholar
  4. 4.
    Adan, I., Resing, J.: Queueing Theory. Department of Mathematics and Computing Science, Eindhoven University of Technology, Eindhoven (2001)Google Scholar
  5. 5.
    Al-tekreeti, M., Naik, K., Abdrabou, A., Zaman, M., Srivastava, P.: Test generation for performance evaluation of mobile multimedia streaming applications. In: Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development, vol. 1, pp. 225–236. SciTePress (2018)Google Scholar
  6. 6.
    Avritzer, A., Kondek, J., Liu, D., Weyuker, E.J.: Software performance testing based on workload characterization. In: Proceedings of the 3rd International Workshop on Software and Performance, pp. 17–24. ACM (2002)Google Scholar
  7. 7.
    Balsamo, S., Marco, A.D., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: a survey. IEEE Trans. Softw. Eng. 30(5), 295–310 (2004)CrossRefGoogle Scholar
  8. 8.
    Balsamo, S., Marzolla, M.: A simulation-based approach to software performance modeling. In: ACM SIGSOFT Software Engineering Notes, vol. 28, pp. 363–366. ACM (2003)Google Scholar
  9. 9.
    Barna, C., Litoiu, M., Ghanbari, H.: Autonomic load-testing framework. In: Proceedings of the 8th ACM International Conference on Autonomic Computing, pp. 91–100. ACM (2011)Google Scholar
  10. 10.
    Barna, C., Litoiu, M., Ghanbari, H.: Model-based performance testing (NIER track). In: Proceedings of the 33rd International Conference on Software Engineering, pp. 872–875. ACM (2011)Google Scholar
  11. 11.
    Briand, L., Nejati, S., Sabetzadeh, M., Bianculli, D.: Testing the untestable: model testing of complex software-intensive systems. In: Proceedings of the 38th International Conference on Software Engineering Companion, pp. 789–792. ACM (2016)Google Scholar
  12. 12.
    Brosig, F., Meier, P., Becker, S., et al.: Quantitative evaluation of model-driven performance analysis and simulation of component-based architectures. IEEE Trans. Softw. Eng. 41(2), 157–175 (2015)CrossRefGoogle Scholar
  13. 13.
    Canfora, G., Mercaldo, F., Visaggio, C.A., DAngelo, M., Furno, A., Manganelli, C.: A case study of automating user experience-oriented performance testing on smartphones. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 66–69. IEEE (2013)Google Scholar
  14. 14.
    Charnes, J.M.: Analyzing multivariate output. In: Proceedings of the 27th Conference on Winter Simulation, pp. 201–208. IEEE Computer Society (1995)Google Scholar
  15. 15.
    Cortellessa, V., Di Marco, A., Inverardi, P.: Model-Based Software Performance Analysis. Springer, Heidelberg (2011). Scholar
  16. 16.
    Costa, L.T., Czekster, R.M., de Oliveira, F.M., Rodrigues, E.d.M., da Silveira, M.B., Zorzo, A.F.: Generating performance test scripts and scenarios based on abstract intermediate models. In: SEKE, pp. 112–117 (2012)Google Scholar
  17. 17.
    Cox, D.R.: The analysis of non-Markovian stochastic processes by the inclusion of supplementary variables. In: Mathematical Proceedings of the Cambridge Philosophical Society, vol. 51, pp. 433–441. Cambridge University Press (1955)Google Scholar
  18. 18.
    Cox, D.R., Miller, H.D.: The Theory of Stochastic Processes. CRC Press, Boca Raton (1977)zbMATHGoogle Scholar
  19. 19.
    Da Silveira, M.B., Rodrigues, E.d.M., Zorzo, A.F., Costa, L.T., Vieira, H.V., De Oliveira, F.M.: Generation of scripts for performance testing based on UML models. In: SEKE, pp. 258–263 (2011)Google Scholar
  20. 20.
    Dantas, V.L.L., Marinho, F.G., da Costa, A.L., Andrade, R.M.: Testing requirements for mobile applications. In: 24th International Symposium on Computer and Information Sciences, ISCIS 2009, pp. 555–560. IEEE (2009)Google Scholar
  21. 21.
    Di Penta, M., Canfora, G., Esposito, G., Mazza, V., Bruno, M.: Search-based testing of service level agreements. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, pp. 1090–1097. ACM (2007)Google Scholar
  22. 22.
    Diaz, A., Merino, P., Rivas, F.J.: Mobile application profiling for connected mobile devices. IEEE Pervasive Comput. 9(1), 54–61 (2010)CrossRefGoogle Scholar
  23. 23.
    Doerner, K., Gutjahr, W.J.: Extracting test sequences from a Markov software usage model by ACO. In: Cantú-Paz, E., et al. (eds.) GECCO 2003. LNCS, vol. 2724, pp. 2465–2476. Springer, Heidelberg (2003). Scholar
  24. 24.
    Fiedler, M., Hossfeld, T., Tran-Gia, P.: A generic quantitative relationship between quality of experience and quality of service. IEEE Netw. 24(2), 36–41 (2010)CrossRefGoogle Scholar
  25. 25.
    German, R.: Performance Analysis of Communication Systems with Non-Markovian Stochastic Petri Nets. Wiley, Hoboken (2000)zbMATHGoogle Scholar
  26. 26.
    Gias, A.U., Sakib, K.: An adaptive Bayesian approach for URL selection to test performance of large scale web-based systems. In: Companion Proceedings of the 36th International Conference on Software Engineering, pp. 608–609. ACM (2014)Google Scholar
  27. 27.
    Gosavi, A.: Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement. ORSIS, vol. 55. Springer, Boston (2015). Scholar
  28. 28.
    Grechanik, M., Fu, C., Xie, Q.: Automatically finding performance problems with feedback-directed learning software testing. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 156–166. IEEE (2012)Google Scholar
  29. 29.
    Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test. Verif. Reliab. 15(3), 167–199 (2005)CrossRefGoogle Scholar
  30. 30.
    Gu, Y., Ge, Y.: Search-based performance testing of applications with composite services. In: International Conference on Web Information Systems and Mining, WISM 2009, pp. 320–324. IEEE (2009)Google Scholar
  31. 31.
    Guderlei, R., Mayer, J.: Statistical metamorphic testing: testing programs with random output by means of statistical hypothesis tests and metamorphic testing. In: 7th International Conference on Quality Software (QSIC), pp. 404–409. IEEE (2007)Google Scholar
  32. 32.
    Guderlei, R., Mayer, J., Schneckenburger, C., Fleischer, F.: Testing randomized software by means of statistical hypothesis tests. In: Fourth International Workshop on Software Quality Assurance: in Conjunction with the 6th ESEC/FSE Joint Meeting, pp. 46–54. ACM (2007)Google Scholar
  33. 33.
    Ivanovici, M., Beuran, R.: Correlating quality of experience and quality of service for network applications, chap. 15. In: Adibi, S. (ed.) Quality of Service Architectures for Wireless Networks: Performance Metrics and Management, pp. 326–351. IGI Global, Hershey (2010)CrossRefGoogle Scholar
  34. 34.
    Jiang, Z., Hassan, A.: A survey on load testing of large-scale software systems. IEEE Trans. Softw. Eng. 41(11), 1091–1118 (2015)CrossRefGoogle Scholar
  35. 35.
    Joorabchi, M.E., Mesbah, A., Kruchten, P.: Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 15–24. IEEE (2013)Google Scholar
  36. 36.
    Kim, Y., Choi, O., Kim, M., Baik, J., Kim, T.H.: Validating software reliability early through statistical model checking. IEEE Softw. 30(3), 35–41 (2013)CrossRefGoogle Scholar
  37. 37.
    Koziolek, H.: Performance evaluation of component-based software systems: a survey. Perform. Eval. 67(8), 634–658 (2010)CrossRefGoogle Scholar
  38. 38.
    Kumar, R., Tomkins, A., Vassilvitskii, S., Vee, E.: Inverting a steady-state. In: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pp. 359–368. ACM (2015)Google Scholar
  39. 39.
    Law, A.M.: Simulation Modeling and Analysis, 5th edn. McGraw-Hill, New York (2015)Google Scholar
  40. 40.
    Li, M., Claypool, M., Kinicki, R.: Playout buffer and rate optimization for streaming over IEEE 802.11 wireless networks. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 5(3), 26 (2009)Google Scholar
  41. 41.
    Liu, Y., Xu, C., Cheung, S.C.: Diagnosing energy efficiency and performance for mobile internetware applications. IEEE Softw. 32(1), 67–75 (2015)CrossRefGoogle Scholar
  42. 42.
    Ma, K.J., Bartos, R., Bhatia, S., Nair, R.: Mobile video delivery with HTTP. IEEE Commun. Mag. 49(4), 166–175 (2011)CrossRefGoogle Scholar
  43. 43.
    Matinnejad, R., Nejati, S., Briand, L.C., Bruckmann, T.: Automated test suite generation for time-continuous Simulink models. In: Proceedings of the 38th International Conference on Software Engineering, pp. 595–606. ACM (2016)Google Scholar
  44. 44.
    Mok, R.K., Chan, E.W., Chang, R.K.: Measuring the quality of experience of HTTP video streaming. In: 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops, pp. 485–492. IEEE (2011)Google Scholar
  45. 45.
    Nayebi, F., Desharnais, J.M., Abran, A.: The state of the art of mobile application usability evaluation. In: CCECE, pp. 1–4 (2012)Google Scholar
  46. 46.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. (CSUR) 43(2), 11 (2011)CrossRefGoogle Scholar
  47. 47.
    Prowell, S.J.: Using Markov chain usage models to test complex systems. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences, HICSS 2005, p. 318c. IEEE (2005)Google Scholar
  48. 48.
    Rios, L.M., Sahinidis, N.V.: Derivative-free optimization: a review of algorithms and comparison of software implementations. J. Glob. Optim. 56(3), 1247–1293 (2013)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Satoh, I.: Software testing for wireless mobile computing. IEEE Wirel. Commun. 11(5), 58–64 (2004)CrossRefGoogle Scholar
  50. 50.
    Sebih, N., Weitl, F., Artho, C., Hagiya, M., Tanabe, Y., Yamamoto, M.: Software model checking of UDP-based distributed applications. In: 2014 Second International Symposium on Computing and Networking, pp. 96–105. IEEE (2014)Google Scholar
  51. 51.
    Siavashi, F., Truscan, D.: Environment modeling in model-based testing: concepts, prospects and research challenges: a systematic literature review. In: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, p. 30. ACM (2015)Google Scholar
  52. 52.
    Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22(5), 297–312 (2012)CrossRefGoogle Scholar
  53. 53.
    Walls, R.J., Brun, Y., Liberatore, M., Levine, B.N.: Discovering specification violations in networked software systems. In: 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp. 496–506. IEEE (2015)Google Scholar
  54. 54.
    Walton, G.H., Poore, J.H.: Generating transition probabilities to support model-based software testing. Softw.: Pract. Exp 30(10), 1095–1106 (2000)Google Scholar
  55. 55.
    Weyuker, E.J., Vokolos, F.I.: Experience with performance testing of software systems: issues, an approach, and case study. IEEE Trans. Softw. Eng. 12, 1147–1156 (2000)CrossRefGoogle Scholar
  56. 56.
    Xu, Q., Mehrotra, S., Mao, Z., Li, J.: PROTEUS: network performance forecast for real-time, interactive mobile applications. In: Proceedings of the 11th Annual International Conference on Mobile Systems, Applications, and Services, pp. 347–360. ACM (2013)Google Scholar
  57. 57.
    Yang, C.S.D., Pollock, L.L.: Towards a structural load testing tool. In: ACM SIGSOFT Software Engineering Notes, vol. 21, pp. 201–208. ACM (1996)Google Scholar
  58. 58.
    Yılmaz, C., Fouche, S., Cohen, M.B., Porter, A., Demiröz, G., Koç, U.: Moving forward with combinatorial interaction testing. Computer 47(2), 37–45 (2014)CrossRefGoogle Scholar
  59. 59.
    Zhang, J., Cheung, S.C.: Automated test case generation for the stress testing of multimedia systems. Softw.: Pract. Exp. 32(15), 1411–1435 (2002)zbMATHGoogle Scholar
  60. 60.
    Zhang, J., Cheung, S.C., Chanson, S.T.: Stress testing of distributed multimedia software systems. In: Proceedings of the IFIP TC6 WG6.1 Joint International Conference on Formal Description Techniques for Distributed Systems and Communication Protocols and Protocol Specification, Testing and Verification, pp. 119–133. Kluwer, BV (1999)Google Scholar
  61. 61.
    Zhang, P., Elbaum, S., Dwyer, M.B.: Automatic generation of load tests. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, pp. 43–52. IEEE Computer Society (2011)Google Scholar
  62. 62.
    Zhang, P., Elbaum, S., Dwyer, M.B.: Compositional load test generation for software pipelines. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 89–99. ACM (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Mustafa Al-tekreeti
    • 1
    Email author
  • Kshirasagar Naik
    • 1
  • Atef Abdrabou
    • 2
  • Marzia Zaman
    • 3
  • Pradeep Srivastava
    • 3
  1. 1.University of WaterlooWaterlooCanada
  2. 2.UAE UniversityAl-AinUAE
  3. 3.Technologie SanstreamGatineauCanada

Personalised recommendations