Model-based testing of apps in real network scenarios

  • Laura PanizoEmail author
  • Almudena Díaz
  • Bruno García
SPIN 2018


Traditional testing methods for mobile apps focus on detecting execution errors. However, the evolution of mobile networks toward 5G will require additional support for app developers to also ensure good performance and user experience. Manual testing in a number of scenarios is not enough to satisfy the expectations of the apps’ end users. This paper presents the testing framework developed in the TRIANGLE project (, which integrates a complete mobile network testbed to test, benchmark and certify mobile apps. In this paper, we focus on a recent extension of the TRIANGLE framework that uses model-based testing based on model checking to support the automatic generation of user interactions. We introduce the complete testing framework and the basis of the model-based extension. Finally, we use the testing framework to evaluate the performance of the ExoPlayer app in different network scenarios. ExoPlayer is a video streaming app for Android that implements different adaptive streaming protocols.


Model-based testing Mobile network testbed Model checking 



  1. 1.
    3GPP: TR37.901:User Equipment (UE) application layer data throughput (Rel. 15). Technical report (2018)Google Scholar
  2. 2.
    5GENESIS project consortium: Deliverable D2.3: Initial planning of tests and experimentation. Pu (2018)Google Scholar
  3. 3.
    Amalfitano, D., Amatucci, N., Memon, A.M., Tramontana, P., Fasolino, A.R.: A general framework for comparing automatic testing techniques of Android mobile apps. J. Syst. Softw. 125, 322–343 (2017). CrossRefGoogle Scholar
  4. 4.
    Amalfitano, D., Fasolino, A.R., Tramontana, P., Ta, B.D., Memon, A.M.: MobiGUITAR: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2015). CrossRefGoogle Scholar
  5. 5.
    Baek, Y.M., Bae, D.H.: Automated model-based android GUI testing using multi-level GUI comparison criteria. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering, ASE 2016, pp. 238–249. ACM (2016).
  6. 6.
    Broy, M., Jonsson, B., Katoen, J.P., Leucker, M., Pretschner, A.: Model-Based Testing of Reactive Systems: Advanced Lectures. Springer, Berlin (2005)CrossRefzbMATHGoogle Scholar
  7. 7.
    Cattoni, A.F., Corrales-Madueño, G., Dieudonne, M., Merino, P., Díaz-Zayas, A., Salmerón, A., Carlier, F., Saint-Germain, B., Morris, D., Figueiredo, R., Caffrey, J., Baos, J., Cárdenas, C., Roche, N., Moore, A.: An end-to-end testing ecosystem for 5G. In: European conference on networks and communications (EuCNC 2016), pp. 307–312 (2016).
  8. 8.
    Cellular Telecommunications Industry Association (CTIA): Battery Life Test Plan v.1.2. Technical report (2018).
  9. 9.
    Díaz-Zayas, A., Panizo, L., Baños, J., Cárdenas, C., Dieudonne, M.: QoE Evaluation: The TRIANGLE testbed approach. Wirel. Commun. Mob. Comput. p. 12 (2018).
  10. 10.
    Espada, A.R., Gallardo, M.M., Salmerón, A., Merino, P.: Performance analysis of spotify® for android with model based testing. Mob. Inf. Syst. 2017, 14 (2017). Google Scholar
  11. 11.
    Global System for Mobile Communications Association (GSMA): Smartphone performance test case guideline v3.0. Technical report (2017).
  12. 12.
    Holzmann, G.: The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Professional, Boston (2003)Google Scholar
  13. 13.
    ISO/IEC: Information technology Dynamic adaptive streaming over HTTP (DASH). (2014)
  14. 14.
    Koumaras, H., Tsolkas, D., Gardikis, G., Merino-Gómez, P., Frascolla, V., Triantafyllopoulou, D., Emmelmann, M., Koumaras, V., Garcia-Osma, M.L., Munaretto, D., Atxutegi, E., de Puga, J.S., Alay, O., Brunstrom, A., Bosneag, A.M.C.: 5GENESIS: The genesis of a flexible 5G Facility. In: IEEE international workshop on computer-aided modeling analysis and design of communication links and networks (CAMAD-2018 ) (2018)Google Scholar
  15. 15.
    Kozamernik, F., Steinmann, V., Sunna, P., Wyckens, E.: SAMVIQA new EBU methodology for video quality evaluations in multimedia. SMPTE Motion Imaging J. 114(4), 152–160 (2005). CrossRefGoogle Scholar
  16. 16.
    Liu, Y., Xu, C., Cheung, S.: Characterizing and detecting performance bugs for smartphone applications. In: Proceedings of the 36th international conference on software engineering, ICSE 2014, pp. 1013–1024. ACM (2014).
  17. 17.
    Massol, V., Husted, T.: JUnit in Action. Manning Publications Co., Greenwich (2003)Google Scholar
  18. 18.
    Mehmood, M.A., Wundsam, A., Uhlig, S., Levin, D., Sarrar, N., Feldmann, A.: QoE-Lab: towards evaluating quality of experience for future internet conditions. In: Korakis, T., Li, H., Tran-Gia, P., Park, H.S. (eds.) Testbeds and Research Infrastructure. Development of Networks and Communities, pp. 286–301. Springer, Berlin (2012)CrossRefGoogle Scholar
  19. 19.
  20. 20.
    NeoLoad by Neotys: The load testing platform accelerating DevOps. Last accessed: 10/26/2018
  21. 21.
    NGMN Alliance: Definition of the testing framework for the NGMN 5G pre-commecial networks trails v1.0. Technical report (2018).
  22. 22.
    Panizo, L., Díaz-Zayas, A., García, B.: An extension of TRIANGLE testbed with model-based testing. In: M.d.M. Gallardo, P. Merino (eds.) Proceedings of the 25th international symposium on model checking software (SPIN2018), pp. 190–195. Springer International Publishing (2018).
  23. 23.
    Panizo, L., Salmerón, A., Gallardo, M.M., Merino, P.: Guided Test Case Generation for Mobile Apps in the TRIANGLE Project: Work in Progress. In: Proc. of the 24th International SPIN Symposium on Model Checking of Software, pp. 192–195. ACM (2017).
  24. 24.
    Pantos, R., May, W.: HTTP Live Streaming. RFC 8216 (2017).
  25. 25.
    Perfecto: The cloud-based platform for continuous testing in a DevOps environment. Last accessed: 10/26/2018
  26. 26.
    Solera, M., Toril, M., Palomo, I., Gomez, G., Poncela, J.: A testbed for evaluating video streaming services in LTE. Wirel. Pers. Commun. 98(3), 2753–2773 (2018). CrossRefGoogle Scholar
  27. 27.
    TRIANGLE project consortium: deliverable D2.1: initial report on the testing scenarios, requirements and use cases. Public (2016)Google Scholar
  28. 28.
    TRIANGLE project consortium: Deliverable D2.6: Final test scenario and test specifications. Public (2018)Google Scholar
  29. 29.
    Yang, S., Yan, D., Rountev, A.: Testing for poor responsiveness in android applications. In: Proceedings of the 1st international workshop on the engineering of mobile-enabled systems (MOBS), pp. 1–6 (2013).
  30. 30.
    Yang, W., Prasad, M.R., Xie, T.: A grey-box approach for automated GUI-model generation of mobile applications. In: V. Cortellessa, D. Varró (eds.) Proceedings of the 16th international conference on fundamental approaches to software engineering (FASE 2013), pp. 250–265. Springer,Berlin (2013).
  31. 31.
    Zein, S., Salleh, N., Grundy, J.: A systematic mapping study of mobile application testing techniques. J. Syst. Softw. 117, 334–356 (2016). CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Departamento de Lenguajes y Ciencias de la ComputaciónUniversidad de Málaga, Andalucía TechMálagaSpain
  2. 2.Universidad de Málaga, Andalucía TechMálagaSpain

Personalised recommendations