Abstract
In search-based software engineering we often use popular heuristics with default configurations, which typically lead to suboptimal results, or we perform experiments to identify configurations on a trial-and-error basis, which may lead to better results for a specific problem. To obtain better results while avoiding trial-and-error experiments, a fitness landscape analysis is helpful in understanding the search problem, and making an informed decision about the heuristics. In this paper, we investigate the search problem of test suite generation for mobile applications (apps) using Sapienz whose heuristic is a default NSGA-II. We analyze the fitness landscape of Sapienz with respect to genotypic diversity and use the gained insights to adapt the heuristic of Sapienz. These adaptations result in Sapienz \(^{div}\) that aims for preserving the diversity of test suites during the search. To evaluate Sapienz \(^{div}\), we perform a head-to-head comparison with Sapienz on 76 open-source apps.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
All experiments were run on single 4.0 Ghz quad-core PC with 16 GB RAM, using 5 Android emulators (KitKat 4.4.2, API level 19) in parallel to test one app.
- 2.
We used ver. 5.207 of k9mail and not ver. 3.512 as in the 68 F-Droid apps benchmark.
- 3.
Sapienz \(^{div}\) is available at: https://github.com/thomas-vogel/sapienzdiv-ssbse19.
- 4.
For each server: 2\(\times \)Intel(R) Xeon(R) CPU E5-2620 @ 2.00 GHz, with 64 GB RAM.
- 5.
We exclude aGrep and frozenbubble as Sapienz/Sapienz \(^{div}\) cannot start these apps.
References
Albunian, N.M.: Diversity in search-based unit test suite generation. In: Menzies, T., Petke, J. (eds.) SSBSE 2017. LNCS, vol. 10452, pp. 183–189. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66299-2_17
Amalfitano, D., Fasolino, A.R., Tramontana, P., Ta, B.D., Memon, A.: Mobiguitar: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2015)
Android: Ui/application exerciser monkey (2017)
Arcuri, A., Fraser, G.: Parameter tuning or default values? an empirical investigation in search-based software engineering. Emp. Softw. Eng. 18(3), 594–623 (2013)
Bachelet, V.: Métaheuristiques Parallèles Hybrides: Application au Problème D’affectation Quadratique. Ph.D. thesis, Université Lille-I (1999)
Choudhary, S.R., Gorla, A., Orso, A.: Automated test input generation for android: are we there yet? In: Proceedings of ASE 2015, pp. 429–440. IEEE (2015)
Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)
Hao, S., Liu, B., Nath, S., Halfond, W.G., Govindan, R.: Puma: programmable ui-automation for large-scale dynamic analysis of mobile apps. In: Proceedings of MobiSys 2014, pp. 204–217. ACM (2014)
Isermann, H.: The enumeration of the set of all efficient solutions for a linear multiple objective program. Oper. Res. Q. 28(3), 711–725 (1977)
Li, M., Yao, X.: Quality evaluation of solution sets in multiobjective optimisation: a survey. ACM Comput. Surv. 52(2), 26:1–26:38 (2019)
Li, Y., Yang, Z., Guo, Y., Chen, X.: Droidbot: a lightweight ui-guided test input generator for android. In: Proceedings of ICSE 2017 Companion, pp. 23–26. IEEE (2017)
Liefooghe, A., Verel, S., Aguirre, H., Tanaka, K.: What makes an instance difficult for black-box 0–1 evolutionary multiobjective optimizers? In: Legrand, P., Corsini, M.-M., Hao, J.-K., Monmarché, N., Lutton, E., Schoenauer, M. (eds.) EA 2013. LNCS, vol. 8752, pp. 3–15. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11683-9_1
Maaranen, H., Miettinen, K., Penttinen, A.: On initial populations of a genetic algorithm for continuous optimization problems. J. Global Optim. 37(3), 405–436 (2006)
Machiry, A., Tahiliani, R., Naik, M.: Dynodroid: an input generation system for android apps. In: Proceedings of ESEC/FSE 2013. pp. 599–609. ACM (2013)
Mahmood, R., Mirzaei, N., Malek, S.: Evodroid: segmented evolutionary testing of android apps. In: Proceedings of FSE 2014. pp. 599–609. ACM (2014)
Malan, K.M., Engelbrecht, A.P.: A survey of techniques for characterising fitness landscapes and some possible ways forward. Inf. Sci. 241, 148–163 (2013)
Mao, K., Harman, M., Jia, Y.: Sapienz: multi-objective automated testing for android applications. In: Proceedings of ISSTA 2016. pp. 94–105. ACM (2016)
Mirzaei, N., Malek, S., Păsăreanu, C.S., Esfahani, N., Mahmood, R.: Testing android apps through symbolic execution. Softw. Eng. Notes 37(6), 1–5 (2012)
Moser, I., Gheorghita, M., Aleti, A.: Identifying features of fitness landscapes and relating them to problem difficulty. Evol. Comp. 25(3), 407–437 (2017)
Olorunda, O., Engelbrecht, A.P.: Measuring exploration/exploitation in particle swarms using swarm diversity. In: Proceedings of CEC 2008. pp. 1128–1134. IEEE (2008)
Panichella, A., Oliveto, R., Penta, M.D., Lucia, A.D.: Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Trans. Software Eng. 41(4), 358–383 (2015)
Paquete, L., Stützle, T.: Clusters of non-dominated solutions in multiobjective combinatorial optimization: an experimental analysis. Multiobjective Programming and Goal Programming. Lecture Notes in Economics and Mathematical Systems, vol. 618, pp. 69–77. Springer, Berlin (2009). https://doi.org/10.1007/978-3-540-85646-7_7
Pitzer, E., Affenzeller, M.: A comprehensive survey on fitness landscape analysis. Recent Advances in Intelligent Engineering. Studies in Computational Intelligence, vol. 378, pp. 161–191. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-23229-9_8
Purshouse, R.C., Fleming, P.J.: On the evolutionary optimization of many conflicting objectives. IEEE Trans. Evolut. Comp. 11(6), 770–784 (2007)
Ronald, S.: Duplicate genotypes in a genetic algorithm. In: Proceedings of ICEC 1998. pp. 793–798. IEEE (1998)
Shir, O.M., Preuss, M., Naujoks, B., Emmerich, M.: Enhancing decision space diversity in evolutionary multiobjective algorithms. In: Ehrgott, M., Fonseca, C.M., Gandibleux, X., Hao, J.-K., Sevaux, M. (eds.) EMO 2009. LNCS, vol. 5467, pp. 95–109. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-01020-0_12
Smith, T., Husbands, P., Layzell, P.J., O’Shea, M.: Fitness landscapes and evolvability. Evol. Comput. 10(1), 1–34 (2002)
Stadler, P.F.: Fitness landscapes. Biological Evolution and Statistical Physics. Lecture Notes in Physics, vol. 585, pp. 183–204. Springer, Berlin (2002). https://doi.org/10.1007/3-540-45692-9_10
Su, T., Meng, G., Chen, Y., Wu, K., et al.: Guided, stochastic model-based gui testing of android apps. In: Proceedings of ESEC/FSE 2017. pp. 245–256. ACM (2017)
Črepinšek, M., Liu, S.H., Mernik, M.: Exploration and exploitation in evolutionary algorithms: a survey. ACM Comput. Surv. 45(3), 35:1–35:33 (2013)
Wang, S., Ali, S., Yue, T., Li, Y., Liaaen, M.: A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering. In: Proceedings of ICSE 2016. pp. 631–642. ACM (2016)
Acknowledgments
This work has been developed in the FLASH project (GR 3634/6-1) funded by the German Science Foundation (DFG) and has been partially supported by the 2018 Facebook Testing and Verification research award.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Vogel, T., Tran, C., Grunske, L. (2019). Does Diversity Improve the Test Suite Generation for Mobile Applications?. In: Nejati, S., Gay, G. (eds) Search-Based Software Engineering. SSBSE 2019. Lecture Notes in Computer Science(), vol 11664. Springer, Cham. https://doi.org/10.1007/978-3-030-27455-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-27455-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-27454-2
Online ISBN: 978-3-030-27455-9
eBook Packages: Computer ScienceComputer Science (R0)