Skip to main content
SpringerLink
Log in

Evaluating testing strategies for resource related failures in mobile applications

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Mobile applications have been used for multiple purposes from entertainment to critical domains. As a consequence, the quality of mobile applications has become a crucial aspect, for instance, by promoting the use of testing as a quality assurance practice. The diversity of mobile platforms is pervaded by several resources related to communication capabilities, sensors, and user-controlled options. As a result, applications can present unexpected behaviors and resource interactions can introduce failures that manifest themselves in specific resource combinations. These failures can compromise the mobile application quality and harm the user experience. We evaluate the failure-detection capability and effectiveness of five sampling testing strategies (Random, One-Enabled, One-Disabled, Most-Enabled-Disabled, and Pairwise) in the context of resource related failures in 15 mobile applications. We focus on 14 common resources of the Android platform and analyze the resource interactions more likely to cause failures. Random had great percentages of failing test cases, followed by One-Enabled, Most-Enabled-Disabled, and Pairwise. We observe that One-Enabled and One-Disabled are the most effective testing strategies for six and four applications, respectively. Surprisingly, resource pairs have more influence on failures than other resource combinations, varying widely among the applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The data used in this study is available in a public repository at https://eulerhm.github.io/samplingapptest.

Notes

  1. support.google.com/android/answer/9083864?hl=en

  2. https://eulerhm.github.io/samplingapptest/

  3. In this work, GPS and network are related to Location, and Wi-Fi/Mobile Data

  4. https://github.com/commons-app/apps-android-commons/issues/1735

  5. https://www.traccar.org

  6. In this work, we do not deal with internal settings of applications (https://developer.android.com/guide/topics/ui/settings)

  7. https://github.com/traccar/traccar-client-android/issues/390

  8. https://github.com/AlDanial/cloc

  9. https://developer.android.com/guide/topics/manifest/uses-permission-element

  10. https://developer.android.com/guide/topics/manifest/uses-feature-element

  11. https://support.google.com/googleplay/android-developer/answer/9859673?hl=en

  12. https://developer.android.com/training/testing/ui-automator

  13. https://github.com/percula/LensCap

  14. https://github.com/wardellbagby/Sensor-Disabler

  15. https://en.wikipedia.org/wiki/Rooting_(Android)

  16. https://junit.org/junit4/javadoc/4.12/org/junit/BeforeClass.html

  17. https://developer.android.com/reference/android/location/LocationManager

  18. https://developer.android.com/reference/android/telephony/TelephonyManager

  19. https://developer.android.com/training/testing/junit-runner

  20. A record include a set of resource states and the value of the Support achieved

  21. https://developer.android.com/about/versions/pie/power

  22. Two Android APIs are mentioned in Section 3.5

  23. https://github.com/ElderDrivers/EdXposed

  24. https://github.com/wardellbagby/sensor-disabler

References

  • Abal, I., Brabrand, C., & Wasowski, A. (2014). 42 variability bugs in the Linux Kernel: A qualitative analysis. In Proceedings of the ACM/IEEE International Conference on Automated Software Engineering (ASE) (p. 421–432).

  • Agrawal, R., Imieliński, T., & Swami, A. (1993). Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data SIGMOD ’93 (p. 207–216). New York, NY, USA: Association for Computing Machinery.

  • Agrawal, R., & Srikant, R. (1994). Fast algorithms for mining association rules. In Proceedings of the International Conference on Very Large Databases (VLDB) (pp. 487–499). Citeseer volume 1215.

  • Al-Hajjaji, M., Krieter, S., Thüm, T., Lochau, M., & Saake, G. (2016). Incling: efficient product-line testing using incremental pairwise sampling. ACM SIGPLAN Notices, 52, 144–155.

    Article  Google Scholar 

  • Amalfitano, D., Amatucci, N., Memon, A. M., Tramontana, P., & Fasolino, A. R. (2017). A general framework for comparing automatic testing techniques of android mobile apps. Journal of Systems and Software, 125, 322–343.

    Article  Google Scholar 

  • Apel, S., Batory, D., Kastner, C., & Saake, G. (2013a). Feature-oriented software product Lines. Berlin/Heidelberg: Springer.

    Book  Google Scholar 

  • Apel, S., Kolesnikov, S., Siegmund, N., Kästner, C., & Garvin, B. (2013b). Exploring feature interactions in the wild: The new feature-interaction challenge. In Proceedings of the 5th International Workshop on Feature-Oriented Software Development (FOSD).

  • Barr, E., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The oracle problem in software testing: a survey. IEEE Transactions on Software Engineering, 41, 507–525.

    Article  Google Scholar 

  • Bertolino, A., Angelis, G.D., Gallego, M., García, B., Gortázar, F., Lonetti, F., & Marchetti, E. (2019). A systematic review on cloud testing. ACM Computing Surveys, 52.

  • Borges, H., & Valente, M. T. (2018). What’s in a GitHub star? understanding repository starring practices in a social coding platform. Journal of Systems and Software, 146, 112–129.

    Article  Google Scholar 

  • Bowen, T. F., Dworack, F., Chow, C., Griffeth, N., Herman, G. E., & Lin, Y. -J. (1989). The feature interaction problem in telecommunications systems. In Proceedings of the 7th International Conference on Software Engineering for Telecommunication Switching Systems (SETSS) (pp. 59–62).

  • Cai, H., & Ryder, B. (2020). A longitudinal study of application structure and behaviors in Android. IEEE Transactions on Software Engineering, 47, 2934–2955.

    Article  Google Scholar 

  • Coelho, J., Valente, M. T., Milen, L., & Silva, L. L. (2020). Is this github project maintained? Measuring the level of maintenance activity of open-source projects. Information and Software Technology, 122, 106274.

    Article  Google Scholar 

  • Cohen, M. B., Dwyer, M. B., & Shi, J. (2007). Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (p. 129–139).

  • Escobar-Velásquez, C., Linares-Vásquez, M., Bavota, G., Tufano, M., Moran, K., Di Penta, M., Vendome, C., Bernal-Cárdenas, C., & Poshyvanyk, D. (2020). Enabling mutant generation for open- and closed-source Android apps. IEEE Transactions on Software Engineering (TSE), 48, 186–208.

    Article  Google Scholar 

  • Ferreira, F., Vale, G., Diniz, J. P., & Figueiredo, E. (2021). Evaluating T-wise testing strategies in a community-wide dataset of configurable software systems. Journal of Systems and Software (JSS), (p. 110990).

  • Galindo, J. A., Turner, H., Benavides, D., & White, J. (2016). Testing variability-intensive systems using automated analysis: an application to Android. Software Quality Journal (SQJ), 24, 365–405.

    Article  Google Scholar 

  • Hornik, K., Grün, B., & Hahsler, M. (2005). arules-a computational environment for mining association rules and frequent item sets. Journal of Statistical Software, 14, 1–25.

    Google Scholar 

  • Júnior, M. C., Amalfitano, D., Garcés, L., Fasolino, A. R., Andrade, S. A., & Delamaro, M. (2022). Dynamic testing techniques of non-functional requirements in mobile apps: A systematic mapping study. ACM Computing Surveys (CSUR), 54, 1–38.

    Article  Google Scholar 

  • Kong, P., Li, L., Gao, J., Liu, K., Bissyandé, T. F., & Klein, J. (2018). Automated testing of Android apps: A systematic literature review. IEEE Transactions on Reliability, 68, 45–66.

    Article  Google Scholar 

  • Lam, W., Winter, S., Astorga, A., Stodden, V., & Marinov, D. (2020). Understanding reproducibility and characteristics of flaky tests through test reruns in Java projects. In ISSRE 2020: 31st IEEE International Conference on Software Reliability Engineering (pp. 403–413).

  • Luo, C., Goncalves, J., Velloso, E., & Kostakos, V. (2020). A survey of context simulation for testing mobile context-aware applications. ACM Computing Surveys (CSUR), 53, 1–39.

    Article  Google Scholar 

  • Lu, Y., Pan, M., Zhai, J., Zhang, T., & Li, X. (2019). Preference-wise testing for Android applications. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (pp. 268–278).

  • Marinho, E. H., Diniz, J. P., Ferreira, F., & Figueiredo, E. (2021). Evaluating sensor interaction failures in mobile applications. In International Conference on the Quality of Information and Communications Technology (QUATIC) (pp. 49–63). Springer.

  • Mateus, B. G., & Martinez, M. (2019). An empirical study on quality of Android applications written in Kotlin language. Empirical Software Engineering, 24, 3356–3393.

    Article  Google Scholar 

  • Medeiros, F., Kästner, C., Ribeiro, M., Gheyi, R., & Apel, S. (2016). A comparison of 10 sampling algorithms for configurable systems. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE) (pp. 643–654). IEEE.

  • Nie, C., & Leung, H. (2011). A survey of combinatorial testing. ACM Computing Surveys (CSUR), 43, 1–29.

    Article  MATH  Google Scholar 

  • Parry, O., Kapfhammer, G.M., Hilton, M., & McMinn, P. (2021). A survey of flaky tests. ACM Transactions on Software Engineering and Methodology (TOSEM), 31.

  • Pecorelli, F., Catolino, G., Ferrucci, F., De Lucia, A., & Palomba, F. (2022). Software testing and Android applications: a large-scale empirical study. Empirical Software Engineering, 27.

  • Silva, D. B., Eler, M. M., Durelli, V. H., & Endo, A. T. (2018). Characterizing mobile apps from a source and test code viewpoint. Information and Software Technnology, 101, 32–50.

    Article  Google Scholar 

  • Souto, S., d’Amorim, M., & Gheyi, R. (2017). Balancing soundness and efficiency for practical testing of configurable systems. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) (pp. 632–642). IEEE.

  • Sun, J., Su, T., Li, J., Dong, Z., Pu, G., Xie, T., & Su, Z. (2021). Understanding and finding system setting-related defects in Android apps. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (pp. 204–215).

  • Thüm, T., Kästner, C., Benduhn, F., Meinicke, J., Saake, G., & Leich, T. (2014). FeatureIDE: An extensible framework for feature-oriented software development. Science of Computer Programming, 79, 70–85.

    Article  Google Scholar 

  • Tramontana, P., Amalfitano, D., Amatucci, N., & Fasolino, A. R. (2019). Automated functional testing of mobile applications: a systematic mapping study. Software Quality Journal (SQJ), 27, 149–201.

    Article  Google Scholar 

  • Vilkomir, S. (2018). Multi-device coverage testing of mobile applications. Software Quality Journal (SQJ), 26, 197–215.

    Article  Google Scholar 

  • Villanes, I. K., Endo, A. T., & Dias-Neto, A. C. (2022). A multivocal literature mapping on mobile compatibility testing. International Journal of Computer Applications in Technology, 69, 173–192.

    Article  Google Scholar 

  • Wei, L., Liu, Y., Cheung, S. -C., Huang, H., Lu, X., & Liu, X. (2018). Understanding and detecting fragmentation-induced compatibility issues for Android apps. IEEE Transactions on Software Engineering (TSE), 46, 1176–1199.

    Article  Google Scholar 

  • Wohlin, C., Runeson, P., Host, M., Ohlsson, M. C., Regnell, B., & Wesslen, A. (2012). Experimentation in Software Engineering. Berlin / Heidelberg: Springer.

    Book  MATH  Google Scholar 

Download references

Funding

This work was partially supported by Brazilian funding agencies: CAPES and CNPq.

Author information

Authors and Affiliations

  1. Computer Science Department, Federal University of Minas Gerais, Av. Antônio Carlos, 6627 - Pampulha, Belo Horizonte, Minas Gerais, 31270-901, Brazil

    Euler Horta Marinho, João P. Diniz & Eduardo Figueiredo

  2. Computer and Systems Department, Federal University of Ouro Preto, Rua Trinta e Seis, 115 - Loanda, João Monlevade, Minas Gerais, 35931-008, Brazil

    Euler Horta Marinho

  3. Sobral campus, Federal University of Ceará, Rua Coronel Estanislau Frota, 563 - Centro, Sobral, Ceará, 62010-560, Brazil

    Fischer Ferreira

Authors
  1. Euler Horta Marinho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fischer Ferreira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. João P. Diniz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eduardo Figueiredo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Euler Horta Marinho, Fischer Ferreira, and João P. Diniz were responsible for the data collection. All authors were involved with data analysis as well as writing and reviewing the manuscript.

Corresponding author

Correspondence to Euler Horta Marinho.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Marinho, E.H., Ferreira, F., Diniz, J.P. et al. Evaluating testing strategies for resource related failures in mobile applications. Software Qual J 31, 1121–1147 (2023). https://doi.org/10.1007/s11219-023-09627-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-023-09627-3

Keywords

Search

Navigation