Skip to main content
SpringerLink
Log in

To the attention of mobile software developers: guess what, test your app!

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software testing is an important phase in the software development lifecycle because it helps in identifying bugs in a software system before it is shipped into the hand of its end users. There are numerous studies on how developers test general-purpose software applications. The idiosyncrasies of mobile software applications, however, set mobile apps apart from general-purpose systems (e.g., desktop, stand-alone applications, web services). This paper investigates working habits and challenges of mobile software developers with respect to testing. A key finding of our exhaustive study, using 1000 Android apps, demonstrates that mobile apps are still tested in a very ad hoc way, if tested at all. However, we show that, as in other types of software, testing increases the quality of apps (demonstrated in user ratings and number of code issues). Furthermore, we find evidence that tests are essential when it comes to engaging the community to contribute to mobile open source software. We discuss reasons and potential directions to address our findings. Yet another relevant finding of our study is that Continuous Integration and Continuous Deployment (CI/CD) pipelines are rare in the mobile apps world (only 26% of the apps are developed in projects employing CI/CD) – we argue that one of the main reasons is due to the lack of exhaustive and automatic testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. Gartner’s study on smartphone sales: https://goo.gl/w757Vh (Visited on April 10, 2019).

  2. The first commercially available smartphone running Android was the HTC Dream, also known as T-Mobile G1, announced on September 23, 2008: https://goo.gl/QPBdw9

  3. Google’s market for Android apps.

  4. Number of available applications in the Google Play Store from December 2009 to December 2017 available at https://goo.gl/8P1KD7.

  5. F-droid’s website: https://goo.gl/NPUusK (Visited on April 10, 2019).

  6. Sonar’s website: https://goo.gl/svp88G (Visited on April 10, 2019).

  7. Source code repository of the tool created to inspect automated testing technologies in Android projects: https://github.com/luiscruz/android_test_inspector

  8. List of Android tools curated by Furiya: https://goo.gl/yLrWgW (Visited on April 10, 2019).

  9. As reported in the F-droid’s wiki page Repository Maintenance: https://goo.gl/VfEQMg (Visited on January 26, 2018).

  10. Getting Started with Testing Android guide available at: https://goo.gl/RxmHq2 (Visited on April 10, 2019).

  11. Questionnaire responses are available online: https://goo.gl/6CFDb9

  12. Five best practices in open source: external engagement by Ben Balter: https://goo.gl/BQRZBa(Visited on April 10, 2019).

  13. Example of a source code file incompatible with Sonar tool: https://git.io/fxNg9 (Visited on April 10, 2019).

  14. The whole set of apps in the Hall of Fame can be accessed online: https://luiscruz.github.io/android_test_inspector/.

References

  • Amalfitano D, Amatucci N, Memon AM, Tramontana P, Fasolino AR (2017) A general framework for comparing automatic testing techniques of android mobile apps. J Syst Softw 125:322–343

    Article  Google Scholar 

  • Bavani R (2012) Distributed agile, agile testing, and technical debt. IEEE Softw 29(6):28–33

    Article  Google Scholar 

  • Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, Reading

    Google Scholar 

  • Bird C, Rigby PC, Barr ET, Hamilton DJ, German DM, Devanbu P (2009) The promises and perils of mining git. In: 2009 MSR’09. 6th IEEE International Working Conference on Mining Software Repositories. IEEE, pp 1–10

  • Brooks ME, Dalal DK, Nolan KP (2014) Are common language effect sizes easier to understand than traditional effect sizes? J Appl Psychol 99(2):332

    Article  Google Scholar 

  • Choudhary SR, Gorla A, Orso A (2015) Automated test input generation for Android: Are we there yet? (E). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pp 429–440

  • Coppola R (2017) Fragility and evolution of android test suites. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp 405–408

  • Coppola R, Morisio M, Torchiano M (2017) Scripted gui testing of android apps: A study on diffusion, evolution and fragility. In: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, pp 22–32

  • Corral L, Fronza I (2015) Better code for better apps: a study on source code quality and market success of android applications. In: Proceedings of the Second ACM International Conference on Mobile Software Engineering and Systems. IEEE Press, pp 22–32

  • Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from github: methods, datasets and limitations. In: 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, pp 137–141

  • Cruz L, Abreu R (2018) Measuring the energy footprint of mobile testing frameworks. In: 2018 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-c). IEEE

  • Das T, Di Penta M, Malavolta I (2016) A quantitative and qualitative investigation of performance-related commits in android apps. In: 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 443–447

  • de Langhe B, Fernbach PM, Lichtenstein DR (2016) Navigating by the stars: Investigating the actual and perceived validity of online user ratings. J Consum Res 42 (6):817–833

    Article  Google Scholar 

  • Gao J, Bai X, Tsai W-T, Uehara T (2014) Mobile application testing: a tutorial. Computer 47(2):46–55

    Article  Google Scholar 

  • Gao Z, Chen Z, Zou Y, Memon AM (2016) Sitar: Gui test script repair. IEEE Trans Softw Eng 42(2):170–186

    Article  Google Scholar 

  • Geiger F-X, Malavolta I, Pascarella L, Palomba F, Di Nucci D, Bacchelli A (2018) A graph-based dataset of commit history of real-world android apps. In: Proceedings of the 15th International Conference on Mining Software Repositories, MSR. ACM, New York

  • Gousios G, Zaidman A, Storey M-A, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, pp 358–368

  • Gousios G, Storey M-A, Bacchelli A (2016) Work practices and challenges in pull-based development: The contributor’s perspective. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, pp 285–296

  • Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pp 426–437

  • Hu C, Neamtiu I (2011) Automating GUI testing for android applications. In: Proceedings of the 6th International Workshop on Automation of Software Test. ACM, pp 77–83

  • Joorabchi ME, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, pp 15–24

  • Kaasila J, Ferreira D, Kostakos V, Ojala T (2012) Testdroid: Automated Remote UI Testing on Android. In: Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, ser. MUM ’12. ACM, New York, pp. 28:1–28:4

  • Karvonen T, Behutiye W, Oivo M, Kuvaja P (2017) Systematic literature review on the impacts of agile release engineering practices. Information and Software Technology

  • Kerby DS (2014) The simple difference formula: An approach to teaching nonparametric correlation, vol 3. [Online]. Available: https://doi.org/10.2466/11.IT.3.1

  • Khalid H, Nagappan M, Shihab E, Hassan AE (2014) Prioritizing the devices to test your app on: A case study of android game apps. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering

  • Kochhar PS (2013) An empirical study of adoption of software testing in open source projects. In: 2013 13th International Conference on Quality Software (QSIC). IEEE, pp 103–112

  • Kochhar PS, Bissyandé TF, Lo D, Jiang L (2013) Adoption of software testing in open source projects — A preliminary study on 50,000 projects. In: 2013 17th European Conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 353–356

  • Kochhar PS, Thung F, Nagappan N, Zimmermann T, Lo D (2015) Understanding the test automation culture of app developers. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 1–10

  • Krutz DE, Mirakhorli M, Malachowsky SA, Ruiz A, Peterson J, Filipski A, Smith J (2015) A dataset of open-source android applications. In: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories (MSR). IEEE, pp 522–525

  • Leech NL, Onwuegbuzie AJ (2002) A call for greater use of nonparametric statistics

  • Li X, Chang N, Wang Y, Huang H, Pei Y, Wang L, Li X (2017) ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 161–171

  • Linares-Vásquez M, Bernal-Cárdenas C, Moran K, Poshyvanyk D (2017a) How do developers test android applications?” In: 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME’17) page to appear

  • Linares-Vásquez M, Moran K, Poshyvanyk D (2017b) Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In: 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME’17) page to appear

  • Maji AK, Hao K, Sultana S, Bagchi S (2010) Characterizing failures in mobile oses: A case study with android and symbian. In: 2010 IEEE 21st International Symposium on Software Reliability Engineering (ISSRE). IEEE, pp 249–258

  • Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng 43(9):817–847

    Article  Google Scholar 

  • McGraw KO, Wong S (1992) A common language effect size statistic. Psychol Bullet 111(2):361

    Article  Google Scholar 

  • Moran K, Linares-Vásquez M, Poshyvanyk D (2017) Automated GUI testing of Android apps: from research to practice. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp 505–506

  • Muccini H, Di Francesco A, Esposito P (2012) Software testing of mobile applications: Challenges and future research directions. In: Proceedings of the 7th International Workshop on Automation of Software Test. IEEE Press, pp 29–35

  • Nagappan M, Shihab E (2016) Future trends in software engineering research for mobile apps. In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, vol 5, pp 21–32

  • Nayebi M, Adams B, Ruhe G (2016) Release practices for mobile apps–what do users and developers think?” In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, vol 1, pp 552–562

  • Nayebi M, Cho H, Ruhe G (2018) App store mining is not enough for app improvement. Empir Softw Eng 23(5):1–31

    Article  Google Scholar 

  • Pascarella L, Geiger F-X, Palomba F, Di Nucci D, Malavolta I, Bacchelli A (2018) Self-reported activities of android developers. In: 5th IEEE/ACM International Conference on Mobile Software Engineering and Systems, New York

  • Picco GP, Julien C, Murphy AL, Musolesi M, Roman G-C (2014) Software engineering for mobility: reflecting on the past, peering into the future In: Proceedings of the on Future of Software Engineering. ACM, pp 13–28

  • Ruiz IM, Nagappan M, Adams B, Berger T, Dienst S, Hassan A (2017) An examination of the current rating system used in mobile app stores. IEEE Software

  • Silva DB, Endo AT, Eler MM, Durelli VH (2016) An analysis of automated tests for mobile android applications. In: 2016 XLII Latin American Computing Conference (CLEI) . IEEE, pp 1–9

  • Stolberg S (2009) Enabling agile testing through continuous integration. In: 2009 AGILE’09 Agile Conference. IEEE, pp 369–374

  • Van Deursen A, risks Program comprehension (2001) Opportunities in extreme programming. In: Eighth Working Conference on Reverse Engineering Proceedings. IEEE, pp 176–185

  • Visser J, Rigal S, van der Leek R, van Eck P, Wijnholds G (2016) Building maintainable software, java edition: Ten Guidelines for Future-Proof Code. ”O’Reilly Media Inc.”

  • Wang Y, Alshboul Y (2015) Mobile security testing approaches and challenges. In: 2015 First Conference on Mobile and Secure Services (MOBISECSERV). IEEE, pp 1–5

  • Zaeem RN, Prasad MR, Khurshid S (2014) Automated generation of oracles for testing user-interaction features of mobile apps. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 183–192 0

  • Zeng X, Li D, Zheng W, Xia F, Deng Y, Lam W, Yang W, Xie T (2016) Automated test input generation for android: Are we really there yet in an industrial case?” In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 987–992

  • Zhao Y, Serebrenik A, Zhou Y, Filkov V, Vasilescu B (2017) The impact of continuous integration on other software development practices: A large-scale empirical study 32nd IEEE/ACM International Conference on Automated Software Engineering

Download references

Acknowledgments

This work is financed by the ERDF — European Regional Development Fund through the Operational Program for Competitiveness and Internationalization - COMPETE 2020 Program and by National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia with reference UID/CEC/50021/2019, and within projects GreenLab (POCI-01-0145-FEDER-016718) and FaultLockerRef (PTDC/CCI-COM/29300/2017). Luis Cruz is sponsored by an FCT scholarship grant number PD/BD/52237/2013.

Author information

Authors and Affiliations

  1. INESC ID, Lisbon, Portugal

    Luis Cruz

  2. INESC ID and IST, University of Lisbon, Lisbon, Portugal

    Rui Abreu

  3. School of Information Systems, Singapore Management University, Singapore, Singapore

    David Lo

Authors
  1. Luis Cruz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Abreu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Lo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luis Cruz.

Additional information

Communicated by: Andrea De Lucia

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cruz, L., Abreu, R. & Lo, D. To the attention of mobile software developers: guess what, test your app!. Empir Software Eng 24, 2438–2468 (2019). https://doi.org/10.1007/s10664-019-09701-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09701-0

Keywords

Search

Navigation