Abstract
A recent research showed that mobile apps represent nowadays 75% of the whole usage of mobile devices. This means that the mobile user experience, while tied to many factors (e.g., hardware device, connection speed, etc.), strongly depends on the quality of the apps being used. With “quality” here we do not simply refer to the features offered by the app, but also to its non-functional characteristics, such as security, reliability, and performance. This latter is particularly important considering the limited hardware resources (e.g., memory) mobile apps can exploit. In this paper, we present the largest study at date investigating performance bugs in mobile apps. In particular, we (i) define a taxonomy of the types of performance bugs affecting Android and iOS apps; and (ii) study the survivability of performance bugs (i.e., the number of days between the bug introduction and its fixing). Our findings aim to help researchers and apps developers in building performance-bugs detection tools and focusing their verification and validation activities on the most frequent types of performance bugs.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
For apps not published on the app stores we manually assigned the category by reading their description.
Note that there is also a category “performance bad practices” in which we group issues related to high-level practices, i.e., issues due to the fact that developers do not apply performance best practices proposed by the platform designers.
References
Ali M, Joorabchi ME, Mesbah A (2017) Same app, different app stores: a comparative study. In: International conference on mobile software engineering and systems (MOBILESoft), pp 79–90
Boehm BW, Brown JR, Lipow M (1976) Quantitative evaluation of software quality. In: Proceedings of the 2nd international conference on Software engineering. IEEE Computer Society Press, pp 592–605
Brebner PC (2016) Automatic performance modelling from application performance management (APM) data: an experience report. In: International conference on performance engineering (ICPE), pp 55–61
Carroll A, Heiser G (2010) An analysis of power consumption in a smartphone. In: USENIX Annual technical conference
Christensen R (2011) Plane Answers to Complex Questions: The Theory of Linear models, fouth edn. Springer Texts in Statistics. Springer
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Earlbaum Associates
Conover WJ (1998) Practical Nonparametric Statistics, 3rd edn. Wiley, New York
Cortellessa V, Marco AD, Inverardi P (2011) Model-Based Software performance analysis. Springer, Berlin
Cruz L, Abreu R (2017) Performance-based guidelines for energy efficient mobile applications. In: International conference on mobile software engineering and systems (MOBILESoft), pp 46–57
Cox DR (1972) Regression models and life tables. J R Stat Soc B(34):187–202
Cruz L, Abreu R (2019) Catalog of energy patterns for mobile applications. Empir Software Eng 24:2209–2235. https://doi.org/10.1007/s10664-019-09682-0
Cumming G (2011) Introduction to the new Statistics:Effect sizes, confidence intervals, and Meta-Analysis. Routledge, Abingdon
Di Franco A, Guo H, Rubio-González C (2017) A comprehensive study of real-world numerical bug characteristics. In: International conference on automated software engineering (ASE), pp 509–519
Fazzini M, Orso A (2017) Automated cross-platform inconsistency detection for mobile apps. In: International conference on automated software engineering (ASE), pp 308–318
Fling B (2009) Mobile design and development: Practical concepts and techniques for creating mobile sites and Web apps. O’Reilly Media Inc., Sebastopol
Franke D, Weise C (2011) Providing a software quality framework for testing of mobile applications. In: International conference on software testing, verification and validation (ICST), pp 431–434
Gao Z, Bird C, Barr ET (2017) To type or not to type: quantifying detectable bugs in javascript. In: International conference on software engineering (ICSE), pp 758–769
Gegick M, Rotella P, Xie T (2010) Identifying security bug reports via text mining: an industrial case study. In: Working conference on mining software repositories (MSR), pp 11–20
Grechanik M, Fu C, Xie Q (2012) Automatically Finding Performance Problems with Feedback-directed Learning Software Testing. In: International conference on software engineering (ICSE), pp 156–166
Gregg B (2013) Systems performance: enterprise and the cloud. Pearson Education, London
Grissom RJ, Kim JJ (2005) Effect sizes for research: A broad practical approach, 2nd edn. Lawrence Earlbaum Associates
Gui J, Mcilroy S, Nagappan M, Halfond WGJ (2015) Truth in advertising: The hidden cost of mobile ads for software developers. In: International conference on software engineering (ICSE), pp 100–110
Guo C, Zhang J, Yan J, Zhang Z, Zhang Y (2013) Characterizing and detecting resource leaks in android applications. In: International conference on automated software engineering (ASE), pp 389–398
Hao S, Li D, Halfond WGJ, Govindan R (2012) Estimating Android applications’ CPU energy usage via Bytecode profiling. In: International workshop on green and sustainable software (GREENS), pp 1–7
Hao S, Li D, Halfond WGJ, Govindan R (2013) Estimating mobile application energy consumption using program analysis. In: ICSE’13, pp 92–101
Harrell FE Jr (2017) With contributions from Charles Dupont, many others.: Hmisc: Harrell Miscellaneous. https://CRAN.R-project.org/package=Hmisc. R package version 4.0-3
Harter DE, Krishnan MS, Slaughter SA (2000) Effects of process maturity on quality, cycle time, and effort in software product development. Manag Sci 46(4):451–466
Hecht G, Moha N, Rouvoy R (2016) An empirical study of the performance impacts of android code smells. In: International conference on mobile software engineering and systems (MOBILESoft), pp 59–69
Hedges LV, Olkin I (1985) Statistical methods for Meta-Analysis. Academic Press, Cambridge
Herzig K, Zeller A (2013) The impact of tangled code changes. In: Working conference on mining software repositories (MSR), pp 121–130
Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6(2):65–70
Incerto E, Tribastone M, Trubiani C (2018) Combined vertical and horizontal autoscaling through model predictive control. In: International conference on parallel and distributed computing (euro-par), pp 147–159
Jain R (1990) The art of computer systems performance analysis: techniques for experimental design, measurement, simulation, and modeling. Wiley, New York
Jin G, Song L, Shi X, Scherpelz J, Lu S (2012) Understanding and detecting real-world performance bugs. ACM SIGPLAN Not 47(6):77–88
Joorabchi ME, Ali M, Mesbah A (2015) Detecting inconsistencies in multi-platform mobile apps. In: International symposium on software reliability engineering (ISSRE), pp 450–460
Joorabchi ME, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: International symposium on empirical software engineering and measurement (ESEM), pp 15–24
Jovic M, Adamoli A, Hauswirth M (2011) Catch me if you can: performance bug detection in the wild. In: ACM SIGPLAN Notices, vol 46, pp 155–170
Kan SH (2002) Metrics and models in software quality engineering. Addison-wesley Longman Publishing Co. Inc
Khalid H, Nagappan M, Hassan AE (2016) Examining the relationship between findbugs warnings and app ratings. IEEE Softw 33(4):34–39
Killian C, Nagaraj K, Pervez S, Braud R, Anderson JW, Jhala R (2010) Finding latent performance bugs in systems implementations. In: International symposium on foundations of software engineering (FSE), pp 17–26
Kleinbaum DG, Klein M (2005) Survival Analysis: A Self-Learning Text
Knoche H, Eichelberger H (2018) Using the raspberry pi and docker for replicable performance experiments: Experience paper. In: International conference on performance engineering (ICPE), pp 305–316
Lampson BW (2004) Computer security in the real world. Computer 37(6):37–46
Lee S, Heo M, Lee C, Kim M, Jeong G (2017) Applying deep learning based automatic bug triager to industrial projects. In: Joint meeting on foundations of software engineering (ESEC/FSE), pp 926–931
Li D, Hao S, Halfond WGJ, Govindan R (2013) Calculating source line level energy information for android applications. In: International symposium on software testing and analysis (ISSTA), pp 78–89
Li D, Hao S, Gui J, Halfond W (2014a) An empirical study of the energy consumption of Android applications. In: International conference on software maintenance and evolution (ICSME), pp 121–130
Li D, Jin Y, Sahin C, Clause J, Halfond W (2014b) Integrated energy-directed test suite optimization. In: International symposium on software testing and analysis (ISSTA), pp 339–350
Li D, Tran AH, Halfond W (2014c) Making web applications more energy efficient for OLED smartphones. In: International conference on software engineering (ICSE), pp 573–538
Li D, Tran AH, Halfond WGJ (2015a) Nyx: a display energy optimizer for mobile web apps. In: Joint meeting on foundations of software engineering (ESEC/FSE), pp 958–961
Li D, Halfond WGJ (2015b) Optimizing energy of http requests in android applications. In: International workshop on software development lifecycle for mobile (demobile), pp 25–28
Li D, Lyu Y, Gui J, Halfond W (2016) Automated energy optimization of http requests for mobile applications. In: International conference on software engineering (ICSE), pp 249–260
Lin Y, Radoi C, Dig D (2014) Retrofitting concurrency for android applications through refactoring. In: International symposium on foundations of software engineering (FSE), pp 341–352
Linares-Vȧsquez M, Bavota G, Bernal-cȧrdenas C, Oliveto R, Penta MD, Poshyvanyk D (2014) Mining energy-greedy API usage patterns in android apps: an empirical study. In: Working conference on mining software repositories (MSR), pp 2–11
Linares-Vȧsquez M, Bavota G, Bernal-cȧrdenas CE, Oliveto R, Penta MD, Poshyvanyk D (2015a) Optimizing energy consumption of guis in android apps: a multi-objective approach. In: Joint meeting on foundations of software engineering (ESEC/FSE), pp 143–154
Linares-Vásquez M, Vendome C, Luo Q, Poshyvanyk D (2015b) How developers detect and fix performance bottlenecks in android apps. In: International conference on software maintenance and evolution (ICSME), pp 352–361
Linares-Vȧsquez M, Vendome C, Tufano M, Poshyvanyk D (2017) How developers micro-optimize android apps. J Syst Softw 130:1–23
Linares-Vásquez M, Bavota G, Bernal-Cárdenas C, Penta MD, Oliveto R, Poshyvanyk D (2018) Multi-objective optimization of energy consumption of guis in android apps. ACM Trans Softw Eng Methodol 27(3):14:1–14:47
Liu Y, Xu C, Cheung SC (2014) Characterizing and detecting performance bugs for smartphone applications. In: International conference on software engineering (ICSE), pp 1013–1024
Madan BB, Gogeva-Popstojanova K, Vaidyanathan K, Trivedi KS (2002) Modeling and quantification of security attributes of software systems. In: International conference on dependable systems and networks (DSN), pp 505–514
Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng 43(9):817–847
Mazuera-Rozo A, Bautista-Mora J, Linares-Vȧsquez M, Rueda S, Bavota G (2019a) The android OS stack and its vulnerabilities: an empirical study. Empir Softw Eng 24(4):2056–2101
Mazuera-Rozo A, Trubiani C, Linares-Vásquez M, Bavota G (2019b) Replication package. https://github.com/amazuerar/perf-bugs-mobile/
Mcilroy S, Shang W, Ali N, Hassan AE (2017) User reviews of top mobile apps in apple and google app stores. Commun ACM 60(11):62–67
Mondal M, Roy CK, Schneider KA (2017) Bug propagation through code cloning: an empirical study. In: International conference on software maintenance and evolution (ICSME), pp 227–237
Nagappan M, Shihab E (2016) Future trends in software engineering research for mobile apps. In: International conference on software analysis, evolution, and reengineering (SANER), vol 5, pp 21–32
Near JP, Jackson D (2016) Finding security bugs in web applications using a catalog of access control patterns. In: International conference on software engineering (ICSE), pp 947–958
Nistor A, Jiang T, Tan L (2013) Discovering, reporting, and fixing performance bugs. In: International working conference on mining software repositories (MSR), pp 237–246
Nistor A, Chang PC, Radoi C, Lu S (2015) CARAMEL: detecting and fixing performance problems that have non-intrusive fixes. In: International conference on software engineering (ICSE), pp 902–912
Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of android apps. Empir Softw Eng 22(6):3088–3116
Oliveira W, Oliveira R, Castor F (2017) A study on the energy consumption of android app development approaches. In: International conference on mining software repositories (MSR), pp 42–52
Olivo O, Dillig I, Lin C (2015) Static detection of asymptotic performance bugs in collection traversals. In: ACM SIGPLAN Notices, vol 50, pp 369–378
Panichella S, Panichella A, Beller M, Zaidman A, Gall HC (2016) The impact of test case summaries on bug fixing performance: an empirical investigation. In: International conference on software engineering (ICSE), pp 547–558
Parsons T, Murphy J (2008) Detecting performance antipatterns in component based enterprise systems. J Object Technol 7(3):55–91
Pathak A, Hu Y, Zhang M (2011a) Bootstrapping energy debugging on smartphones: a first look at energy bugs in mobile devices. In: Workshop on hot topics in networks (hotnets), pp Article no 5
Pathak A, Hu Y, Zhang M, Bahl P, Wang YM (2011b) Fine-grained power modeling for smartphones using system call tracing. In: European conference on computer systems (eurosys), pp 153–168
Pathak A, Hu Y, Zhang M (2012a) Where is the energy spent inside my app? fine grained energy accounting on smartphones with eprof. In: European conference on computer systems (eurosys), pp 29–42
Pathak A, Jindal A, Hu Y, Midkiff SP (2012b) What is keeping my phone awake? characterizing and detecting no-sleep energy bugs in smartphone apps. In: International conference on mobile systems, applications, and services (mobisys), pp 267–280
Ramakrishnan R, Kaur A (2017) Technique for detecting early-warning signals of performance deterioration in large scale software systems. In: International conference on performance engineering (ICPE), pp 213–222
Rodríguez RJ, Trubiani C, Merseguer J (2012) Fault-tolerant techniques and security mechanisms for model-based performance prediction of critical systems. In: International Symposium on Architecting Critical Systems, ISARCS, pp 21–30
Sahin C, Wan M, Tornquist P, McKenna R, Pearson Z, Halfond WGJ, Clause J (2016) How does code obfuscation impact energy usage? J Softw Evol Process 28,:565–588. https://doi.org/10.1002/smr.1762
Schulz H, Okanovic D, van Hoorn A, Ferme V, Pautasso C (2019) Behavior-driven load testing using contextual knowledge - approach and experiences. In: International conference on performance engineering (ICPE), pp 265–272
Selakovic M, Pradel M (2016) Performance issues and optimizations in javascript: an empirical study. In: International conference on software engineering (ICSE), pp 61–72
Sliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes?. In: Proceedings of the 2005 International Workshop on Mining Software Repositories
Syer MD, Jiang ZM, Nagappan M, Hassan AE, Nasser M, Flora P (2014) Continuous validation of load test suites. In: International conference on performance engineering (ICPE), pp 259–270
Syer MD, Nagappan M, Hassan AE, Adams B (2013) Revisiting prior empirical findings for mobile apps: an empirical case study on the 15 most popular open-source android apps. In: Conference of the center for advanced studies on collaborative research, pp 283–297
Trivedi KS, Bobbio A (2017) Reliability and availability engineering - modeling, analysis, and applications. Cambridge University Press, Cambridge
Trubiani C, Bran A, van Hoorn A, Avritzer A, Knoche H (2018) Exploiting load testing and profiling for performance antipattern detection. Inf Softw Technol 95:329–345
Wan M, Jin Y, Li D, Halfond WGJ (2015) Detecting display energy hotspots in Android apps. In: International conference on software testing, verification and validation (ICST)
Wan Z, Lo D, Xia X, Cai L (2017) Bug characteristics in blockchain systems: a large-scale empirical study. In: International conference on mining software repositories (MSR), pp 413–424
Wang J, Dou W, Gao Y, Gao C, Qin F, Yin K, Wei J (2017) A comprehensive study on real world concurrency bugs in node.js. In: International conference on automated software engineering (ASE), pp 520–531
Wasserman AI (2010) Software engineering issues for mobile application development. In: International workshop on future of software engineering research (FSE/SDP), pp 397–400
Wert A, Happe J, Happe L (2013) Supporting swift reaction: Automatically uncovering performance problems by systematic experiments. In: International conference on software engineering (ICSE), pp 552–561
Williams LG, Smith CU (2003) Making the business case for software performance engineering. In: International conference on computer measurement group (CMG), pp 349–358
Woodside CM, Franks G, Petriu DC (2007) The future of software performance engineering. In: International workshop on the future of software engineering (FOSE), pp 171–187
Woodside CM (2015) Wosp-c’15: Workshop on challenges in performance methods for software development. In: International conference on performance engineering (ICPE), pp 349–350
Yang Y, Xiang P, Mantor M, Zhou H (2012) Fixing performance bugs: an empirical study of open-source gpgpu programs. In: International conference on parallel processing (ICPP), pp 329–339
Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: International conference on mining software repositories (MSR), pp 93–102
Zaman S, Adams B, Hassan AE (2012) A qualitative study on performance bugs. In: International conference of mining software repositories MSR, pp 199–208
Zhou Y, Sharma A (2017) Automated identification of security issues from commit messages and bug reports. In: Joint meeting on foundations of software engineering (ESEC/FSE), pp 914–919
Acknowledgements
Mazuera-Rozo and Bavota gratefully acknowledge the financial support of the Swiss National Science Foundation for the CCQR project (SNF Project No. 175513).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Meiyappan Nagappan
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mazuera-Rozo, A., Trubiani, C., Linares-Vásquez, M. et al. Investigating types and survivability of performance bugs in mobile apps. Empir Software Eng 25, 1644–1686 (2020). https://doi.org/10.1007/s10664-019-09795-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-019-09795-6