Advertisement

Problems and Solutions in Mobile Application Testing

  • Triin Samuel
  • Dietmar PfahlEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10027)

Abstract

In recent years the amount of literature published about mobile application testing has significantly grown. However, it is unclear to what degree stated problems and proposed solutions are relevant to industry. To shed light on this issue, we conducted a literature survey to provide an overview of what current scientific literature considers problems and potential solutions in mobile application testing, and how often proposed solutions were reportedly evaluated in industry. Then we conducted a case study involving six software companies in Estonia to find out which of the problems are considered relevant by professionals, and which of the proposed solutions are considered novel and applicable. In total, we identified 49 potential problems or challenges in the mobile application testing domain and 39 potential solutions, some of which were implemented software tools while others were just theoretical concepts. Although some of the solutions were reportedly applied in practice, in most cases the literature did not give much information on the actual usage in industry of the proposed solutions. The case study revealed that while the relevance of each identified problem was highly variable from one company to another, there are some key problems that are generally considered vital both by research and industry. Regarding solution proposals, it turned out they are often described too much on the conceptual level or are too unrelated to the most urgent test-related problems of our case companies to be of interest to them.

Notes

Acknowledgements

We would like to thank Fob Solutions, Testlio, Mobi Lab, Wazombi, TestDevLab and MoonCascade. This research was supported by the Estonian Research Council.

References

  1. 1.
    Muccini, H., Di Francesco, A., Esposito, P.: Software testing of mobile applications: challenges and future research directions. In: Proceedings of AST 2012, Piscataway, NJ, USA (2012)Google Scholar
  2. 2.
    Paul, S.: Role of mobile handhelds in redefining how we work, live and experience the world around us: some challenges and opportunities. In: Proceedings of SIGCOMM 2010, New Delhi, India (2010)Google Scholar
  3. 3.
    Wasserman, A.I.: Software engineering issues for mobile application development. In: Software Engineering Issues for Mobile Application Development, Santa Fe, New Mexico, USA (2010)Google Scholar
  4. 4.
    Santos, A., Correia, I.: Mobile testing in software industry using agile: challenges and opportunities. In: Proceedings of ICST 2015, Graz, Austria (2015)Google Scholar
  5. 5.
    N. T., Did you know what was the first smartphone ever? PhoneArena, 31 July 2014. http://www.phonearena.com/news/Did-you-know-what-was-the-first-smartphone-ever_id58842. Accessed 10 May 2016
  6. 6.
    Apple Inc., Apple Reinvents the Phone with iPhone. http://www.apple.com/pr/library/2007/01/09Apple-Reinvents-the-Phone-with-iPhone.html. Accessed 12 May 2016
  7. 7.
    Statista Inc., Number of smartphones sold to end users worldwide from 2007 to 2015 (in million units). http://www.statista.com/statistics/263437/global-smartphone-sales-to-end-users-since-2007/. Accessed 14 May 2016
  8. 8.
    Martinie, C., Palanque, P.: Design, development and evaluation challenges for future mobile user interfaces in safety-critical contexts. In: Proceedings of the 2015 Workshop on Future Mobile User Interfaces, Florence, Italy (2015)Google Scholar
  9. 9.
    Corral, L., Sillitti, A., Succi, G.: Software assurance practices for mobile applications. Computing 97(10), 1001–1022 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Net Applications, Mobile/Tablet Operating System Market Share January–March 2016. https://www.netmarketshare.com/operating-system-market-share.aspx?qprid=8&qpcustomd=1&qpsp=68&qpnp=1&qptimeframe=Q&qpmr=10&qpdt=0&qpct=3. Accessed 12 May 2016
  11. 11.
    Albanesius, C.: Nadella Raises Eyebrows With Plans to ‘Streamline’ Windows, PC Magazine, 23 July 2014. http://www.pcmag.com/article2/0,2817,2461253,00.asp. Accessed 13 May 2016
  12. 12.
    Kitchenham, B.A., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Technical report EBSE-2007-01, School of Computer Science and Mathematics, Keele University (2007)Google Scholar
  13. 13.
    Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14(2), 131–164 (2009)CrossRefGoogle Scholar
  14. 14.
    Charland, A., Leroux, B.: Mobile application development: web vs. native. Commun. ACM 54(5), 49–53 (2011)CrossRefGoogle Scholar
  15. 15.
    Samuel, T.: Problems and solutions in mobile application testing, MSc thesis, University of Tartu, Estonia (2016). https://comserv.cs.ut.ee/ati_thesis/datasheet.php?id=54422&year=2016
  16. 16.
    Zhifang, L., Bin, L., Xiaopeng, G.: Test automation on mobile device. In: Proceedings of the 5th Workshop on Automation of Software Test, Cape Town, South Africa (2010)Google Scholar
  17. 17.
    Yan, M., Sun, H., Liu, X.: ITest: testing software with mobile crowdsourcing. In: Proceedings of CrowdSoft 2014, Hong Kong, China (2014)Google Scholar
  18. 18.
    Kaasila, J., Ferreira, D., Kostakos, V., Ojala, T.: Testdroid: automated remote UI testing on android. In: Proceedings of 11th International Conference on Mobile and Ubiquitous Multimedia, Ulm, Germany (2012)Google Scholar
  19. 19.
    Ravindranath, L., Nath, S., Padhye, J., Balakrishnan, H.: Automatic and scalable fault detection for mobile applications. In: Proceeidngs of MobiSys 2014, Bretton Woods, New Hampshire, USA (2014)Google Scholar
  20. 20.
    Ham, H., Park, Y.: Designing knowledge base mobile application compatibility test system for android fragmentation. Intl. J. Softw. Eng. Appl. 8(1), 303–314 (2014)Google Scholar
  21. 21.
    Tang, L., Yu, Z., Zhou, X., Wang, H., Becker, C.: Supporting rapid design and evaluation of pervasive applications: challenges and solutions. Pers. Ubiquit. Comput. 15(3), 253–269 (2011)CrossRefGoogle Scholar
  22. 22.
    Galindo, J.A., Turner, H., Benavides, D., White, J.: Testing variability-intensive systems using automated analysis: an application to Android. Softw. Qual. J. 42(2), 365–405 (2014)Google Scholar
  23. 23.
    Baride, S., Dutta, K.: A cloud based software testing paradigm for mobile applications. SIGSOFT Softw. Eng. Notes 36(3), 1–4 (2011)CrossRefGoogle Scholar
  24. 24.
    Azim, T., Neamtiu, I.: Targeted and depth-first exploration for systematic testing of Android apps. ACM SIGPLAN Not. 48(10), 641–660 (2013)CrossRefGoogle Scholar
  25. 25.
    Arzt, S., Rasthofer, S., Fritz, C., Bodden, E., Bartel, A., Klein, J., Le Traon, Y., Octeau, D., McDaniel, P.: FLOWDROID: precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps. ACM SIGPLAN Notes 49(6), 259–269 (2014)CrossRefGoogle Scholar
  26. 26.
    Haller, K.: Mobile testing. SIGSOFT Softw. Eng. Notes 38(6), 1–8 (2013)CrossRefGoogle Scholar
  27. 27.
    Bastien, J.C.: Usability testing: a review of some methodological and technical aspects of the method. Intl. J. Med. Inf. 79(4), e18–e23 (2010)CrossRefGoogle Scholar
  28. 28.
    Ma, X., Yan, B., Chen, G., Zhang, C., Huang, K., Drury, J., Wang, L.: Design and implementation of a toolkit for usability testing of mobile apps. Mob. Netw. Appl. 18(1), 81–97 (2013)CrossRefGoogle Scholar
  29. 29.
    Mirzaei, N., Malek, S., Pasareanu, C.S., Esfahani, N., Mahmood, R.: Testing Android apps through symbolic execution. SIGSOFT Softw. Eng. Notes 37(6), 1–5 (2012)CrossRefGoogle Scholar
  30. 30.
    Kim, H.-K.: Hybrid model based testing for mobile applications. Intl. J. Softw. Eng. Appl. 7(3), 223–238 (2013)Google Scholar
  31. 31.
    van der Merwe, H., Tkachuk, O., van der Merwe, B., Visser, W.: Generation of library models for verification of android applications. SIGSOFT Softw. Eng. Notes 40(1), 1–5 (2015)CrossRefGoogle Scholar
  32. 32.
    Hussain, A., Hashim, N., Nordin, N., Tahir, H.: A metric-based evaluation model for applications on mobile phones. J. Inf. Commun. Technol. 12(1), 55–71 (2013)Google Scholar
  33. 33.
    Koivisto, E.M.I., Suomela, R.: Using prototypes in early pervasive game development. In: Proceedings of ACM SIGGRAPH Symposium on Video Games, San Diego, California, USA (2007)Google Scholar
  34. 34.
    van der Merwe, H., Tkachuk, O., Nel, S., van der Merwe, B., Visser, W.: Environment modeling using runtime values for JPF-Android. SIGSOFT Softw. Eng. Notes 40(6), 1–5 (2015)CrossRefGoogle Scholar
  35. 35.
    Agarwal, S., Mahajan, R., Zheng, A., Bahl, V.: Diagnosing mobile applications in the wild. In: Proceedings of ACM SIGCOMM Workshop on Hot Topics in Networks, Monterey, California (2010)Google Scholar
  36. 36.
    Kim, H., Choi, B., Yoon, S.: Performance testing based on test-driven development for mobile applications. In: Proceedings of ICUIMC 2009, Suwon, South Korea, (2009)Google Scholar
  37. 37.
    Ceccato, M., Avancini, A.: Security testing of the communication among Android applications. In: Proceedings of AST 2013, San Francisco, CA, USA (2013)Google Scholar
  38. 38.
    Adamsen, C.Q., Mezzetti, G., Moller, A.: Systematic execution of android test suites in adverse conditions. In: Proceedings of ISSTA 2015, Baltimore, MD, USA (2015)Google Scholar
  39. 39.
    Zhang, P., Elbaum, S.: Amplifying tests to validate exception handling code: an extended study in the mobile application domain. ACM Trans. Softw. Eng. Methodol. 23(4), 32:1–32:28 (2014)CrossRefGoogle Scholar
  40. 40.
    De Cleva Farto, G., Endo, A.: Evaluating the model-based testing approach in the context of mobile applications. Electron. Notes Theor. Comput. Sci. 314, 3–21 (2015)CrossRefGoogle Scholar
  41. 41.
    Zapata, B.C., Fernandez-Aleman, J.L., Idri, A., Toval, A.: Empirical studies on usability of mHealth apps: a systematic literature review. J. Med. Syst. 39(2), 1–19 (2015)CrossRefGoogle Scholar
  42. 42.
    Diewald, S., Geilhof, B., Siegrist, M., Lindemann, P., Koelle, M., Halle, M., Kranz, M.: Mobile AgeCI: potential challenges in the development and evaluation of mobile applications for elderly people. In: Computer Aided Systems Theory – EUROCAST 2015, Las Palmas, Spain (2015)Google Scholar
  43. 43.
    Oulasvirta, A.: Rethinking experimental designs for field evaluations. IEEE Pervasive Comput. 11(4), 60–67 (2012)CrossRefGoogle Scholar
  44. 44.
    Biel, B., Grill, T., Gruhn, V.: Exploring the benefits of the combination of a software architecture analysis and a usability evaluation of a mobile application. J. Syst. Softw. 83(11), 2031–2044 (2010)CrossRefGoogle Scholar
  45. 45.
    Rapp, A., Cena, F., Gena, C., Marcengo, A., Console, L.: Using game mechanics for field evaluation of prototype social applications: a novel methodology. Behav. Inf. Technol. 35(3), 184–195 (2015)CrossRefGoogle Scholar
  46. 46.
    Billi, M., Burzagli, L., Catarci, T., Santucci, G., Bertini, E., Gabbanini, F., Palchetti, E.: A unified methodology for the evaluation of accessibility and usability of mobile applications. Univ. Access Inf. Soc. 9(4), 337–356 (2010)CrossRefGoogle Scholar
  47. 47.
    Nascimento, L.H.D., Machado, P.D.: An experimental evaluation of approaches to feature testing in the mobile phone applications domain. In: Proceedings of DOSTA 2007: in Conjunction with the 6th ESEC/FSE Joint Meeting, Dubrovnik, Croatia (2007)Google Scholar
  48. 48.
    van der Merwe, H., van der Merwe, B., Visser, W.: Verifying android applications using Java pathfinder. SIGSOFT Softw. Eng. Notes 37(6), 1–5 (2012)CrossRefGoogle Scholar
  49. 49.
    Salva, S., Zafimiharisoa, S.R.: APSET, an Android aPplication SEcurity Testing tool for detecting intent-based vulnerabilities. Intl. J. Softw. Tools Technol. Transfer 17, 201–221 (2015)CrossRefGoogle Scholar
  50. 50.
    Aranha, E., Borba, P.: Estimating manual test execution effort and capacity based on execution points. Intl. J. Comput. Appl. 31(3), 167–172 (2009)Google Scholar
  51. 51.
    Serra, L.C., Carvalho, L.P., Ferreira, L.P., Vaz, J.B.S., Freire, A.P.: Accessibility evaluation of e-government mobile applications in Brazil. Procedia Comp. Sci. 37, 348–357 (2015)CrossRefGoogle Scholar
  52. 52.
    Morgado, I.C., Paiva, A.C.R.: Test patterns for android mobile applications. In: Proceedings of the 20th European Conference on Pattern Languages of Programs, Kaufbeuren, Germany (2015)Google Scholar
  53. 53.
    Wang, X.S., Balasubramanian, A., Krishnamurthy, A., Wetherall, D.: Demystifying page load performance with WProf. In: Proceedings of the 10th USENIX Conference on Networked Systems Design and Implementation, Lombard, IL (2013)Google Scholar
  54. 54.
    Hutflesz, P., Holzmann, C.: Multivariate testing of native mobile applications. In: Proceedings of MoMM 2014, Kaohsiung, Taiwan (2014)Google Scholar
  55. 55.
    Choi, W., Necula, G., Sen, K.: Guided GUI testing of Android apps with minimal restart and approximate learning. ACM SIGPLAN Not. 48(10), 623–639 (2013)CrossRefGoogle Scholar
  56. 56.
    Hu, Y., Azim, T., Neamtiu, I.: Versatile yet lightweight record-and-replay for Android. In: Proceedings of 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, vol. 50(10), pp. 349–366 (2015)Google Scholar
  57. 57.
    Gao, J., Tsai, W.-T., Paul, R., Bai, X., Uehara, T.: Mobile testing-as-a-service (MTaaS) - infrastructures, issues, solutions and needs. In: Proceedings of 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering, Miami, FL, USA (2014)Google Scholar
  58. 58.
    Costa, P., Paiva, A.C.R., Nabuco, M.: Pattern based GUI testing for mobile applications. In: 9th International Conference on the Quality of Information and Communications, Guimaraes, Portugal (2014)Google Scholar
  59. 59.
    Bo, J., Xiang, L., Xiaopeng, G.: MobileTest: a tool supporting automatic black box test for software on smart mobile devices. In: Proceedings of AST 2007, Washington, DC, USA (2007)Google Scholar
  60. 60.
    Google, Performance focus. http://developer.android.com/about/versions/lollipop.html#Perf. Accessed 8 Apr 2016
  61. 61.
    Amalfitano, D., Fasolino, A.R., Tramontana, P., Ta, B.D., Memon, A.M.: MobiGUITAR: automated model-based testing of mobile apps. IEEE Softw. 32(5), 53–59 (2015)CrossRefGoogle Scholar
  62. 62.
    Amalfitano, D., Amatucci, N., Fasolino, A.R., Tramontana, P.: AGRippin: a novel search based testing technique for android applications. In: Proceedings of 3rd International Workshop on Software Development Lifecycle for Mobile, Bergamo, Italy (2015)Google Scholar
  63. 63.
    Amalfitano, D., Amatucci, N., Fasolino, A.R., Tramontana, P., Kowalczyk, E., Memon, A.M.: Exploiting the saturation effect in automatic random testing of android applications. In: Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems, Florence, Italy (2015)Google Scholar
  64. 64.
    Amalfitano, D., Fasolino, A.R., Tramontana, P., De Carmine, S., Memon, A.M.: Using GUI ripping for automated testing of android applications. In: Proceedings of ASE 2012, Essen, Germany (2012)Google Scholar
  65. 65.
    Bergvall-Kareborn, B., Larsson, S.: A case study of real-world testing. In: Proceedings of the 7th International Conference on Mobile and Ubiquitous Multimedia, Umeå, Sweden (2008)Google Scholar
  66. 66.
    Guo, C., Xu, J., Yang, H., Zeng, Y., Xing, S.: An automated testing approach for inter-application security in android. In: Proceedings of AST 2014, Hyderabad, India (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Institute of Computer ScienceUniversity of TartuTartuEstonia

Personalised recommendations