This paper evaluates the effectiveness of coverage approaches for selecting mobile devices (i.e., smartphones and tablets) to test mobile software applications. Due to the large number of such devices on the market and the variations in their characteristics, it is hard to guarantee that an application will work as intended on all devices. For this reason, multi-device testing is necessary. The goal of this research was to determine how many devices must be tested and which methods for device selection are best for revealing device-specific faults. We experimentally investigated a simple coverage of all values of each device’s features separately and the each-choice coverage (i.e., the coverage of all device characteristics at the same time). To collect the experimental data, 15 Android applications were tested on 30 mobile devices and 24 device-specific faults were detected. Our research shows that a random selection of 13 devices achieved 100% effectiveness. However, coverage of device characteristics in the selection process yielded an acceptable 90% level of effectiveness with a set of only five devices. The most successful approaches were the coverage of different types of Android operating systems and the each-choice coverage. Our results include recommendations for increasing the effectiveness while decreasing the costs of mobile testing.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Adobe Edge Inspect (2016). https://creative.adobe.com/products/inspect. Accessed 27 December 2016
Barksdale M.: Mobile platform fragmentation—what it is and how we deal with it at Engage Mobile (2014). http://www.engagemobile.com/mobile-platform-fragmentation-deal-engage-mobile/. Accessed 27 December 2016
Cutler K.: How do top Android developers QA test their apps? (2012). http://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/. Accessed 27 December 2016
Duggan M.: Cell phone activities 2013 (2013). http://www.pewinternet.org/2013/09/19/cell-phone-activities-2013/. Accessed 27 December 2016
Gao, J., Bai, X., Tsai, W. T., & Uehara, T. (2014). Mobile application testing: a tutorial. Computer, 2, 46–55.
Ghostlab (2016). http://vanamco.com/ghostlab/. Accessed 27 December 2016
Google Play Services (2016). https://developer.android.com/google/play-services/index.html. Accessed 27 December 2016
Grindal, M., Offutt, J., & Andler, S. F. (2005). Combination testing strategies: a survey. Software Testing, Verification and Reliability, 15(3), 167–199.
Han, D., Zhang, C., Fan, X., Hindle, A., Wong, K., Stroulia, E.: Understanding Android fragmentation with topic analysis of vendor-specific bugs. Proceedings of the 19th Working Conference on Reverse Engineering (WCRE 2012), Kingston, ON, Canada, October 15–18, pp. 83–92 (2012)
Huang, J.: AppACTS: mobile app automated compatibility testing service. Proceedings of the 2nd IEEE Int. Conference on Mobile Cloud Computing, Services, and Engineering (MobileCloud), Oxford, UK, 8–11 April, pp. 85–90 (2014)
Keynote Mobile Testing. DeviceAnywhere (2016). http://www.keynote.com/solutions/testing/mobile-testing. Accessed 27 December 2016
Kuhn, D. R., Kacker, R. N., Lei, Y.: Introduction to combinatorial testing, Chapman and Hall/CRC, 341 pages (2013)
Kuhn, D. R., Lei, Y., Kacker, R. N.: Practical combinatorial testing—beyond pairwise. IEEE IT Professional, June, pp. 19–23 (2008)
Moore, M. F., Vilkomir, S.: Practical combinatorial testing approaches: a case study of a university portal application. Proceedings of the 28th International Conference on Software Engineering & Knowledge Engineering (SEKE 2016), Redwood City, CA, USA, July 1–3 (2016)
Muccini, H., Di Francesco, A., Esposito, P.: Software testing of mobile applications: challenges and future research directions. Proceedings of the 7th IEEE/ACM International Workshop on Automation of Software Test (AST 2012), Zurich, Switzerland, 2–3 June (2012)
OpenSignal. Android Fragmentation Visualized (2015). https://opensignal.com/reports/2015/08/android-fragmentation/. Accessed 27 December 2016
Park, J.H., Park, Y.B., Ham, H.K.: Fragmentation problem in Android. Proceedings of the International Conference on Information Science and Applications (ICISA 2013), 24–26 June, Pattaya, Thailand (2013)
pCloudy (2016). http://www.pcloudy.com. Accessed 27 December 2016
Perez, S.: Consumers spend 85% of time on smartphones in apps, but only 5 apps see heavy use (2015). http://techcrunch.com/2015/06/22/consumers-spend-85-of-time-on-smartphones-in-apps-but-only-5-apps-see-heavy-use/. Accessed 27 December 2016.
Perfecto Mobile (2016). www.perfectomobile.com. Accessed 27 December 2016
Silk Mobile (2016). http://www.borland.com/Products/Software-Testing/Automated-Testing/Silk-Mobile. Accessed 27 December 2016
Starov, O., Vilkomir S., Kharchenko, V.: Cloud testing for mobile software systems: concept and prototyping. Proceedings of the 8th International Conference on Software Engineering and Applications (ICSOFT-EA 2013), July 29–31, Reykjavik, Iceland, pp. 124–131 (2013)
Starov, O., Vilkomir, S.: Integrated TaaS platform for mobile development: architecture solutions. Proceedings of the Eighth International Workshop on Automation of Software Test (AST’13), San Francisco, USA, May 18–19 (2013)
Tao, C., Gao, J.: Modeling mobile application test platform and environment: testing criteria and complexity analysis. Proceedings of 2014 Workshop on Joining AcadeMiA and Industry Contributions to Test Automation and Model-Based Testing (JAMAICA 2014), San Jose, CA, USA, July 21, pp. 28–33 (2014)
TestObject (2016). www.testobject.com.Accessed 27 December 2016
Vilkomir, S., Amstutz, B.: Using combinatorial approaches for testing mobile applications. Proceedings of the IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2014), March 31–April 4, Cleveland, Ohio, USA, pp. 78–83 (2014)
Vilkomir, S., Marszalkowski, K., Perry C., Mahendrakar, S.: Effectiveness of multi-device testing mobile applications. Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems (MobileSoft 2015), May 16–17, Florence, Italy, pp. 44–47 (2015)
Zhang, T., Gao, J., Cheng, J., Uehara, T.: Compatibility testing service for mobile applications. IEEE Symposium on Service-Oriented System Engineering (SOSE), Mar 30, San Francisco Bay, USA, pp. 179–186 (2015)
The author wishes to thank his former students Brandi Amstutz, Katherine Marszalkowski, Swetha Mahendrakar, and Chauncey Perry for their contribution at the early stages of this research.
About this article
Cite this article
Vilkomir, S. Multi-device coverage testing of mobile applications. Software Qual J 26, 197–215 (2018). https://doi.org/10.1007/s11219-017-9357-7
- Mobile testing
- Device-specific faults
- Device selection