Studying the consistency of star ratings and the complaints in 1 & 2-star user reviews for top free cross-platform Android and iOS apps

Article
  • 14 Downloads

Abstract

How users rate a mobile app via star ratings and user reviews is of utmost importance for the success of an app. Recent studies and surveys show that users rely heavily on star ratings and user reviews that are provided by other users, for deciding which app to download. However, understanding star ratings and user reviews is a complicated matter, since they are influenced by many factors such as the actual quality of the app and how the user perceives such quality relative to their expectations, which are in turn influenced by their prior experiences and expectations relative to other apps on the platform (e.g., iOS versus Android). Nevertheless, star ratings and user reviews provide developers with valuable information for improving the overall impression of their app. In an effort to expand their revenue and reach more users, app developers commonly build cross-platform apps, i.e., apps that are available on multiple platforms. As star ratings and user reviews are of such importance in the mobile app industry, it is essential for developers of cross-platform apps to maintain a consistent level of star ratings and user reviews for their apps across the various platforms on which they are available. In this paper, we investigate whether cross-platform apps achieve a consistent level of star ratings and user reviews. We manually identify 19 cross-platform apps and conduct an empirical study on their star ratings and user reviews. By manually tagging 9,902 1 & 2-star reviews of the studied cross-platform apps, we discover that the distribution of the frequency of complaint types varies across platforms. Finally, we study the negative impact ratio of complaint types and find that for some apps, users have higher expectations on one platform. All our proposed techniques and our methodologies are generic and can be used for any app. Our findings show that at least 79% of the studied cross-platform apps do not have consistent star ratings, which suggests that different quality assurance efforts need to be considered by developers for the different platforms that they wish to support.

Keywords

Mobile apps Cross-platform apps Android iOS Star rating User reviews 

References

  1. Akdeniz (2014) Google play crawler JAVA API. https://github.com/Akdeniz/google-play-crawler, (last visited: May 30, 2017)
  2. Ali M, Joorabchi M, Mesbah A (2017) Same app, different app stores: a comparative study. In: Proceedings of the IEEE/ACM international conference on mobile software engineering and systems (mobilesoft). IEEE Computer Society, p 12Google Scholar
  3. Android Authority (2017) Android 7 nougat update tracker. http://www.androidauthority.com/android-7-0-update-679175/, (last visited: May 30, 2017)
  4. APK4Fun (2016) Version history of the app facebook in Android. http://www.apk4fun.com/history/2430/, (last visited: May 30, 2017)
  5. AppAnnie (2015) U.s version of the top 50 apps chart of the app store. https://www.appannie.com/apps/ios/top/, (last visited: May 30, 2017)
  6. Apple (2008) RSS Feed provided by apple for the app facebook. https://itunes.apple.com/us/rss/customerreviews/id=284882215/page=1/json, (last visited: May 30, 2017)
  7. Apple (2016) If an app you installed unexpectedly quits, stops responding, or won’t open. https://support.apple.com/en-nz/HT201398, (last visited: May 30, 2017)
  8. Apple (2017) App store review guidelines. https://developer.apple.com/app-store/review/guidelines/, (last visited: May 30, 2017)
  9. Bavota G, Linares-Vásquez M, Bernal-Cárdenas CE, Penta MD, Oliveto R, Poshyvanyk D (2015) The impact of API change- and fault-proneness on the user ratings of Android apps. IEEE Trans Softw Eng (TSE) 41(4):384–407CrossRefGoogle Scholar
  10. Benenson Z, Gassmann F, Reinfelder L (2013) Android and iOS users’ differences concerning security and privacy. In: Extended abstracts on human factors in computing systems (CHI), pp 817–822Google Scholar
  11. Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res (JMLR) 3:993–1022MATHGoogle Scholar
  12. Chen Y, Xu H, Zhou Y, Zhu S (2013) Is this app safe for children?: a comparison study of maturity ratings on Android and iOS applications. In: Proceedings of the 22nd international conference on world wide web (WWW). ACM, New York, pp 201–212Google Scholar
  13. Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) AR-Miner: mining informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th international conference on software engineering (ICSE). ACM, New York, pp 767–778Google Scholar
  14. Dalmasso I, Datta SK, Bonnet C, Nikaein N (2013) Survey, comparison and evaluation of cross platform mobile application development tools. In: 9th international wireless communications and mobile computing conference (IWCMC), pp 323–328Google Scholar
  15. Dann J (2012) Under the hood: Rebuilding facebook for iOS. https://www.facebook.com/notes/facebook-engineering/under-the-hood-rebuilding-facebook-for-ios/10151036091753920/, (last visited: May 30, 2017)
  16. Di Sorbo A, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT international symposium on foundations of software engineering (FSE), ACM, pp 499–510Google Scholar
  17. Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining (KDD). ACM, New York, pp 1276–1284Google Scholar
  18. Graphpad Software (2015) Interpreting results: skewness and kurtosis. http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_skewness_and_kurtosis.htm, (last visited: May 30, 2017)
  19. Guzman E, Maalej W (2014) How do users like this feature? a fine grained sentiment analysis of app reviews. In: 22nd international requirements engineering conference (RE). IEEE, pp 153–162Google Scholar
  20. Harman M, Jia Y, Zhang Y (2012) App store mining and analysis: MSR for app stores. In: 9th working conference on mining software repositories (MSR). IEEE, pp 108–111Google Scholar
  21. Hartmann G, Stead G, DeGani A (2011) Cross-platform mobile development. Mobile Learning Environment, CambridgeGoogle Scholar
  22. Hassan S, Shang W, Hassan AE (2017) An empirical study of emergency updates for top Android mobile apps. Empir Softw Eng (EMSE) 22(1):505–546CrossRefGoogle Scholar
  23. Heitkötter H, Hanschke S, Majchrzak TA (2013) Evaluating cross-platform development approaches for mobile applications. In: Web information systems and technologies, springer, pp 120–138Google Scholar
  24. Herraiz I, Shihab E, Nguyen THD, Hassan AE (2011) Impact of installation counts on perceived quality: a case study on Debian. In: 18th working conference on reverse engineering (WCRE), pp 219–228Google Scholar
  25. Hixon T (2014) What kind of person prefers an iphone? https://www.forbes.com/sites/toddhixon/2014/04/10/what-kind-of-person-prefers-an-iphone/, (last visited: May 30, 2017)
  26. Howison J, Crowston K (2004) The perils and pitfalls of mining SourceForge. In: International workshop on mining software repositories (MSR), pp 7–11Google Scholar
  27. Hu H, Bezemer CP, Hassan AE (2017) Supplementary material: Studying the consistency of star ratings and 1 & 2-star user reviews for top free cross-platform Android and iOS apps. http://sailhome.cs.queensu.ca/replication/cross-platform-mobile-apps/, (last visited: May 30, 2017)
  28. Joorabchi M, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: International symposium on empirical software engineering and measurement (ESEM). IEEE/ACM, pp 15–24Google Scholar
  29. Joorabchi ME, Ali M, Mesbah A (2015) Detecting inconsistencies in multi-platform mobile apps. In: 26th international symposium on software reliability engineering (ISSRE). IEEE, pp 450–460Google Scholar
  30. Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th working conference on mining software repositories, MSR 2014, pp 92–101Google Scholar
  31. Khalid H, Nagappan M, Shihab E, Hassan AE (2014) Prioritizing the devices to test your app on: A case study of Android game apps. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering (FSE). ACM, pp 610–620Google Scholar
  32. Khalid H, Shihab E, Nagappan M, Hassan AE (2015) What do mobile app users complain about?. IEEE Softw 32(3):70–77CrossRefGoogle Scholar
  33. Kim HW, Lee HL, Son JE (2011) An exploratory study on the determinants of smartphone app purchase. In: The 11th international DSI and the 16th APDSI joint meeting, Taipei, TaiwanGoogle Scholar
  34. Laurence G, Janessa R (2015) Gartner says smartphone sales surpassed one billion units in 2014. http://www.gartner.com/newsroom/id/2996817, (last visited: May 30, 2017)
  35. Lim SL, Bentley PJ, Kanakam N, Ishikawa F, Honiden S (2015) Investigating country differences in mobile app user behavior and challenges for software engineering. IEEE Trans Softw Eng (TSE) 41(1):40–64CrossRefGoogle Scholar
  36. Linares-Vásquez M, Bavota G, Bernal-Cárdenas C, Di Penta M, Oliveto R, Poshyvanyk D (2013) API Change and fault proneness: a threat to the success of Android apps. In: Proceedings of the 9th joint meeting on foundations of software engineering (ESEC/FSE). ACM, pp 477–487Google Scholar
  37. Long JD, Feng D, Cliff N (2003) Ordinal analysis of behavioral data. Wiley, New YorkCrossRefGoogle Scholar
  38. Martin W, Harman M, Jia Y, Sarro F, Zhang Y (2015) The app sampling problem for app store mining. In: Proceedings of the 12th working conference on mining software repositories, MSR ’15. IEEE Press, Piscataway, pp 123–133Google Scholar
  39. Martin W, Sarro F, Harman M (2016) Causal impact analysis for app releases in google play. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2016. ACM, New York, pp 435–446.  https://doi.org/10.1145/2950290.2950320
  40. Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng (TSE) PP(99):1–32Google Scholar
  41. McIlroy S, Ali N, Khalid H, E Hassan A (2015) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng (EMSE) 21(3):1067–1106CrossRefGoogle Scholar
  42. McIlroy S, Ali N, Khalid H, E Hassan A (2016) Analyzing and automatically labelling the types of user issues that are raised in mobile app reviews. Empir Softw Eng (EMSE) 21(3):1067–1106CrossRefGoogle Scholar
  43. Mercado IT, Munaiah N, Meneely A (2016) The impact of cross-platform development approaches for mobile applications from the user’s perspective. In: Proceedings of the international workshop on app market analytics (WAMA). ACM, pp 43–49Google Scholar
  44. Morani L (2015) There are now more than 24,000 different Android devices. http://qz.com/472767/there-are-now-more-than-24000-different-Android-devices/, (last visited: May 30, 2017)
  45. Mudambi SM, Schuff D (2010) What makes a helpful online review? a study of customer reviews on amazon.com. MIS Q 34(1):185–200CrossRefGoogle Scholar
  46. Nayebi M, Adams B, Ruhe G (2016) Release practices for mobile apps – what do users and developers think?. In: 23Nd international conference on software analysis, evolution and reengineering (SANER), vol 1. IEEE, pp 552–562Google Scholar
  47. Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of Android apps. Empir Softw Eng (EMSE) 22(6):3088–3116CrossRefGoogle Scholar
  48. Pagano D, Maalej W (2013) User feedback in the appstore: an empirical study. In: 21st international requirements engineering conference (RE). IEEE, pp 125–134Google Scholar
  49. Palmieri M, Singh I, Cicchetti A (2012) Comparison of cross-platform mobile development tools. In: 16th international conference on intelligence in next generation networks (ICIN). IEEE, pp 179–186Google Scholar
  50. Palomba F, Linares-Vásquez M, Bavota G, Oliveto R, Penta MD, Poshyvanyk D, Lucia AD (2015) User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In: 2015 IEEE international conference on software maintenance and evolution (ICSME), pp 291–300Google Scholar
  51. Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th international conference on software engineering (ICSE). IEEE Press, Piscataway, pp 106–117Google Scholar
  52. Panichella S, Sorbo AD, Guzman E, Visaggio CA, Canfora G, Gall HC (2015) How can I improve my app? Classifying user reviews for software maintenance and evolution. In: International conference on software maintenance and evolution (ICSME). IEEE, pp 281–290Google Scholar
  53. Patterson B (2016) Blake’s iOS device specifications grid. http://blakespot.com/ios_device_specifications_grid.html, (last visited: May 30, 2017)
  54. Pettey C, Rob van der M (2012) Gartner says free apps will account for nearly 90 percent of total mobile app store downloads in 2012. http://www.gartner.com/newsroom/id/2153215, (last visited: May 30, 2017)
  55. Porter MF (1997) An algorithm for suffix stripping. In: Readings in information retrieval. Morgan Kaufmann Publishers Inc., San Francisco, pp 313–316Google Scholar
  56. Poschenrieder M (2015) 77% Will not download a retail app rated lower than 3 stars. https://blog.testmunk.com/77-will-not-download-a-retail-app-rated-lower-than-3-stars/, (last visited: May 30, 2017)
  57. Ramon L, Ryan R, Kathy N (2015a) Number of apps available in leading app stores as of july 2015. http://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/, (last visited: May 30, 2017)
  58. Ramon L, Ryan R, Kathy N (2015b) Smartphone OS market share, 2015 q2. http://www.idc.com/prodserv/smartphone-os-market-share.jsp, (last visited: May 30, 2017)
  59. Raphael J (2016) Android upgrade report card: grading the manufacturers on marshmallow. http://www.computerworld.com/article/3052937/android/android-upgrade-report-card-marshmallow.html, (last visited: May 30, 2017)
  60. Romano J, Kromrey JD, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: are the t-test and Cohen’s d indices the most appropriate choices. In: Annual meeting of the southern association for institutional researchGoogle Scholar
  61. Schick S (2014) Report: iOS app users are often richer than Android users. http://www.fiercewireless.com/developer/report-ios-app-users-are-often-richer-than-android-users, (last visited: May 30, 2017)
  62. Shaw H, Ellis DA, Kendrick LR, Ziegler F, Wiseman R (2016) Predicting smartphone operating system from personality and individual differences. Cyberpsychol Behav Soc Netw 19(12):727–732CrossRefGoogle Scholar
  63. Syer MD, Nagappan M, Adams B, Hassan AE (2015) Studying the relationship between source code quality and mobile platform dependence. Software Quality Journal (SQJ) 23(3):485–508CrossRefGoogle Scholar
  64. Tian Y, Nagappan M, Lo D, Hassan AE (2015) What are the characteristics of high-rated apps? A case study on free Android applications. In: IEEE international conference on software maintenance and evolution (ICSME), pp 301–310Google Scholar
  65. Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th international conference on software engineering (ICSE). ACM, New York, pp 14–24Google Scholar
  66. Yichuan M, Cuiyun G, Michael RL, Jiuchun J (2016) Experience report: understanding cross-platform app issues from user reviews. In: 27th international symposium on software reliability engineering (ISSRE). IEEEGoogle Scholar
  67. Zaki MJ (2000) Scalable algorithms for association mining. IEEE Trans Knowl Data Eng (TKDE) 12(3):372–390CrossRefGoogle Scholar
  68. Zhao WX, Jiang J, Weng J, He J, Lim EP, Yan H, Li X (2011) Comparing twitter and traditional media using topic models. Springer, Berlin, pp 338–349Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Hanyang Hu
    • 1
  • Cor-Paul Bezemer
    • 1
  • Ahmed E. Hassan
    • 1
  1. 1.Queen’s UniversityKingstonCanada

Personalised recommendations