Skip to main content
SpringerLink
Log in

Studying the consistency of star ratings and reviews of popular free hybrid Android and iOS apps

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Nowadays, many developers make their mobile apps available on multiple platforms (e.g., Android and iOS). However, maintaining several versions of a cross-platform app that is built natively (i.e., using platform-specific tools) is a complicated task. Instead, developers can choose to use hybrid development tools, such as PhoneGap, to build hybrid apps. Hybrid apps are based on a single codebase across platforms. There exist two ways to use a hybrid development tool to build a hybrid app that runs on multiple platforms: (1) using web technologies (i.e., HTML, Javascript and CSS) and (2) in a common language, which is then converted to native code. We study whether these hybrid development tools achieve their main purpose: delivering an app that is perceived similarly by users across platforms. Prior studies show that users refer to star ratings and user reviews, when deciding to download an app. Given the importance of star ratings and user reviews, we study whether the usage of a hybrid development tool assists app developers in achieving consistency in the star ratings and user reviews across multiple platforms. We study 68 hybrid app-pairs, i.e., apps that exist both in the Google Play store and Apple App store. We find that 33 out of 68 hybrid apps do not receive consistent star ratings across platforms. We run Twitter-LDA on user reviews and find that the star ratings of the reviews that discuss the same topic could be up to three times as high across platforms. Our findings suggest that while hybrid apps are better at providing consistent star ratings and user reviews when compared to cross-platform apps that are built natively, hybrid apps do not guarantee such consistency. Hence, developers should not solely rely on hybrid development tools to achieve consistency in the star ratings and reviews that are given by users of their apps. In particular, developers should track closely the ratings and reviews of their apps across platforms, so that they can act accordingly when platform-specific issues arise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. https://zenodo.org/record/1181881

References

  • Adobe (2017) Phonegap. https://phonegap.com/, (last visited: Oct 3, 2017)

  • Akdeniz (2014) Google Play Crawler JAVA API. https://github.com/Akdeniz/google-play-crawler, (last visited: Jan 25, 2017)

  • Ali M, Mesbah A (2016) Mining and characterizing hybrid apps. In: Proceedings of the International Workshop on App Market Analytics (WAMA), ACM, pp 50–56

  • Apple (2008) RSS feed provided by Apple for the app “Facebook”. https://itunes.apple.com/us/rss/customerreviews/id=284882215/page=1/json, (last visited: Jan 25, 2017)

  • Benenson Z, Gassmann F, Reinfelder L (2013) Android and iOS users’ differences concerning security and privacy. In: Extended Abstracts on Human Factors in Computing Systems (CHI), pp 817–822

  • Blei D M, Ng A Y, Jordan M I (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  • Chen N, Lin J, Hoi S C H, Xiao X, Zhang B (2014) Ar-miner: Mining informative reviews for developers from mobile app marketplace. In: Proceedings of the 36th International Conference on Software Engineering (ICSE). ACM, New York, pp 767–778

  • Dalmasso I, Datta SK, Bonnet C, Nikaein N (2013) Survey, comparison and evaluation of cross platform mobile application development tools. In: 2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC), pp 323–328

  • Di Sorbo A, Panichella S, Alexandru C V, Shimagaki J, Visaggio C A, Canfora G, Gall HC (2016) What would users change in my app? Summarizing app reviews for recommending software changes. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), ACM, pp 499–510

  • Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: Making sense of user feedback in a mobile app store. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, pp 1276–1284

  • Graphpad Software (2015) Interpreting results: Skewness and kurtosis. http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_skewness_and_kurtosis.htm, (last visited: Jan 30, 2016)

  • Gu X, Kim S (2015) What parts of your apps are loved by users? In: 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 760–770

  • Guzman E, Maalej W (2014) How do users like this feature? [a] fine grained sentiment analysis of app reviews. In: 22nd International Requirements Engineering Conference (RE), IEEE, pp 153–162

  • Harman M, Jia Y, Zhang Y (2012) App store mining and analysis: MSR for app stores. In: 9th Working Conference on Mining Software Repositories (MSR), IEEE, pp 108–111

  • Hassan S, Shang W, Hassan A E (2017) An empirical study of emergency updates for top Android mobile apps. Empir Softw Eng 22(1):505–546

    Article  Google Scholar 

  • Heitkötter H, Hanschke S, Majchrzak T A (2013) Evaluating cross-platform development approaches for mobile applications. Springer Berlin Heidelberg, Berlin, pp 120–138

    Google Scholar 

  • Hu H, Bezemer CP, Hassan AE (2016) Studying the consistency of star ratings and the complaints in 1 & 2-star user reviews for top free cross-platform Android and iOS apps. https://peerj.com/preprints/2589/

  • Joanes D N, Gill C A (1998) Comparing measures of sample skewness and kurtosis. J R Stat Soc Ser D (Stat) 47(1):183–189

    Article  Google Scholar 

  • Joorabchi M, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE/ACM, pp 15–24

  • Joorabchi ME, Ali M, Mesbah A (2015) Detecting inconsistencies in multi-platform mobile apps. In: IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pp 450–460

  • Khalid H, Shihab E, Nagappan M, Hassan A E (2015) What do mobile app users complain about IEEE Soft 32(3):70–77

    Article  Google Scholar 

  • Long J D, Feng D, Cliff N (2003) Ordinal analysis of behavioral data. Wiley, New York

    Book  Google Scholar 

  • Man Y, Gao C, Lyu MR, Jiang J (2016) Experience report: Understanding cross-platform app issues from user reviews. In: IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), pp 138–149

  • Mann H B, Whitney D R (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Statist 18(1):50–60

    Article  MathSciNet  MATH  Google Scholar 

  • Martin W, Harman M, Jia Y, Sarro F, Zhang Y (2015) The app sampling problem for app store mining. In: Proceedings of the 12th Working Conference on Mining Software Repositories (MSR), IEEE Press, pp 123–133

  • Martin W, Sarro F, Harman M (2016a) Causal impact analysis for app releases in google play. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York, pp 435–446

  • Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2016b) A survey of app store analysis for software engineering. IEEE Trans. Softw. Eng. PP(99):1–32

    Google Scholar 

  • Microsoft (2017) Xamarin: Mobile app development and app creation software. https://www.xamarin.com/, (last visited: Oct 3, 2017)

  • NIST/SEMATECH (2012) e-handbook of statistical methods: Measures of skewness and kurtosis. http://www.itl.nist.gov/div898/handbook/eda/section3/eda35b.htm, (last visited: Oct 3, 2017)

  • Noei E, Syer M D, Zou Y, Hassan A E, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of Android apps. Empir Softw Eng 22:1–29

    Article  Google Scholar 

  • Ohrt J, Turau V (2012) Cross-platform development tools for smartphone applications. Computer 45(9):72–79

    Article  Google Scholar 

  • Pagano D, Maalej W (2013) User feedback in the appstore: An empirical study. In: 21st International Requirements Engineering Conference (RE), IEEE, pp 125–134

  • Palmieri M, Singh I, Cicchetti A (2012) Comparison of cross-platform mobile development tools. In: 2012 16th International Conference on Intelligence in Next Generation Networks (ICIN), pp 179–186

  • Palomba F, Linares-Vásquez M, Bavota G, Oliveto R, Penta MD, Poshyvanyk D, Lucia AD (2015) User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 291–300

  • Palomba F, Salza P, Ciurumelea A, Panichella S, Gall H, Ferrucci F, De Lucia A (2017) Recommending and localizing change requests for mobile apps based on user reviews. In: Proceedings of the 39th International Conference on Software Engineering (ICSE). IEEE Press, Piscataway, pp 106–117

  • Panichella S, Sorbo A D, Guzman E, Visaggio C A, Canfora G, Gall HC (2015) How can I improve my app? Classifying user reviews for software maintenance and evolution. In: International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp 281–290

  • Pettey C, Rob van der M (2012) Gartner says free apps will account for nearly 90 percent of total mobile app store downloads in 2012. http://www.gartner.com/newsroom/id/2153215, (last visited: Jan 28, 2016)

  • Porter M F (1997) Readings in information retrieval. Morgan Kaufmann Publishers Inc., chap An Algorithm for Suffix Stripping, pp 313–316

  • Poschenrieder M (2015) 77% will not download a retail app rated lower than 3 stars. https://blog.testmunk.com/77-will-not-download-a-retail-app-rated-lower-than-3-stars/, (last visited: Jan 28, 2016)

  • Ramon L, Ryan R, Kathy N (2015) Smartphone OS market share, 2015 q2. http://www.idc.com/prodserv/smartphone-os-market-share.jsp, (last visited: Jan 25, 2017)

  • Romano J, Kromrey J D, Coraggio J, Skowronek J, Devine L (2006) Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen’s d indices the most appropriate choices. In: Annual meeting of the Southern Association for Institutional Research

  • Smutný P (2012) Mobile development tools and cross-platform solutions. In: Carpathian Control Conference (ICCC), 2012 13th International, pp 653–656

  • Statista (2017) Most popular Apple App Store categories in July 2017, by share of available apps. https://www.statista.com/statistics/270291/popular-categories-in-the-app-store/

  • Thomas S W, Adams B, Hassan A E, Blostein D (2011) Modeling the evolution of topics in source code histories. In: Proceedings of the 8th Working Conference on Mining Software Repositories (MSR), ACM, pp 173–182

  • Tian Y, Nagappan M, Lo D, Hassan AE (2015) What are the characteristics of high-rated apps? A case study on free Android applications. In: IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 301–310

  • Vashistha C (2015) Native vs hybrid mobile app: 5 ways to choose right plarform. http://www.ipragmatech.com/native-hybrid-mobile-app-right-platform/

  • Viennot N, Garcia E, Nieh J (2014) A measurement study of google play. SIGMETRICS Perform Eval Rev 42(1):221–233

    Article  Google Scholar 

  • Villarroel L, Bavota G, Russo B, Oliveto R, Di Penta M (2016) Release planning of mobile apps based on user reviews. In: Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM, New York, pp 14–24

  • Vu PM, Nguyen TT, Pham HV, Nguyen TT (2015) Mining user opinions in mobile app reviews: A keyword-based approach. In: 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 749–759

  • Zhao W X, Jiang J, Weng J, He J, Lim E P, Yan H, Li X (2011) Comparing Twitter and traditional media using topic models. In: Proceedings of the 33rd European Conference on Advances in Information Retrieval (ECIR), Springer-Verlag, pp 338–349

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Queen’s University, Kingston, ON, Canada

    Hanyang Hu, Shaowei Wang, Cor-Paul Bezemer & Ahmed E. Hassan

Authors
  1. Hanyang Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaowei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cor-Paul Bezemer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaowei Wang.

Additional information

Communicated by: Sunghun Kim

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, H., Wang, S., Bezemer, CP. et al. Studying the consistency of star ratings and reviews of popular free hybrid Android and iOS apps. Empir Software Eng 24, 7–32 (2019). https://doi.org/10.1007/s10664-018-9617-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9617-6

Keywords

Search

Navigation