Skip to main content

An empirical study on release notes patterns of popular apps in the Google Play Store

Empirical Software Engineering volume 27, Article number: 55 (2022) Cite this article

Abstract

Release notes of a new mobile release provide valuable information for app users about the updated functionality of an app. Moreover, app developers can use the release notes to inform users about the resolution of a previously reported issue in user reviews. Prior work shows that release notes are an essential artifact for app developers to announce the emergency fixes and the newly adopted features. However, little is known about the common practices adapted by app developers in preparing their release notes. In this paper, we are interested in capturing the common practices as release notes patterns. First, we conduct an online survey with 102 respondents to investigate their views on mobile release notes. Our results show that most developers find release notes to be useful for notifying their user-base. Then, we study release notes patterns by analyzing 69,851 releases and 67.7 million user reviews of 2,232 top free-to-download apps in the Google Play Store over three years (from April 2016 until April 2019). We observe that app developers tend to write either long release notes (over 50 words) or short release notes (less than 7 words). We use the characteristics of release notes, such as the number of words, to identify six patterns of release notes in mobile apps. We manually investigate the release notes from each of the six patterns, and find 17 release drivers for the release notes. We also find that apps with longer release notes tend to have higher average user ratings. Furthermore, we observe that a shift from rarely updated patterns to frequently updated patterns tend to have higher average user ratings. Our work shows potential directions for developers to improve the release note mechanisms in app stores.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. https://stackoverflow.com/questions/638423/how-should-release-notes-be-written

  2. https://stackoverflow.com/questions/56774669/is-there-any-way-to-order-group-work-item-type-in-azure-devops-release-noteshttps://stackoverflow.com/questions/56774669/is-there-any-way-to-order-group-work-item-type-in-azure-devops-release-notes

  3. https://medium.com/@scottydocs/do-people-actually-read-release-notes-449d099d73ee

  4. https://medium.com/@scottydocs/do-people-actually-read-release-notes-449d099d73ee

  5. https://uxdesign.cc/design-better-release-notes-3e8c8c785231

  6. https://spectrum.ieee.org/tech-talk/telecom/internet/the-art-of-writing-app-release-notes

  7. https://f-droid.org/en/

  8. https://www.reddit.com/r/androiddev

  9. https://www.reddit.com/r/appdev/

  10. Apps in pattern 2 and pattern 5 do not have major releases during our studied period, so the centroid for major releases updatability in patterns 2 and 5 are NA

  11. Models 1-6 are the models for the patterns 1-6 as follows: (1) short non-updating steady, (2) short updating steady, (3) short rising-updatability with major releases, (4) long non-updating steady, (5) long updating steady, and (6) long rising-updatability with major releases.

  12. The bold text highlights the app attributes with the highest impact on the response variable.

  13. https://clutch.co/

References

  • App annie (2019) The app analytics and app data industry standard. https://www.appannie.com/. Accessed: 2019-08-30

  • Abebe SL, Ali N, Hassan AE (2016) An empirical study of software release notes. Empir Softw Eng 21(3):1107–1142

    Article  Google Scholar 

  • Ahasanuzzaman M, Hassan S, Bezemer C-P, Hassan AE (2020) A longitudinal study of popular ad libraries in the Google Play Store. Empir Softw Eng 25(1):824–858

    Article  Google Scholar 

  • Akdeniz (2019) Google Play Crawler. https://github.com/Akdeniz/google-play-crawler, Feb 2014. Accessed: 2019-09-30

  • Appcues (2020) 5 excellent product release note examples and how to write your own. https://www.appcues.com/blog/release-notes-examples. Accessed: 2020-01-23

  • Bi T, Xia X, Lo D, Grundy J, Zimmermann T (2020) An empirical study of release note production and usage in practice. IEEE Trans Softw Eng

  • Brockwell PJ, Davis RA, Calder MV (2002) Introduction to time series and forecasting, vol 2. Springer

  • Camargo Cruz AE, Ochimizu K (2009) Towards logistic regression models for predicting fault-prone code across software projects. In: Proceedings of the 2009 3rd international symposium on empirical software engineering and measurement. IEEE Computer Society, pp 460–463

  • Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

    Article  Google Scholar 

  • Cuzick J (1985) A wilcoxon-type test for trend. Stat Med 4(1):87–90

    Article  Google Scholar 

  • Gao C, Zeng J, Lyu MR, King I (2018) Online app review analysis for identifying emerging issues. In: 2018 IEEE/ACM 40Th international conference on software engineering (ICSE). IEEE, pp 48–58

  • Gao C, Zeng J, Xia X, Lo D, Lyu MR, King I (2019) Automating app review response generation. In: 2019 34Th IEEE/ACM international conference on automated software engineering (ASE). IEEE, pp 163–175

  • Gao C, Zhou W, Xia X, Lo D, Xie Q, Lyu MR (2020) Automating app review response generation based on contextual knowledge. arXiv:2010.06301

  • Goldberg Y, Levy O (2014) word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv:1402.3722

  • Google (2020) Prepare & roll out releases - play console help. https://support.google.com/googleplay/android-developer/answer/7159011?hl=en. Accessed: 2020-01-23

  • Gyimothy T, Ferenc R, Siket I (2005) Empirical validation of object-oriented metrics on open source software for fault prediction. IEEE Trans Softw Eng 31(10):897–910

    Article  Google Scholar 

  • Hartigan JA, Wong MA (1979) Algorithm as 136: a k-means clustering algorithm. J Royal Stat Soc Ser C (Applied Statistics) 28(1):100–108

    MATH  Google Scholar 

  • Hassan C, Bezemer C-P, Hassan AE (2018a) Tantithamthavorn Studying the dialogue between users and developers of free apps in the Google Play Store. Empir Softw Eng 23(3):1275–1312

    Article  Google Scholar 

  • Hassan S, Bezemer C-P, Hassan AE (2018b) Studying bad updates of top free-to-download apps in the Google Play Store. IEEE Trans Softw Eng

  • Hassan S, Shang W, Hassan AE (2017) An empirical study of emergency updates for top Android mobile apps. Empir Softw Eng 22(1):505–546

    Article  Google Scholar 

  • Huang J, Ling CX (2005) Using auc and accuracy in evaluating learning algorithms. IEEE Trans Knowl Data Eng 17(3):299–310

    Article  Google Scholar 

  • Khalfallah M (2018) Generation and visualization of release notes for systems engineering software. In: Proceedings of the international conference on complex systems design & management. Springer, pp 133–144

  • Khoshgoftaar TM, Allen EB (1999) Logistic regression modeling of software quality. Int J Reliab Qual Safety Eng 6(04):303–317

    Article  Google Scholar 

  • Klepper S, Krusche S, Bruegge B (2016) Semi-automatic generation of audience-specific release notes. In: Proceedings of the 2016 IEEE/ACM international workshop on continuous software evolution and delivery (CSED). IEEE, pp 19–22

  • Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics: 159–174

  • Manning C, Surdeanu M, Bauer J, Finkel J, Bethard S, McClosky D (2014) The stanford corenlp natural language processing toolkit. In: Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pp 55–60

  • Marschner I, Donoghoe MW, Donoghoe MMW (2018) Package ‘glm2’. J Vol 3(2):12–15

    Google Scholar 

  • Martin W, Sarro F, Harman M (2016) Causal impact analysis for app releases in Google Play. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 435–446

  • McIlroy S, Ali N, Hassan AE (2016) Fresh apps: an empirical study of frequently-updated mobile apps in the google play store. Empir Softw Eng 21(3):1346–1370

    Article  Google Scholar 

  • Meng X-L, Rosenthal R, Rubin DB (1992) Comparing correlated correlation coefficients. Psychol Bull 111(1):172

    Article  Google Scholar 

  • Moore DS (1977) Generalized inverses, wald’s method, and the construction of chi-squared tests of fit. J Am Stat Assoc 72(357):131–137

    MathSciNet  Article  Google Scholar 

  • Moreno L, Bavota G, Di Penta M, Oliveto R, Marcus A, Canfora G (2014) Automatic generation of release notes. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 484–495

  • Moreno L, Bavota G, Di Penta M, Oliveto R, Marcus A, Canfora G (2016) Arena: an approach for the automated generation of release notes. IEEE Trans Softw Eng 43(2):106–127

    Article  Google Scholar 

  • Nagappan N, Murphy B, Basili V (2008) The influence of organizational structure on software quality. In: Proceedings of the 2008 ACM/IEEE 30th international conference on software engineering. IEEE, pp 521–530

  • Noei E, Da Costa DA, Zou Y (2018) Winning the app production rally. In: Proceedings of the 2018 26th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering. ACM, pp 283–294

  • Noei E, Syer MD, Zou Y, Hassan AE, Keivanloo I (2017) A study of the relation of mobile device attributes with the user-perceived quality of Android apps. Empir Softw Eng 22(6):3088–3116

    Article  Google Scholar 

  • Noei E, Zhang F, Zou Y (2019) Too many user-reviews, what should app developers look at first? IEEE Trans Softw Eng

  • Plisson J, Lavrac N, Mladenic D et al (2004) A rule based approach to word lemmatization. Proceedings of IS-2004:83–86

  • Preston-Werner T (2019) Semantic versioning 2.0.0. https://spectrum.ieee.org/tech-talk/telecom/internet/the-art-of-writing-app-release-notes. Accessed: 2019-08-30

  • Rousseeuw PJ (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53–65

    Article  Google Scholar 

  • Sowell F (1992) Maximum likelihood estimation of stationary univariate fractionally integrated time series models. J Econ 53(1-3):165–188

    MathSciNet  Article  Google Scholar 

  • Statista (2020) Average number of new Android app releases per day from 3rd quarter 2016 to 1st quarter 2018. https://www.statista.com/statistics/276703/android-app-releases-worldwide/. Accessed: 2020-01-23.

  • Tibshirani R, Walther G, Hastie T (2001) Estimating the number of clusters in a data set via the gap statistic. J Royal Stat Soc Ser B (Statistical Methodology) 63(2):411–423

    MathSciNet  Article  Google Scholar 

  • Toda HY, Yamamoto T (1995) Statistical inference in vector autoregressions with possibly integrated processes. J Econ 66(1-2):225–250

    MathSciNet  Article  Google Scholar 

  • Wang C, Li J, Liang P, Daneva M, Sinderen M (2019) Developers’ eyes on the changes of apps: an exploratory study on app changelogs. In: 2019 IEEE 27Th international requirements engineering conference workshops (REW). IEEE, pp 207–212

  • Yuan W, Feng Z, Chen S, Huang K, Yao J (2017) What biscuits to put in the basket? features prediction in release management for Android system. In: Proceedings of the 2017 IEEE international conference on web services (ICWS). IEEE, pp 73–80

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Aidan Z. H. Yang

  2. Department of Engineering, Thompson Rivers University, Kamloops, British Columbia, Canada

    Safwat Hassan

  3. Software Evolution and Analytics Lab (SEAL), Queen’s University, Kingston, Ontario, Canada

    Ying Zou

  4. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, Ontario, Canada

    Ahmed E. Hassan

Authors
  1. Aidan Z. H. Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Safwat Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Safwat Hassan.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Mark Harman

Appendix

Appendix

Table 13 List of refined bug-related, improvement-related, and emergency-related keywords for identifying release notes that resolve bugs, introduce new features, and provide emergency updates to the app
Full size table
Table 14 List of defined questions in the conducted survey
Full size table

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yang, A.Z.H., Hassan, S., Zou, Y. et al. An empirical study on release notes patterns of popular apps in the Google Play Store. Empir Software Eng 27, 55 (2022). https://doi.org/10.1007/s10664-021-10086-2

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10086-2

Keywords

  • Android mobile apps
  • Release notes
  • Google Play Store
  • Longitudinal study