Skip to main content

Voice of the users: an extended study of software feedback engagement

Abstract

Many software users give feedback online about the applications they use. This feedback often contains valuable requirements information that can be used to guide the effective maintenance and evolution of a software product. Yet, not all software users give online feedback. If the demographics of a user-base aren’t fairly represented, there is a danger that the needs of less vocal users won’t be considered in development. This work investigates feedback on three prominent online channels: app stores, product forums, and social media. We directly survey software users about their feedback habits, as well as what motivates and dissuades them from providing feedback online. In an initial survey of 1040 software users, we identify statistically significant differences in the demographics of users who give feedback (gender, age, etc.), and key differences in what motivates them to engage with each of the three studied channels. In a second survey of 936 software users, we identify the top reasons users don’t give feedback, including significant differences between demographic groups. We also present a detailed list of user-rated methods to encourage their feedback. This work provides meaningful context for requirements sourced from online feedback, identifying demographic groups who are underrepresented. Findings on what motivates and discourages user feedback give insight into how feedback channels and developers can increase engagement with their user-base.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. App stores comprise typical sources of apps, such as the Apple app store, or the Google Play Store, where users can provide written feedback and star ratings for apps. Product forums are websites separate from store pages and devoted to specific products or companies. Social media include outlets such as Facebook, Reddit, Instagram, and allow users to comment and share feedback without special moderation, oftentimes on dedicated company pages.

  2. https://zenodo.org/record/3674076#.XkxNFygzZPY.

  3. https://zenodo.org/record/4320164#.X9beD9gzZ3g.

  4. https://zenodo.org/record/4320182#.X9bmt9gzZ3g

References

  1. Pagano D, and Maalej W (July 2013)“User feedback in the appstore: An empirical study,” in 2013 21st IEEE international requirements engineering conference (RE), pp. 125–134

  2. Guzman E, Alkadhi R, Seyff N (Sep. 2016) “A needle in a haystack: What do twitter users say about software?” in 2016 IEEE 24th international requirements engineering conference (RE), pp. 96–105

  3. Tizard J, Wang H, Yohannes L, Blincoe K, “Can a conversation paint a picture? mining requirements in software forums,” in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019:17–27

  4. Guzman E, Alkadhi R, Seyff N (2017) An exploratory study of twitter messages about software applications. Requirements Engineering 22:387–412

  5. Guzman E, Ibrahim M, Glinz M, “A little bird told me: mining tweets for requirements and software evolution,” in, (2017) IEEE 25th international requirements engineering conference (RE). IEEE 2017:11–20

  6. Maalej W, Nabil H (Aug. 2015) “Bug report, feature request, or simply praise? on automatically classifying app reviews,” in 2015 IEEE 23rd international requirements engineering conference (RE), vol. 00, pp. 116–125

  7. Sorbo AD, Panichella S., Alexandru CV, Visaggio CA, Canfora G (May 2017)“Surf: Summarizer of user reviews feedback,” in 2017 IEEE/ACM 39th international conference on software engineering companion (ICSE-C), pp. 55–58

  8. Guzman E, Rojas AP, “Gender and user feedback: an exploratory study, in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019: 381–385

  9. Guzman E, Oliveira L, Steiner Y, Wagner LC, Glinz M (2018) “User feedback in the app store: a cross-cultural study,” in 2018 IEEE/ACM 40th international conference on software engineering: software engineering in society (ICSE-SEIS). IEEE, pp. 13–22

  10. Tizard J, Rietz T, Blincoe K (2020) “Voice of the users: A demographic study of software feedback behaviour,” in (2020) IEEE 28th international requirements engineering conference (RE). IEEE:55–65

  11. Groen EC, Seyff N, Ali R, Dalpiaz F, Doerr J, Guzman E, Hosseini M, Marco J, Oriol M, Perini A et al (2017) The crowd in requirements engineering: the landscape and challenges. IEEE Softw 34(2):44–52

    Article  Google Scholar 

  12. Johnson D, Tizard J, Damian D, Blincoe K, Clear T (2020) “Open crowdre challenges in software ecosystems,” in 2020 4th international workshop on crowd-based requirements engineering (CrowdRE), pp. 1–4

  13. Panichella S, Di Sorbo A, Guzman E, Visaggio CA, Canfora G, Gall HC (2016) “Ardoc: App reviews development oriented classifier,” in Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, ser. FSE 2016. New York, NY, USA: ACM, pp. 1023–1027. [Online]. Available: http://doi.acm.org/10.1145/2950290.2983938

  14. Chen N, Lin J, Hoi SCH, Xiao X, Zhang B (2014) “Ar-miner: Mining informative reviews for developers from mobile app marketplace,” in Proceedings of the 36th international conference on software engineering, ser. ICSE 2014. New York, NY, USA: ACM, pp. 767–778. [Online]. Available: http://doi.acm.org/10.1145/2568225.2568263

  15. Khan JA, Xie Y, Liu L, Wen L, “Analysis of requirements-related arguments in user forums,” in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019:63–74

  16. Tizard J “Requirement mining in software product forums,” in 2019 IEEE 27th international requirements engineering conference (RE), 2019, pp. 428–433

  17. Buhrmester MD, Kwang T, Gosling S (2011) Amazon’s mechanical turk. Perspec Psychol Sci 6:3–5

    Article  Google Scholar 

  18. Marder B, Gattig D, Collins E, Pitt L, Kietzmann J, Erz A (2019) The avatar’s new clothes: understanding why players purchase non-functional items in free-to-play games. Comput Human Behav 91:72–83

    Article  Google Scholar 

  19. Bleize DN, Antheunis ML (2019) Factors influencing purchase intent in virtual worlds: a review of the literature. J Market Commun 25(4):403–420

    Article  Google Scholar 

  20. Stade M, Seyff N, Baikenova A, Scherr SA, “Towards a user feedback approach for smart homes: An explorative interview study,” in, (2020) 4th international workshop on crowd-based requirements engineering (CrowdRE). IEEE 2020:5–10

  21. Papadopoulos N, Martín OM, Cleveland M, Laroche M (2011) Identity, demographics, and consumer behaviors. International Marketing Review (2011)

  22. “New Zealand census, 2018,” https://www.stats.govt.nz/information-releases/2018-census-population-and-dwelling-counts, accessed: December 2019

  23. Likert R (1932) “A technique for the measurement of attitudes.” Archives of psychology (1932)

  24. ISCED U (2012) “International standard classification of education 2011,”

  25. Galitz WO (2007) The essential guide to user interface design: an introduction to GUI design principles and techniques. Wiley, Hoboken

    Google Scholar 

  26. Turk AM (2012) “Amazon mechanical turk,” Retrieved August, vol. 17, p. 2012

  27. Guo Y, Barnes S (2007) Why people buy virtual items in virtual worlds with real money. ACM SIGMIS Database: the DATABASE for Advances in Information Systems 38(4):69–76

    Article  Google Scholar 

  28. Etikan I (2016) Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat 5(1):1

    MathSciNet  Article  Google Scholar 

  29. “Qualtrics survey platform,” https://www.qualtrics.com, accessed: December (2019)

  30. Bock O, Nicklisch A, Baetge I (2012) “hroot: Hamburg registration and organization online tool,” in WiSo-HH Working Paper Series

  31. McHugh ML (2013) The chi-square test of independence. Biochemia medica Biochemia medica 23(2):143–149

    Article  Google Scholar 

  32. Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77–101

    Article  Google Scholar 

  33. Burnett M, Stumpf S, Macbeth J, Makri S, Beckwith L, Kwan I, Peters A, Jernigan W (2016) Gendermag: a method for evaluating software’s gender inclusiveness. Interact Comput 28(6):760–787

    Article  Google Scholar 

  34. Rietz T, Maedche A, “Ladderbot: A requirements self-elicitation system,’ in, (2019) IEEE 27th international requirements engineering conference (RE). IEEE 2019: 357–362

  35. De Oliveira GF, Ferreira B, Marques AB (2020) “USARP method: Eliciting and describing USAbility Requirements with Personas and user stories,” in ACM international conference proceeding series. association for computing machinery 10: 437–446

  36. Ferreira B, Silva W, Barbosa SD, Conte T (2018) “Technique for representing requirements using personas: A controlled experiment,” IET Software, 12(3)

  37. Martens D, Maalej W (2019) Towards understanding and detecting fake reviews in app stores. Emp Softw Eng 24(6):3316–3355

    Article  Google Scholar 

Download references

Acknowledgements

The data collection in Zhejiang University was supported by the Provincial Key Research and Development Plan of Zhejiang Province, China (No. 2019C03137).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to James Tizard.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tizard, J., Rietz, T., Liu, X. et al. Voice of the users: an extended study of software feedback engagement. Requirements Eng 27, 293–315 (2022). https://doi.org/10.1007/s00766-021-00357-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00766-021-00357-1