Advertisement

Impact of Gamification on Trace Link Vetting: A Controlled Experiment

  • Salome MaroEmail author
  • Emil Sundklev
  • Carl-Oscar Persson
  • Grischa LiebelEmail author
  • Jan-Philipp Steghöfer
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11412)

Abstract

[Context] Automatically generated trace links must be vetted by human analysts before use. The task of vetting trace links is considered boring due to its repetitive nature and tools that are not engaging to the analyst. Therefore, a lack of developer engagement can hamper the successful implementation of a traceability strategy in an organisation. [Objective] In this study, we examine whether two gamification features, levels and badges, have a positive effect on human analysts’ engagement and ultimately on the quality of vetted trace links. [Method] We have conducted a controlled experiment with 24 participants that vetted trace link candidates and recorded their speed, correctness, enjoyment, and perceived usability of the tool. [Results] The results indicate that there was no significant difference between the speed, correctness, and perceived usability of the control and the experiment group. However, gamification features significantly increased the users’ perceived enjoyment. Levels and badges were perceived positively by the majority of the participants while some pitfalls and improvements were pointed out. [Conclusion] Our study indicates the need for further research as the results raise several questions, in particular w.r.t. what analyst behaviour gamification incentivises, and the impact of gamification on long-term enjoyment.

Keywords

Software engineering Gamification Traceability Traceability management 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usability Stud. 4(3), 114–123 (2009)Google Scholar
  2. 2.
    Borg, M., Runeson, P., Ardö, A.: Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability. Empir. Softw. Eng. (ESE) 19(6), 1565–1616 (2014)CrossRefGoogle Scholar
  3. 3.
    Brooke, J., et al.: SUS-a quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)Google Scholar
  4. 4.
    Charles, D., Charles, T., McNeill, M., Bustard, D., Black, M.: Game-based feedback for educational multi-user virtual environments. Br. J. Educ. Technol. 42(4), 638–654 (2011)CrossRefGoogle Scholar
  5. 5.
    Cleland-Huang, J., Czauderna, A., Gibiec, M., Emenecker, J.: A machine learning approach for tracing regulatory codes to product specific requirements. In: 32nd ACM/IEEE International Conference on Software Engineering ICSE 2010, pp. 155–164 (2010)Google Scholar
  6. 6.
    Cuddeback, D., Dekhtyar, A., Hayes, J.: Automated requirements traceability: the study of human analysts. In: RE 2010, pp. 231–240. IEEE (2010)Google Scholar
  7. 7.
    Cuddeback, D., Dekhtyar, A., Hayes, J.H., Holden, J., Kong, W.K.: Towards overcoming human analyst fallibility in the requirements tracing process. In: ICSE 2011, pp. 860–863. ACM (2011)Google Scholar
  8. 8.
    Dalpiaz, F., Snijders, R., Brinkkemper, S., Hosseini, M., Shahri, A., Ali, R.: Engaging the crowd of stakeholders in requirements engineering via gamification. In: Stieglitz, S., Lattemann, C., Robra-Bissantz, S., Zarnekow, R., Brockmann, T. (eds.) Gamification. PI, pp. 123–135. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-45557-0_9CrossRefGoogle Scholar
  9. 9.
    Deci, E.L., Koestner, R., Ryan, R.M.: A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychol. Bull. 125(6), 627 (1999)CrossRefGoogle Scholar
  10. 10.
    Dekhtyar, A., Dekhtyar, O., Holden, J., Hayes, J.H., Cuddeback, D., Kong, W.K.: On human analyst performance in assisted requirements tracing: statistical analysis. In: RE 2011, pp. 111–120. IEEE (2011)Google Scholar
  11. 11.
    Dubois, D.J., Tamburrelli, G.: Understanding gamification mechanisms for software development. In: FSE 2013, pp. 659–662. ACM (2013)Google Scholar
  12. 12.
    Eickhoff, C., Harris, C.G., de Vries, A.P., Srinivasan, P.: Quality through flow and immersion: gamifying crowdsourced relevance assessments. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 871–880. ACM (2012)Google Scholar
  13. 13.
    Falessi, D., Juristo, N., Wohlin, C., Turhan, B., Münch, J., Jedlitschka, A., Oivo, M.: Empirical software engineering experts on the use of students and professionals in experiments. Empir. Softw. Eng. 23(1), 452–489 (2018)CrossRefGoogle Scholar
  14. 14.
    García, F., Pedreira, O., Piattini, M., Cerdeira-Pena, A., Penabad, M.: A framework for gamification in software engineering. J. Syst. Softw. 132, 21–40 (2017)CrossRefGoogle Scholar
  15. 15.
    Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? – A literature review of empirical studies on gamification. In: 47th Hawaii International Conference on System Sciences (HICSS), pp. 3025–3034. IEEE (2014)Google Scholar
  16. 16.
    Hanus, M.D., Fox, J.: Assessing the effects of gamification in the classroom: a longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Comput. Educ. 80, 152–161 (2015)CrossRefGoogle Scholar
  17. 17.
    Hayes, J.H., Dekhtyar, A., Larsen, J., Guéhéneuc, Y.G.: Effective use of analysts’ effort in automated tracing. Requir. Eng. 23(1), 119–143 (2018)CrossRefGoogle Scholar
  18. 18.
    Hayes, J.H., Dekhtyar, A., Osborne, J.: Improving requirements tracing via information retrieval. In: RE 2003, pp. 138–147. IEEE (2003)Google Scholar
  19. 19.
    Hayes, J.H., Dekhtyar, A., Sundaram, S.: Text mining for software engineering: how analyst feedback impacts final results. In: ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 1–5. ACM (2005)Google Scholar
  20. 20.
    Kappen, D.L., Nacke, L.E.: The kaleidoscope of effective gamification: deconstructing gamification in business applications. In: Proceedings of the 1st International Conference on Gameful Design, Research, and Applications, pp. 119–122. ACM (2013)Google Scholar
  21. 21.
    Kifetew, F.M., et al.: Gamifying collaborative prioritization: does pointsification work? In: RE 2017, pp. 322–331. IEEE (2017)Google Scholar
  22. 22.
    Kolpondinos, M.Z.H., Glinz, M.: Behind points and levels–the influence of gamification algorithms on requirements prioritization. In: RE 2017, pp. 332–341. IEEE (2017)Google Scholar
  23. 23.
    Kong, W.K., Hayes, J.H., Dekhtyar, A., Dekhtyar, O.: Process improvement for traceability: a study of human fallibility. In: RE 2012, pp. 31–40. IEEE (2012)Google Scholar
  24. 24.
    Kong, W.K., Huffman Hayes, J., Dekhtyar, A., Holden, J.: How do we trace requirements: an initial study of analyst behavior in trace validation tasks. In: Proceedings of the 4th International Workshop on Cooperative and Human Aspects of Software Engineering, pp. 32–39. ACM (2011)Google Scholar
  25. 25.
    Lombriser, P., Dalpiaz, F., Lucassen, G., Brinkkemper, S.: Gamified requirements engineering: model and experimentation. In: Daneva, M., Pastor, O. (eds.) REFSQ 2016. LNCS, vol. 9619, pp. 171–187. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-30282-9_12CrossRefGoogle Scholar
  26. 26.
    Maro, S., Steghöfer, J.P.: Capra: a configurable and extendable traceability management tool. In: RE 2016, pp. 407–408. IEEE (2016)Google Scholar
  27. 27.
    Maro, S., Steghöfer, J.P., Huffman Hayes, J., Cleland-Huang, J., Staron, M.: Vetting automatically generated trace links: what information is useful to human analysts? In: RE 2018, pp. 52–63. IEEE (2018)Google Scholar
  28. 28.
    Maro, S., Sundklev, E., Persson, C.O., Liebel, G., Steghöfer, J.P.: Impact of gamification on trace link vetting: a controlled experiment, January 2019.  https://doi.org/10.5281/zenodo.2540646. Dataset
  29. 29.
    Mekler, E.D., Brühlmann, F., Opwis, K., Tuch, A.N.: Do points, levels and leaderboards harm intrinsic motivation?: an empirical analysis of common gamification elements. In: Proceedings of the 1st International Conference on Gameful Design, Research, and Applications. pp. 66–73. ACM (2013)Google Scholar
  30. 30.
    Morschheuser, B., Hamari, J., Werder, K., Abe, J.: How to gamify? A method for designing gamification (2017)Google Scholar
  31. 31.
    Parizi, R.M.: On the gamification of human-centric traceability tasks in software testing and coding. In: Software Engineering Research, Management and Applications (SERA), pp. 193–200. IEEE (2016)Google Scholar
  32. 32.
    Pedreira, O., García, F., Brisaboa, N., Piattini, M.: Gamification in software engineering-a systematic mapping. Inf. Softw. Technol. 57, 157–168 (2015)CrossRefGoogle Scholar
  33. 33.
    Schütze, H., Manning, C.D., Raghavan, P.: Introduction to Information Retrieval, vol. 39. Cambridge University Press, Cambridge (2008)zbMATHGoogle Scholar
  34. 34.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-29044-2CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Software Engineering DivisionChalmers | University of GothenburgGothenburgSweden
  2. 2.School of Computer ScienceReykjavik UniversityReykjavikIceland

Personalised recommendations

Cite paper

Buy options