Empirical Software Engineering

, Volume 23, Issue 1, pp 521–564 | Cite as

An exploratory qualitative and quantitative analysis of emotions in issue report comments of open source systems

  • Alessandro MurgiaEmail author
  • Marco Ortu
  • Parastou Tourani
  • Bram Adams
  • Serge Demeyer


Software development—just like any other human collaboration—inevitably evokes emotions like joy or sadness, which are known to affect the group dynamics within a team. Today, little is known about those individual emotions and whether they can be discerned at all in the development artifacts produced during a project. This paper analyzes (a) whether issue reports—a common development artifact, rich in content—convey emotional information and (b) whether humans agree on the presence of these emotions. From the analysis of the issue comments of 117 projects of the Apache Software Foundation, we find that developers express emotions (in particular gratitude, joy and sadness). However, the more context is provided about an issue report, the more human raters start to doubt and nuance their interpretation. Based on these results, we demonstrate the feasibility of a machine learning classifier for identifying issue comments containing gratitude, joy and sadness. Such a classifier, using emotion-driving words and technical terms, obtains a good precision and recall for identifying the emotion love, while for joy and sadness a lower recall is obtained.


Emotion mining Issue report Text analysis Parrott‘s framework 



This work was sponsored by (a) the Institute for the Promotion of Innovation through Science and Technology in Flanders by means of a project entitled Change-centric Quality Assurance (CHAQ) with number 120028, as well as (b) the Regione Autonoma della Sardegna (RAS), Regional Law No. 7-2007, project CRP-17938, “LEAN 2.0”.


  1. Ahmed T, Srivastava A (2017) Understanding and evaluating the behavior of technical users. a study of developer interaction at stackoverflow. Human-centric Computing and Information Sciences 7(1):8CrossRefGoogle Scholar
  2. Amabile T M, Barsade S G, Mueller J S, Staw B M (2005) Affect and creativity at work. Adm Sci Q 50(3):367–403. doi: 10.2307/30037208 CrossRefGoogle Scholar
  3. Aman S, Szpakowicz S (2007) Identifying expressions of emotion in text 10th international conference on text, speech and dialogue (TSD). Springer, pp 196–205Google Scholar
  4. Ambler S (2002) Agile modeling: effective practices for extreme programming and the unified process. Wiley, New YorkGoogle Scholar
  5. Bacchelli A, Lanza M, Robbes R (2010) Linking e-mails and source code artifacts Proceedings of the international conference on software engineering (ICSE), pp 375–384Google Scholar
  6. Bacchelli A, Sasso TD, D’Ambros M, Lanza M (2012) Content classification of development emails Proceedings of the international conference on software engineering (ICSE), pp 375–385Google Scholar
  7. Balabantaray R, Mohammad M, Sharma N (2012) Multi-class twitter emotion classification: a new approach. International Journal of Applied Information Systems 4 (1):48–53CrossRefGoogle Scholar
  8. Bazelli B, Hindle A, Stroulia E (2013) On the personality traits of stackoverflow users International conference on software maintenance (ICSM). doi: 10.1109/ICSM.2013.72, pp 460–463Google Scholar
  9. Bollen J, Mao H, Zeng X (2011) Twitter mood predicts the stock market. Journal of Computational Science 2(1):1–8CrossRefGoogle Scholar
  10. Brodkin J (2013) Linus torvalds defends his right to shame linux kernel developers.
  11. Brooks FP Jr (1987) No silver bullet essence and accidents of software engineering. Computer 20(4):10–19CrossRefGoogle Scholar
  12. Campbell DT, Stanley JC (1963) Experimental and quasi-experimental designs for generalized causal inference. Houghton MifflinGoogle Scholar
  13. Cataldi M, Ballatore A, Tiddi I, Aufaure M A (2013) Good location, terrible food: detecting feature sentiment in user-generated reviews. Social Netw Analys Mining 3(4):1149–1163CrossRefGoogle Scholar
  14. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46MathSciNetCrossRefGoogle Scholar
  15. Das S R, Chen M Y (2007) Yahoo! for amazon: sentiment extraction from small talk on the web. Manag Sci 53(9):1375–1388. CrossRefGoogle Scholar
  16. De Choudhury M, Counts S (2013) Understanding affect in the workplace via social media Proceedings of the conference on computer supported cooperative work. ACM, New York. doi: 10.1145/2441776.2441812, pp 303–316Google Scholar
  17. DeMarco T, Lister T (1999) Peopleware: productive projects and teams, 2nd edn. Dorset House Publishing Co. Inc, New YorkGoogle Scholar
  18. Destefanis G, Marco O, Steve C, Steve S, Michele M, Roberto T (2016) Software development: do good manners matter? PeerJ Comp Sci 2:e73. doi: 10.7717/peerj-cs.73
  19. Elfenbein H A, Ambady N (2002) On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychol Bull 128(2):203CrossRefGoogle Scholar
  20. Feldman R (2013) Techniques and applications for sentiment analysis. Commun ACM 56(4):82–89. doi: 10.1145/2436256.2436274 CrossRefGoogle Scholar
  21. Fleiss J L (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378CrossRefGoogle Scholar
  22. Fowler J H, Christakis N A (2008) Dynamic spread of happiness in a large social network: longitudinal analysis over 20 years in the framingham heart study. BMJ 337. doi: 10.1136/bmj.a2338
  23. Fredrickson B L (2001) The role of positive emotions in positive psychology: the broaden-and-build theory of positive emotions. Am Psychol 56(3):218CrossRefGoogle Scholar
  24. Fritz T, Müller S (2016) Leveraging biometric data to boost software developer productivity International conference on software analysis, evolution and reengeneering (future of software engineering track), s.nGoogle Scholar
  25. Gold J (2015) A prominent linux kernel developer is stepping down from her direct work in the kernel community.
  26. Graziotin D, Wang X, Abrahamsson P (2014) Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ e289. doi: 10.7717/peerj.289
  27. Guillory J, Spiegel J, Drislane M, Weiss B, Donner W, Hancock J (2011) Upset now?: emotion contagion in distributed groups Proceedings of the conference on human factors in computing systems (CHI), pp 745–748Google Scholar
  28. Guzman E, Bruegge B (2013) Towards emotional awareness in software development teams Proceedings of the joint meeting on foundations of software engineering (ESEC/FSE), pp 671–674Google Scholar
  29. Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in github: an empirical study Proceedings of the working conference on mining software repositories (MSR). ACM, New York, MSR 2014. doi: 10.1145/2597073.2597118, pp 352–355
  30. Guzzi A, Bacchelli A, Lanza M, Pinzger M, van Deursen A (2013) Communication in open source software development mailing lists Proceedings of the working conference on mining software repositories (MSR), pp 277–286Google Scholar
  31. Hancock JT, Gee K, Ciaccio K, Lin JMH (2008) I’m sad you’re sad: emotional contagion in CMC Proceedings of the 2008 ACM conference on computer supported cooperative work (CSCW), pp 295–298Google Scholar
  32. Heritage Dictionary A (2005) The american heritage science dictionary.
  33. Hu M, Liu B (2004) Mining and summarizing customer reviews Proceedings of the international conference on knowledge discovery and data mining, ACM, New York, KDD ’04. doi: 10.1145/1014052.1014073, pp 168–177
  34. Jongeling R, Datta S, Serebrenik A (2015) (2015) Choosing Your weapons: On sentiment analysis tools for software engineering research IEEE international conference on software maintenance and evolution (ICSME)Google Scholar
  35. Mäntylä M, Adams B, Destefanis G, Graziotin D, Ortu M (2016) Mining valence, arousal, and dominance: possibilities for detecting burnout and productivity? Proceedings of the 13th international workshop on mining software repositories, ACM, pp 247–258Google Scholar
  36. Mitchell T M (1997) Machine learning, 1st edn. McGraw-Hill Inc, New YorkzbMATHGoogle Scholar
  37. Murgia A, Tourani P, Adams B, Ortu M (2014) Do developers feel emotions? An exploratory analysis of emotions in software artifacts Proceedings of the working conference on mining software repositories (MSR). ACM, pp 262–271Google Scholar
  38. Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research Proceedings of the 2013 9th joint meeting on foundations of software engineering, ACM, New York, ESEC/FSE 2013. doi: 10.1145/2491411.2491415, pp 466–476
  39. Ortu M, Adams B, Destefanis G, Tourani P, Marchesi M, Tonelli R (2015a) Are bullies more productive? Empirical study of affectiveness vs. issue fixing time Proceedings of the working conference on mining software repositories (MSR). Florence, ItalyGoogle Scholar
  40. Ortu M, Destefanis G, Kassab M, Counsell S, Marchesi M, Tonelli R (2015b) Would you mind fixing this issue? International conference on agile software development. Springer, pp 129–140Google Scholar
  41. Ortu M, Destefanis G, Counsell S, Swift S, Tonelli R, Marchesi M (2016a) Arsonists or firefighters? Affectiveness in agile software development International conference on agile software development. Springer, pp 144–155Google Scholar
  42. Ortu M, Murgia A, Destefanis G, Tourani P, Tonelli R, Marchesi M, Adams B (2016b) The emotional side of software developers in jira Proceedings of the 13th international conference on mining software repositories, ACM, New York, MSR ’16. doi: 10.1145/2901739.2903505, pp 480–483
  43. Pak A, Paroubek P (2010) Twitter as a corpus for sentiment analysis and opinion mining. In: Chair N C C, Choukri K, Maegaard B, Mariani J, Odijk J, Piperidis S, Rosner M, Tapias D (eds) Proceedings of the international conference on language resources and evaluation (LREC), European language resources association (ELRA). Valletta, MaltaGoogle Scholar
  44. Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends Inf Retr 2(1-2):1–135CrossRefGoogle Scholar
  45. Parrott W (2001) Emotions in social psychology. Psychology PressGoogle Scholar
  46. Piller C (1999) Everyone is a critic in cyberspace. Los Angeles Times 3(12):A1Google Scholar
  47. Plutchik R (2001) The nature of emotions human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am Sci 89(4):344–350CrossRefGoogle Scholar
  48. Rigby PC, Hassan AE (2007) What can OSS mailing lists tell us? a preliminary psychometric text analysis of the apache developer mailing list Proceedings of the working conference on mining software repositories (MSR), p 23Google Scholar
  49. Robinson M D (2004) Personality as performance categorization tendencies and their correlates. Curr Dir Psychol Sci 13(3):127–129CrossRefGoogle Scholar
  50. Sehgal V, Song C (2007) Sops: stock prediction using web sentiment Proceedings of the international conference on data mining workshops (ICDMW). IEEE Computer Society, Washington, DC, pp 21–26Google Scholar
  51. semotion (2016) The first international workshop on emotion awareness in software engineering, ICSE 2016, Workshop, Austin, Texas (USA)Google Scholar
  52. Shivhare S N, Khethawat S (2012) Emotion detection from text. Computer Science, Engineering and ApplicationsGoogle Scholar
  53. Strapparava C, Valitutti A, et al. (2004) Wordnet affect: an affective extension of wordnet LREC, vol 4, pp 1083–1086Google Scholar
  54. Tepperman J, Traum D, Narayanan SS (2006) “Yeah right”: sarcasm recognition for spoken dialogue systems Proceedings of interspeech, pp 1838–1841Google Scholar
  55. Tourani P, Adams B (2016) The impact of human discussions on just-in-time quality assurance Proceedings of the 23rd IEEE international conference on software analysis, evolution, and reengineering (SANER). Osaka, Japan, pp 189–200Google Scholar
  56. Tourani P, Jiang Y, Adams B (2014) Monitoring sentiment in open source mailing lists — exploratory study on the apache ecosystem Proceedings of the 2014 conference of the center for advanced studies on collaborative research (CASCON). Toronto, ON, Canada, pp 34–44Google Scholar
  57. Witten IH, Frank E, Hall MA (2011) Data mining: practical machine learning tools and techniques, 3rd edn. Morgan KaufmannGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.University of AntwerpAntwerpBelgium
  2. 2.University of CagliariCagliariItaly
  3. 3.MCISPolytechnique MontréalMontréalCanada
  4. 4.AntwerpenBelgium

Personalised recommendations