Skip to main content
SpringerLink
Log in

A comparative study and analysis of developer communications on Slack and Gitter

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software developers are often using instant messaging platforms to communicate with each other and other stakeholders. Among these platforms, Gitter has emerged as a popular choice and the messages it contains can reveal important information to researchers studying open source software systems. Uncovering what developers are communicating about through Gitter is an essential first step towards successfully understanding and leveraging this information. In this paper, we first describe the largest manually labeled and curated dataset of Gitter developer messages, named GitterCom, obtained by manually analyzing and labeling 10,000 Gitter messages in 10 software projects. We then present a qualitative study to understand the extent to which the categories identified in previous work by Lin et al. (2016) found on Slack through surveys are applicable to developer messages exchanged on Gitter. Further, in an effort to automate the labeling process, we investigate the accuracy of 9 traditional machine learning and deep learning algorithms in predicting the intent of Gitter messages. We found that Decision Trees and Random Forest performed the best, achieving an accuracy of 88%, which is very promising for this multi-class classification task. Finally, we discuss the potential directions for future research enabled by labeled Gitter datasets such as GitterCom.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://gitter.im/

  2. https://slack.com/

  3. https://creativecommons.org/licenses/by-nc-sa/3.0/us/

  4. https://figshare.com/s/576d328da4a5b50ea155

  5. https://gitter.im/gitterHQ/home

  6. https://gitter.im/explore

  7. https://developer.gitter.im

  8. http://keras.io/

  9. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

  10. https://imbalanced-learn.readthedocs.io/en/stable/api.html

References

  • Alkadhi R, Johanssen JO, Guzman E, Bruegge B (2017a) REACT: An Approach for Capturing Rationale in Chat Messages. In: Proceedings of the 11th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM’17), pp 175–180

  • Alkadhi R, Lata T, Guzmany E, Bruegge B (2017b) Rationale in development chat messages: An exploratory study. In: Proceedings of the 14th IEEE/ACM international conference on mining software repositories (MSR’17), pp 436–446

  • Allamanis M, Sutton C (2013) Why, when, and what: Analyzing stack overflow questions by topic, type, and code. In: Proceedings of the 10th IEEE working conference on mining software repositories (MSR’13), pp 53–56

  • Aly M (2005) Survey on multiclass classification methods. Neural Network 19:1–9

    Google Scholar 

  • Anders A (2016) Team communication platforms and emergent social collaboration practices. Int J Business Commun 53(2):224–261

    Article  Google Scholar 

  • Arora P, Ganguly D, Jones GJF (2015) The good, the bad and their Kins: Identifying questions with negative scores in StackOverflow. In: Proceedings of the 2nd IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM’15), pp 1232–1239

  • Bergstra J, Bengio Y (2012) Random Search for Hyper-Parameter Optimization. J Mach Learn Res 13(2):281–305

    MathSciNet  MATH  Google Scholar 

  • Beyer S, Macho C, Pinzger M, Di Penta M (2018) Automatically classifying posts into question categories on stack overflow. In: Proceedings of the 26th IEEE international conference on program comprehension (ICPC’18), Association for Computing Machinery, New York, NY, USA, ICPC ’18, pp 211–221 https://doi.org/10.1145/3196321.3196333

  • Bottou L (2010) Large-Scale Machine Learning with Stochastic Gradient Descent. In: Lechevallier Y, Saporta G (eds) Proceedings of the 19th international conference on computational statistics (COMPSTAT’10), pp 177–186

  • Breiman L (2001) Random Forests. Mach Learn 45(1):5–32

    Article  Google Scholar 

  • Buda M, Maki A, Mazurowski MA (2018) A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks. Neural Networks 106:249–259

    Article  Google Scholar 

  • Chatterjee P, Damevski K, Pollock L, Augustine V, Kraft NA (2019) Exploratory study of slack Q&A chats as a mining source for software engineering tools. In: Proceedings of the 16th IEEE international conference on mining software repositories (MSR’19), pp 490–501

  • Chatterjee P, Damevski K, Kraft NA, Pollock L (2020) Software-related slack chats with disentangled conversations. In: Proceedings of the 17th IEEE international conference on mining software repositories (MSR’20), pp 588–592

  • Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: Synthetic Minority Over-sampling Technique. J Artif Intell Res 16(1):321–357

    Article  Google Scholar 

  • Chowdhury SA, Hindle A (2015) Mining StackOverflow to filter out Off-topic IRC discussion. In: Proceedings of the 12th IEEE working conference on mining software repositories (MSR’15), pp 422–425

  • Constantino K, Zhou S, Souza M, Figueiredo E, Kastner C (2020) Understanding collaborative software development: An interview study. In: Proceedings of the 15th ACM/IEEE international conference on global software engineering (ICGSE’20), pp 55–65

  • Cortes C, Vapnik V (1995) Support-vector Networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  • cuc (2020) Cucumber. https://github.com/cucumber/cucumber

  • Deng L (2014) A Tutorial Survey of Architectures, Algorithms, and Applications for Deep Learning. Transactions on Signal and Information Processing 3

  • Efstathiou V, Chatzilenas C, Spinellis D (2018) Word embeddings for the software engineering domain. In: Proceedings of the 15th IEEE international conference on mining software repositories (MSR’18), MSR ’18, p 38–41 DOI: https://doi.org/10.1145/3196398.3196448

  • Ehsan O, Hassan S, Mezouar ME, Zou Y (2020) An Empirical Study of Developer Discussions in the Gitter Platform. TOSEM pp 1–39

  • Elsner M, Charniak E (2011) Disentangling chat with local coherence models. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies (ACL’11), pp 1179–1189

  • Fang H, Klug D, Lamba H, Herbsleb J, Vasilescu B (2020) Need for tweet: How open source developers Talk about their GitHub work on twitter. In: Proceedings of the 17th IEEE international conference on mining software repositories (MSR’20), pp 322–326

  • Ford D, Lustig K, Banks J, Parnin C (2018) “We Don’t Do That Here”: How collaborative editing with mentors improves engagement in social Q&A communities. In: Proceedings of the 2018 conference on human factors in computing systems (CHI’18)’, pp 1–12

  • fre (2020) Freezingmoon. https://github.com/FreezingMoon

  • Gers FA, Schraudolph NN, Schmidhuber J (2002) Learning Precise Timing with LSTM Recurrent Networks. J Mach Learn Res 3(1):115–143

    MathSciNet  MATH  Google Scholar 

  • Gomes Pimentel M, Fuks H, de Lucena CJP (2003) Co-text loss in textual chat tools. In: Procceedings of the 4th international and interdisciplinary conference on modeling and using context (CONTEXT’03), pp 483–490

  • Guzman E, Ibrahim M, Glinz M (2017) A little bird told me: Mining tweets for requirements and software evolution. In: Proceedings of the 25th IEEE international requirements engineering conference (RE’17), pp 11–20

  • Hastie T, Rosset S, Zhu J, Zou H (2009) Multi-class AdaBoost. Statistics and Its. Interface 2(3):349–360

    MathSciNet  MATH  Google Scholar 

  • ima (2020) Imagej. https://github.com/imagej/imagej

  • jhi (2020) jhipster. https://github.com/jhipster/jhipster/

  • jsp (2020) jspm. https://github.com/jspm

  • Käfer V, Graziotin D, Bogicevic I, Wagner S, Ramadani J (2018) Communication in Open-Source Projects-End of the E-mail Era? In: Proceedings of the 40th IEEE/ACM international conference on software engineering(ICSE’18), pp 242–243

  • Keivanloo I, Rilling J, Zou Y (2014) Spotting working code examples. In: Proceedings of the 36th IEEE international conference on software engineering (ICSE’14), pp 664–675

  • Khan A, Baharudin B, Lee LH, Khan K (2010) A Review of Machine Learning Algorithms for Text-Documents Classification. J Adv Inform Technol 1(1):4–20

    Google Scholar 

  • Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the 11th SIGDAT conference on empirical methods in natural language processing (EMNLP’14), pp 1746–1751

  • Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Proceedings of the 29th AAAI conference for artificial intelligence (AAAI’15), pp 2267–2273

  • Laurikkala J (2001) Improving identification of difficult small classes by balancing class distribution. In: Conference in artificial intelligence in medicine in Europe (AIME’01), Lecture Notes in Computer Science, pp 63–66

  • Lin B, Zagalsky A, Storey MA, Serebrenik A (2016) Why developers are slacking off: Understanding how software teams use slack. In: Proceedings of the 19th ACM conference on computer supported cooperative work and social computing (CSCW’16), pp 333–336

  • Linares-Vasquez M, Dit B, Poshyvanyk D (2013) An exploratory analysis of mobile development issues using stack overflow. In: Proceedings of the 10th IEEE working conference on mining software repositories (MSR’13), pp 93–96

  • Liu F, Liu Y (2008) Correlation between ROUGE and human evaluation of extractive meeting summaries. In: Proceedings of the 46th ACL annual meeting of the association for computational linguistics on human language technologies (HTL’08), pp 201–204

  • mar (2020) Marionette. https://github.com/marionettejs/backbone.marionette

  • McCallum A, Nigam K (1998) A comparison of event models for naïve bayes text classification. In: Proceedings of the 1st AAAI workshop on learning for text categorization (ICML/AAAI’98), pp 41–48

  • Mikolov T, Deoras A, Povey D, Burget L, Cernocký J (2011) Strategies for training large scale neural network language models. In: Proceedings of the 12th IEEE workshop on automatic speech recognition understanding (ASRU’11), pp 196–201

  • Murgia A, Janssens D, Demeyer S, Vasilescu B (2016) Among the machines: Human-Bot interaction on social Q&A websites. In: Proceedings of the 2016 conference extended abstracts on human factors in computing systems (CHI/EA’16), pp 1272–1279

  • Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example?: A study of programming Q&A in StackOverflow. In: Proceedings of the 28th IEEE international conference on software maintenance (ICSM’12), pp 25–34

  • Novielli N, Calefato F, Lanubile F (2014) Towards discovering the role of emotions in stack overflow. In: Proceedings of the 6th international workshop on social software engineering, SSE’2014, pp 33–36

  • Novielli N, Calefato F, Lanubile F (2015) The challenges of sentiment detection in the social programmer ecosystem. In: Proceedings of the 7th international workshop on social software engineering (SSE’15), pp 33–40

  • Paikari E, van der Hoek A (2018) A framework for understanding chatbots and their future. In: Proceedings of the 11th international workshop on cooperative and human aspects of software engineering (CHASE’18), pp 13–16

  • Panichella S, Di Sorbo A, Guzman E, Visaggio C, Canfora G, Gall H (2015) How can I improve my App? Classifying user reviews for software maintenance and evolution. In: Proceedings of the 31st IEEE international conference on software maintenance and evolution (ICSME’15), pp 281–290

  • Parra E (2020) Gittercom, dataset. https://doi.org/10.6084/m9.figshare.11626008

  • Parra E (2021) Replication package. https://figshare.com/s/576d328da4a5b50ea155

  • Parra E, Ellis A, Haiduc S (2020) GitterCom - A dataset of open source developer communications in gitter. In: Proceedings of the 17th IEEE international conference on mining software repositories (MSR’20), pp 563–567

  • Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Empirical methods in natural language processing (EMNLP), pp 1532–1543

  • Poché EH (2017) Analyzing User Comments On YouTube Coding Tutorial Videos. mathesis, Louisiana State University, Baton Rouge, LA, USA

  • Ponzanelli L, Mocci A, Bacchelli A, Lanza M (2014) Understanding and classifying the quality of technical forum questions. In: Proceedings of the 14th international conference on quality software (QSIC’14), pp 343–352

  • Rekha S, Divya N, Bagavathi S (2014) A hybrid auto-tagging system for StackOverflow forum questions. In: Proceedings of the 1st international conference on interdisciplinary advances in applied computing (ICONIAAC’14), pp 1–5

  • Russell SJ, Norvig P (1995) Artificial Intelligence: A Modern Approach

  • Safavian SR, Landgrebe D (1991) A survey of Decision Tree Classifier Methodology. IEEE Trans Syst Man Cybern 21(3):660–674

    Article  MathSciNet  Google Scholar 

  • Sahar H, Hindle A, Bezemer CP (2020) How are Issue Reports Discussed in Gitter Chat Rooms? Journal of Systems and Software pp 110852, https://doi.org/10.1016/j.jss.2020.110852., http://www.sciencedirect.com/science/article/pii/S0164121220302429

  • Sajedi Badashian A, Hindle A, Stroulia E (2016) Crowdsourced bug triaging: Leveraging Q&A platforms for bug assignment. In: fundamental approaches to software engineering, lecture notes in computer science, pp 231–248

  • sci (2020) scikit-learn. https://github.com/scikit-learn/scikit-learn

  • Seiffert C, Khoshgoftaar TM, Van Hulse J, Folleco A (2014) An Empirical Study of the Classification Performance of Learners on Imbalanced and Noisy Software Quality Data. Inform Sci 259(1):571–595

    Article  Google Scholar 

  • Shi L, Chen X, Yang Y, Jiang H, Jiang Z, Niu N, Wang Q (2021) A first look at developers’ live chat on gitter. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering (ESEC/FSE’21), Association for Computing Machinery, pp 391–403 https://doi.org/10.1145/3468264.3468562

  • Shihab E, Jiang ZM, Hassan AE (2009) Studying the use of developer IRC meetings in open source projects. In: Proceedings of the IEEE international conference on software maintenance (ICSM’09), pp 147–156

  • Singh A, Thakur N, Sharma A (2016) A review of supervised machine learning algorithms. In: Proceedings of the 3rd IEEE international conference on computing for sustainable global development (INDIACom’16), pp 1310–1315

  • Storey M, Zagalsky A, Filho FF, Singer L, German DM (2017) How Social and Communication Channels Shape and Challenge a Participatory Culture in Software Development. IEEE Trans Softw Eng 43(2):185–204

    Article  Google Scholar 

  • Storey MA, Zagalsky A (2016) Disrupting developer productivity one bot at a time. In: Proceedings of the 24th ACM/SIGSOFT international symposium on foundations of software engineering (FSE’16), pp 928–931

  • Storey MA, Singer L, Cleary B, Figueira Filho F, Zagalsky A (2014) The (R) Evolution of social media in software engineering. In: Proceedings of the 36th ACM/IEEE international conference in software engineering, future of software engineering (FOSE’14), pp 100–116

  • Stray V, Moe NB, Noroozi M (2019) Slack me if you can!: Using enterprise social networking tools in virtual agile teams. In: Proceedings of the 14th IEEE international conference on global software engineering (ICGSE’19), pp 101–111

  • Subramanian S, Holmes R (2013) Making sense of online code snippets. In: Proceedings of the 10th IEEE working conference on mining software repositories (MSR’13), pp 85–88

  • thw (2020) Theholywaffle. https://github.com/TheHolyWaffle

  • Tian Y, Lo D, Lawall J (2014) Automated construction of a Software-Specific word similarity database. In: Proceedings of the 1st joint meeting of The IEEE conference on software maintenance, reengineering, and reverse engineering (CSMR-WCRE’04), pp 44–53

  • Tomek I (1976) Two Modifications of CNN. IEEE Transactions on Systems, Man, and Cybernetics SMC–6(11):769–772

    MathSciNet  MATH  Google Scholar 

  • Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the Web? In: Proceedings of the 33rd IEEE/ACM international conference on software engineering (ICSE’11), pp 804–807

  • uik (2020) uikit. https://github.com/uikit/uikit

  • Vassallo C, Panichella S, Di Penta M, Canfora G (2014) CODES: Mining source code descriptions from developers discussions. In: Proceedings of the 22nd IEEE international conference on program comprehension (ICPC’14), pp 106–109

  • West D (2000) Neural Network Credit Scoring Models. Computers & Operations Research 27(11):1131–1152

    Article  Google Scholar 

  • xen (2020) Xenko3d. https://gitter.im/xenko3d/xenko

  • Xu B, Xing Z, Xia X, Lo D (2017) Answerbot: Automated generation of answer summary to developers’ technical questions. In: Proceedings of the 32nd IEEE/ACM international conference on automated software engineering (ASE’17), pp 706–716

Download references

Author information

Authors and Affiliations

  1. Florida State University, 600 W College Ave, 32306, Tallahassee, FL, USA

    Esteban Parra, Ashley Ellis & Sonia Haiduc

  2. Department of Software Engineering, College of Computer Science and Engineering, University of Jeddah, Jeddah, Saudi Arabia

    Mohammad Alahmadi

Authors
  1. Esteban Parra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Alahmadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ashley Ellis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sonia Haiduc
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Esteban Parra.

Additional information

Communicated by : Georgios Gousios and Sarah Nadi.

This work is supported in part by the National Science Foundation grant CCF-1526929.

This article belongs to the Topical Collection: Mining Software Repositories (MSR)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Parra, E., Alahmadi, M., Ellis, A. et al. A comparative study and analysis of developer communications on Slack and Gitter. Empir Software Eng 27, 40 (2022). https://doi.org/10.1007/s10664-021-10095-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10095-1

Keywords

Search

Navigation