Skip to main content
SpringerLink
Log in

Can Source Code Analysis Indicate Programming Skills? A Survey with Developers

  • Conference paper
  • First Online:
Quality of Information and Communications Technology (QUATIC 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1621))

Abstract

Background: Both open-source and proprietary software systems have become increasingly complex. Despite their growing complexity and increasing size, software systems must satisfy strict release requirements that impose quality, putting significant pressure on developers. Therefore, software projects’ success depends on the identification and hiring of qualified developers. Several approaches aim to address this problem by automatically proposing models and tools to automatically identify programming skills through source code. However, we still lack empirical knowledge on the applicability of these models in practice. Aims: Our goal is to evaluate and compare two models proposed to support programming skill identification. Method: This paper presents a survey with 110 developers from GitHub. This survey was conducted to evaluate the applicability of two models for computing programming skills of developers based on the metrics Changed Files and Changed Lines of Code. Results: Based on the survey results, we conclude that both models often fail to identify the developer’s programming skills. Concerning precision, the Changed Files model obtained 54% to identify programming languages, 53% for back-end & front-end profiles, and 45% for testing skills. About the Changed Lines of Code model, we obtained 36% of precision to identify programming languages, 45% for back-end & front-end profiles, and 30% for testing. Conclusion: Practitioners can use our survey to refine the practical evaluation of professional skills for several purposes, from hiring procedures to the evaluation of team.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/trending/developers.

  2. 2.

    https://madnight.github.io/githut/#/pull_requests/2021/4.

  3. 3.

    https://githut.info/.

  4. 4.

    https://pypi.org/project/ghcloneall/.

  5. 5.

    https://www.google.com/forms/.

References

  1. Baltes, S., Diehl, S.: Towards a theory of software development expertise, pp. 187–200 (2018)

    Google Scholar 

  2. Basili, V., Caldiera, G., Rombach, H.D.: The goal question metric approach. Online Technical Report (1994)

    Google Scholar 

  3. Bizer, C., Heath, T., Berners-Lee, T.: Linked data: the story so far. In: Semantic Services, Interoperability and Web Apps: Emerging Concepts, pp. 205–227 (2011)

    Google Scholar 

  4. Constantino, K., Zhou, S., Souza, M., Figueiredo, E., Kästner, C.: Understanding collaborative software development: an interview study. In: Proceedings of the 15th International Conference on Global Software Engineering, pp. 55–65 (2020)

    Google Scholar 

  5. Constantinou, E., Kapitsaki, G.M.: Identifying developers’ expertise in social coding platforms. In: 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) (2016)

    Google Scholar 

  6. Easterbrook, S., Singer, J., Storey, MA., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds) Guide to Advanced Empirical Software Engineering. Springer, London (2008). https://doi.org/10.1007/978-1-84800-044-5_11

  7. Greene, G.J., Fischer, B.: CVExplorer: identifying candidate developers by mining and exploring their open source contributions. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 804–809 (2016)

    Google Scholar 

  8. Hauff, C., Gousios, G.: Matching GitHub developer profiles to job advertisements. In: IEEE/ACM 12th Working Conference on Mining Software Repositories (MSR), pp. 362–366 (2015)

    Google Scholar 

  9. Huang, W., Mo, W., Shen, B., Yang, Y., Li, N.: CPDScorer: modeling and evaluating developer programming ability across software communities. In: International Conference on Software Engineering and Knowledge Engineering (ICISDM) (2016)

    Google Scholar 

  10. Kitchenham, B.A., Pfleeger, S.L.: Personal opinion surveys. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds) Guide to Advanced Empirical Software Engineering. Springer, London (2008). https://doi.org/10.1007/978-1-84800-044-5_3

  11. Kourtzanidis, S., Chatzigeorgiou, A., Ampatzoglou, A.: RepoSkillMiner: identifying software expertise from GitHub repositories using natural language processing. In: 35th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1353–1357 (2020)

    Google Scholar 

  12. Marlow, J., Dabbish, L.: Activity traces and signals in software developer recruitment and hiring. In: Proceedings of the 2013 Conference on Computer Supported Cooperative Work, pp. 145–156 (2013)

    Google Scholar 

  13. Matturro, G.: Soft skills in software engineering: a study of its demand by software companies in Uruguay. In: 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pp. 1–10 (2013)

    Google Scholar 

  14. Mockus, A., Herbsleb, J.D.: Expertise browser: a quantitative approach to identifying expertise. In: Proceedings of the 24th International Conference on Software Engineering (ICSE), pp. 503–512 (2002)

    Google Scholar 

  15. Montandon, J.E., Silva, L.L., Valente, M.T.: Identifying experts in software libraries and frameworks among GitHub users. In: IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp. 276–287 (2019)

    Google Scholar 

  16. Mori, A., et al.: Evaluating domain-specific metric thresholds: an empirical study. In: Proceedings of the 2018 International Conference on Technical Debt, pp. 41–50. TechDebt 2018, Association for Computing Machinery, NY (2018)

    Google Scholar 

  17. Oliveira, J., Viggiato, M., Figueiredo, E.: How well do you know this library? Mining experts from source code analysis. In: Proceedings of the XVIII Brazilian Symposium on Software Quality, pp. 49–58. SBQS 2019, Association for Computing Machinery, NY (2019)

    Google Scholar 

  18. Pfleeger, S.L., Kitchenham, B.A.: Principles of survey research: part 1: turning lemons into lemonade. SIGSOFT Softw. Eng. Notes 26, 16–18 (2001)

    Article  Google Scholar 

  19. Rosner, B., Glynn, R.J., Lee, M.L.: The Wilcoxon signed rank test for paired comparisons of clustered data. Biometrics 62, 185–192 (2006)

    Article  MathSciNet  Google Scholar 

  20. Saxena, R., Pedanekar, N.: I know what you coded last summer: mining candidate expertise from GitHub repositories. In: 17th Conference on Computer Supported Cooperative Work and Social Computing (CSCW), pp. 299–302 (2017)

    Google Scholar 

  21. da Silva, J.R., Clua, E., Murta, L., Sarma, A.: Niche vs. breadth: calculating expertise over time through a fine-grained analysis. In: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER) (2015)

    Google Scholar 

  22. Singer, L., Filho, F.F., Cleary, B., Treude, C., Storey, M.A., Schneider, K.: Mutual assessment in the social programmer ecosystem: an empirical investigation of developer profile aggregators. In: 13th Conference on Computer Supported Cooperative Work (CSCW), pp. 103–116 (2013)

    Google Scholar 

  23. Stol, K.J., Ralph, P., Fitzgerald, B.: Grounded theory in software engineering research: a critical review and guidelines. In: 38th International Conference on Software Engineering (ICSE), pp. 120–131 (2016)

    Google Scholar 

  24. Tantisuwankul, J., et al.: A topological analysis of communication channels for knowledge sharing in contemporary GitHub projects. J. Syst. Softw. 158, 110416 (2019)

    Article  Google Scholar 

  25. Teyton, C., Palyart, M., Falleri, J.R., Morandat, F., Blanc, X.: Automatic extraction of developer expertise. In: 18th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 1–10 (2014)

    Google Scholar 

  26. Vadlamani, S.L., Baysal, O.: Studying software developer expertise and contributions in stack overflow and GitHub. In: 36th International Conference on Software Maintenance and Evolution (ICSME), pp. 312–323 (2020)

    Google Scholar 

Download references

Acknowledgments

This research was partially supported by Brazilian funding agencies: CNPq, CAPES, and FAPEMIG.

Author information

Authors and Affiliations

  1. Department of Computer Science, Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil

    Johnatan Oliveira & Eduardo Figueiredo

  2. Department of Computer Science, Federal University of Lavras (UFLA), Lavras, Brazil

    Maurício Souza, Matheus Flauzino & Rafael Durelli

Authors
  1. Johnatan Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maurício Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matheus Flauzino
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rafael Durelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eduardo Figueiredo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Johnatan Oliveira .

Editor information

Editors and Affiliations

  1. University of Malaga, Málaga, Spain

    Antonio Vallecillo

  2. Leiden University, Leiden, The Netherlands

    Joost Visser

  3. University of Castila-La Mancha, Ciudad Real, Spain

    Ricardo Pérez-Castillo

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Oliveira, J., Souza, M., Flauzino, M., Durelli, R., Figueiredo, E. (2022). Can Source Code Analysis Indicate Programming Skills? A Survey with Developers. In: Vallecillo, A., Visser, J., Pérez-Castillo, R. (eds) Quality of Information and Communications Technology. QUATIC 2022. Communications in Computer and Information Science, vol 1621. Springer, Cham. https://doi.org/10.1007/978-3-031-14179-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14179-9_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14178-2

  • Online ISBN: 978-3-031-14179-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation