Skip to main content
SpringerLink
Log in

Code and commit metrics of developer productivity: a study on team leaders perceptions

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Context

Developer productivity is essential to the success of software development organizations. Team leaders use developer productivity information for managing tasks in a software project. Developer productivity metrics can be computed from software repositories data to support leaders’ decisions. We can classify these metrics in code-based metrics, which rely on the amount of produced code, and commit-based metrics, which rely on commit activity. Although metrics can assist a leader, organizations usually neglect their usage and end up sticking to the leaders’ subjective perceptions only.

Objective

We aim to understand whether productivity metrics can complement the leaders’ perceptions. We also aim to capture leaders’ impressions about relevance and adoption of productivity metrics in practice.

Method

This paper presents a multi-case empirical study performed in two organizations active for more than 18 years. Eight leaders of nine projects have ranked the developers of their teams by productivity. We quantitatively assessed the correlation of leaders’ rankings versus metric-based rankings. As a complement, we interviewed leaders for qualitatively understanding the leaders’ impressions about relevance and adoption of productivity metrics given the computed correlations.

Results

Our quantitative data suggest a greater correlation of the leaders’ perceptions with code-based metrics when compared to commit-based metrics. Our qualitative data reveal that leaders have positive impressions of code-based metrics and potentially would adopt them.

Conclusions

Data triangulation of productivity metrics and leaders’ perceptions can strengthen the organization conviction about productive developers and can reveal productive developers not yet perceived by team leaders and probably underestimated in the organization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Plots of the joint distributions between calculated metric values and rankings (metric-based and leader-informed) are available online at https://doi.org/10.5281/zenodo.3534258

  2. http://slack.com

References

  • Amrit C, Daneva M, Damian D (2014) Human factors in software development: On its underlying theories and the value of learning from related disciplines. a guest editorial introduction to the special issue. Inf Softw Technol (IST) 56(12):1537–1542

    Article  Google Scholar 

  • de Aquino Junior G, Meira S (2009) Towards effective productivity measurement in software projects. In: Proceedings of the 4th International Conference on Software Engineering Advances (ICSEA), pp 241–249

  • Basili V, Rombach HD (1988) The tame project: Towards improvement-oriented software environments. IEEE Trans Softw Eng (TSE) 14(6):758–773

    Article  Google Scholar 

  • Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code! examining the effects of ownership on software quality. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE), pp 4–14

  • Boehm B, Clark B, Horowitz E, Westland C, Madachy R, Selby R (1995) Cost models for future software life cycle processes: COCOMO 2.0. Ann Softw Eng 1(1):57–94. https://doi.org/10.1007/BF02249046

    Article  Google Scholar 

  • Boehm BW (1981) Software engineering economics. Prentice Hall, Englewood Cliffs

    MATH  Google Scholar 

  • Chand D, Gowda R (1993) An exploration of the impact of individual and group factors on programmer productivity. In: Proceedings of the Conference On Computer Science (CSC), pp 338–345

  • Chávez A, Ferreira I, Fernandes E, Cedrim D, Garcia A (2017) How does refactoring affect internal quality attributes? a multi-project study. In: Proceedings of the 31st Brazilian Symposium On Software Engineering (SBES), pp 74–83

  • Chrissis MB, Konrad M, Shrum S (2011) CMMI For development: guidelines for process integration and product improvement. Pearson Education

  • Cohen J (1988) Statistical power analysis for the behavioural sciences, 2nd edn. Routledge, Evanston

    MATH  Google Scholar 

  • Croux C, Dehon C (2010) Influence functions of the spearman and kendall correlation measures. Stat Methods Appl 19(4):497–515

    Article  MathSciNet  Google Scholar 

  • Cruzes D, Dyba T (2011) Recommended steps for thematic synthesis in software engineering. In: Proceedings of the 5th international symposium on Empirical Software Engineering and Measurement (ESEM), pp 275–284

  • De Silva L, Balasubramaniam D (2012) Controlling software architecture erosion: a survey. J Syst Softw (JSS) 85(1):132–151

    Article  Google Scholar 

  • Dyer R, Nguyen HA, Rajan H, Nguyen T (2015) Boa: Ultra-large-scale software repository and source-code mining. ACM Trans Softw Eng Methodol (TOSEM) 25(1):7

    Article  Google Scholar 

  • Fenton NE, Neil M (2000) Software metrics: Roadmap. Proc Conf Futur Softw Eng ICSE 2000:357–370. https://doi.org/10.1145/336512.336588

    Article  Google Scholar 

  • Fernandes E, Ferreira LP, Figueiredo E, Valente MT (2017) How clear is your code? an empirical study with programming challenges. In: Proceedings of the Ibero-American conference on software engineering: experimental software engineering track (CIbSE-ESELAW), pp 1–14

  • Fernandes E, Oliveira J, Vale G, Paiva T, Figueiredo E (2016) A review-based comparative study of bad smell detection tools. In: Proceedings of the 20th international conference on Evaluation And Assessment In Software Engineering (EASE), pp 18:1–18:12

  • Ferreira AIF, Santos G, Cerqueira R, Montoni M, Barreto A, Barreto AOS, Rocha AR (2007) Applying iso 9001: 2000, mps. br and cmmi to achieve software process maturity: Bl informatica’s pathway. In: 29Th International Conference On Software Engineering (ICSE’07). IEEE, pp 642–651

  • Ferreira M, Valente MT, Ferreira K (2017) A comparison of three algorithms for computing truck factors. In: Proceedings of the 25th International Conference On Program Comprehension (ICPC), pp 207–217

  • Field A (2009) Discovering Statistics using IBM SPSS Statistics, 2nd edn. Sage Publications Ltd, Thousand Oaks, California 91320

    Google Scholar 

  • Fowler M (2003) Cannot measure productivity. https://martinfowler.com/bliki/CannotMeasureProductivity.html. [Online; posted 29-August-2003]

  • Furey S (1997) Why we should use function points. IEEE Softw 14(2):28

    Article  Google Scholar 

  • Gilpin AR (1993) Table for conversion of kendall’s tau to spearman’s rho within the context of measures of magnitude of effect for meta-analysis. Educ Psychol Meas 53 (1):87–92. https://doi.org/10.1177/0013164493053001007

    Article  MathSciNet  Google Scholar 

  • Gousios G, Kalliamvakou E, Spinellis D (2008) Measuring developer contribution from software repository data. In: Proceedings of the 5th international working conference on Mining Software Repositories (MSR), pp 19–132

  • Greiler M, Herzig K, Czerwonka J (2015) Code ownership and software quality: a replication study. In: Proceedings of the 12th working conference on Mining Software Repositories (MSR), pp 2–12

  • Halstead M (1977) Elements of software science, 1st edn. Elsevier, Amsterdam

    MATH  Google Scholar 

  • Hernȧndez-Lȯpez A, Colomo-Palacios R, Garci̇a-Crespo A (2013) Software engineering job productivity – a systematic review. Int J Softw Eng Knowl Eng 23 (03):387–406. https://doi.org/10.1142/S0218194013500125

    Article  Google Scholar 

  • Huselid M (1995) The impact of human resource management practices on turnover, productivity, and corporate financial performance. Acad Manag J (AMJ) 38(3):635–672

    Google Scholar 

  • International Standard Organization (2015) ISO 9001:2015 Quality management systems – Requirements. https://www.iso.org/standard/62085.html

  • Kendall MG (1955) Rank correlation methods, 2nd edn. Charles Griffin & Co. Ltd., London

    MATH  Google Scholar 

  • Lawrence M (1981) Programming methodology, organizational environment, and programming productivity. J Syst Softw (JSS) 2(3):257–269

    Article  Google Scholar 

  • Levenshtein V (1966) Binary codes capable of correcting deletions, insertions, and reversals. Dokl Phys 10(8):707–710

    MathSciNet  Google Scholar 

  • Lokan C (1999) An empirical study of the correlations between function point elements. In: Proceedings of the 6th international software metrics symposium (METRICS), pp 200–206

  • Lorenz M, Kidd J (1994) Object-oriented software metrics: A practical guide, 1st edn., Prentice Hall, Englewood Cliffs

  • Melo C, Cruzes D, Kon F, Conradi R (2011) Agile team perceptions of productivity factors. In: Proceedings of the Agile Conference (Agile2011), pp 57–66

  • Meyer A, Fritz T, Murphy G, Zimmermann T (2014) Software developers’ perceptions of productivity. In: Proceedings of the 22nd international symposium on Foundations of Software Engineering (FSE), pp 19–29

  • Mockus A, Fielding R, Herbsleb J (2002) Two case studies of open source software development: Apache and Mozilla. ACM Trans Softw Eng Methodol (TOSEM) 11(3):309–346

    Article  Google Scholar 

  • Mordal K, Anquetil N, Laval J, Serebrenik A, Vasilescu B, Ducasse S (2013) Software quality metrics aggregation in industry. Soft Evol Process (S:E&P) 25 (10):1117–1135

    Article  Google Scholar 

  • Munson J, Elbaum S (1998) Code churn: a measure for estimating the impact of code change. In: Proceedings of the 6th International Conference on Software Maintenance (ICSM), pp 24–31

  • Oliveira E, Conte T, Cristo M, Mendes E (2016) Software project managers’ perceptions of productivity factors: Findings from a qualitative study. In: Proceedings of the 10th international symposium on Empirical Software Engineering and Measurement (ESEM), pp 15:1–15:6

  • Oliveira E, Viana D, Cristo M, Conte T (2017) How have software engineering researchers been measuring software productivity? a systematic mapping study. In: Proceedings of the 19th International Conference on Enterprise Information Systems (ICEIS), pp 76–87

  • Oliveira R, de Mello R, Fernandes E, Garcia A, Lucena C (2020) Collaborative or individual identification of code smells? on the effectiveness of novice and professional developers. Inf Softw Technol (IST) 106242:120

    Google Scholar 

  • Ordonez M, Haddad H (2008) The state of metrics in software industry. In: Proceedings of the 5th international conference on Information Technology: New Generations (ITNG), pp 453–458

  • Petersen K (2011) Measuring and predicting software productivity: a systematic map and review. Inf Softw Technol (IST) 53(4):317–343

    Article  Google Scholar 

  • Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 432–441

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empiric Soft Eng (EMSE) 14(2):131

    Article  Google Scholar 

  • Runeson P, Host M, Rainer A, Regnell B (2012) Case study research in software engineering: Guidelines and examples, 1st edn. Wiley, New York

    Book  Google Scholar 

  • Scholtes I, Mavrodiev P, Schweitzer F (2016) From Aristotle to ringelmann: A large-scale analysis of team productivity and coordination in Open Source Software projects. Empiric Soft Eng (EMSE) 21(2):642–683

    Article  Google Scholar 

  • Sheetz S, Henderson D, Wallace L (2009) Understanding developer and manager perceptions of function points and source lines of code. J Syst Softw (JSS) 82 (9):1540–1549

    Article  Google Scholar 

  • Thongtanunam P, McIntosh S, Hassan A, Iida H (2016) Revisiting code ownership and its relationship with software quality in the scope of modern code review. In: Proceedings of the 38th International Conference on Software Engineering (ICSE), pp 1039–1050

  • Trendowicz A, Münch J (2009) Factors influencing software development productivity: State-of-the-art and industrial experiences. Adv Comput 77:185–241

    Article  Google Scholar 

  • Verner J, Babar M, Cerpa N, Hall T, Beecham S (2014) Factors that motivate software engineering teams: a four country empirical study. J Syst Softw (JSS) 92(1):115–127

    Article  Google Scholar 

  • Wagner S, Ruhe M (2008) A systematic review of productivity factors in software development. In: Proceedings of the 2nd international workshop on Software Productivity Analysis And Cost Estimation (SPACE), pp 1–6

  • Weber KC, Araújo EE, da Rocha ARC, Machado CA, Scalet D, Salviano CF (2005) Brazilian software process reference model and assessment method. In: International symposium on computer and information sciences. Springer, pp 402–411

  • Wen M, Wu R, Cheung SC (2016) Locus: Locating bugs from software changes. In: Proceedings of the 31st international conference on Automated Software Engineering (ASE), pp 262–273

  • Wloka J, Ryder B, Tip F, Ren X (2009) Safe-commit analysis to facilitate team software development. In: Proceedings of the 31st International Conference on Software Engineering (ICSE), pp 507–517

  • Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2012) Experimentation in software engineering, 1st edn. Springer Science & Business Media, Berlin

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Federal University of Amazonas (UFAM), Manaus, Brazil

    Edson Oliveira, Marco Cristo & Tayana Conte

  2. Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil

    Eduardo Fernandes & Alessandro Garcia

  3. Federal University of Technology – Paraná (UTFPR), Curitiba, Brazil

    Igor Steinmacher

  4. Northern Arizona University, Flagstaff, AZ, USA

    Igor Steinmacher

Authors
  1. Edson Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eduardo Fernandes
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Igor Steinmacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marco Cristo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tayana Conte
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alessandro Garcia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edson Oliveira.

Additional information

Communicated by: Gabriele Bavota

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We thank the financial support from SEFAZ/AM, UFAM, CNPq via grants 430642/2016-4, 423149/2016-4, 311494/2017-0, 204081/2018-1/PDE, 465614/2014-0, 308380/2016-9 and 434969/2018-4, CAPES via grants 175956/2013, 175956, 117875 and 153363/2018-5, FAPERJ via grants E-26/200.773/2019, 102166/2013, 225207/2016, 211033/2019, 202621/2019, National Science Foundation #1815503. Finally, we also thank the participating organizations and their employees, and the support of USES Research Group members.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Oliveira, E., Fernandes, E., Steinmacher, I. et al. Code and commit metrics of developer productivity: a study on team leaders perceptions. Empir Software Eng 25, 2519–2549 (2020). https://doi.org/10.1007/s10664-020-09820-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-020-09820-z

Keywords

Search

Navigation