Skip to main content
SpringerLink
Log in

On the Investigation of Empirical Contradictions - Aggregated Results of Local Studies on Readability and Comprehensibility of Source Code

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Context

Reading and understanding the source code are fundamental to supporting software programmers’ daily activities. Still, there is no agreement regarding the program attributes needed to achieve the readability and comprehensibility of source code.

Objective

To analyze the influence of comments presence, indentation spacing, identifiers length, and code size on the readability and comprehensibility of source code from the perspective of novice and experienced programmers.

Method

We performed three primary studies and collected quantitative (Likert) and qualitative data representing the programmers’ judgment regarding the readability and comprehensibility of code snippets. For each study, the influence of the four attributes on the readability and comprehensibility of source code was analyzed. The results were aggregated using the odds-ratio approach and analyzed concerning participants’ programming experience.

Results

The quality characteristics were not significantly affected (alpha = 5%) by either the indentation spacing or the code size, whereas the presence of comments and identifier length affect source code quality positively under such characteristics, according to both novices and experienced programmers.

Conclusions

Although the results presented findings with statistical significance, the controlled factors and participants’ stratification between novices and experienced were insufficient to explain the contradictory findings in the technical literature concerning the impact of the attributes under evaluation on the readability and comprehensibility of source code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The datasets generated and analyzed during the current work are openly available in the Figshare repository at https://doi.org/10.6084/m9.figshare.22791104, reference number 22791104.

Notes

  1. https://github.com/

  2. There are three source code attributes associated to each quality characteristic (see Section 3.1.4 for the list of quality characteristics evaluated for each attribute). Considering we are going to analyze results from three studies (observing that the analysis of S1 was split into two –novices and experienced), the 48 statistical tests come from 2 (quality characteristics) × 3 (source code attributes) × 4 (three studies with one split analysis) × 2 (perceptions and explanations/answers).

  3. https://www.python.org/dev/peps/pep-0008/#indentation

  4. https://google.github.io/styleguide/pyguide.html#Indentation

References

  • Alawad D, Panta M, Zibran M, Islam M (2019) "An empirical study of the relationships between code readability and software complexity," in arXiv preprint arXiv:1909.01760

  • Aman H (2012a) "An empirical analysis of the impact of comment statements on fault-proneness of small-size module," in Proceedings of the Asia-Pacific Software Engineering Conference, Hong Kong

  • Aman H (2012b) "An empirical analysis on fault-proneness of well-commented modules," in Proceedings of the International Workshop on Empirical Software Engineering in Practice, Osaka

  • Aman H, Amasaki S, Sasaki T, Kawahara M (2015) Lines of comments as a noteworthy metric for analyzing fault-proneness in method. IEICE Trans Inf Syst:2218–2228

  • Aman H, Amasaki S, Yokogawa T, Kawahara M (2016) "Local variables with compound names and comments as signs of fault-prone Java methods," in Proceedings of the International Workshop on Quantitative Approaches to Software Quality, Hamilton

  • Aman H, Amasaki S, Yokogawa T, Kawahara M (2017) "Empirical analysis of words in comments written for Java methods," in Proceedings of the Euromicro Conference on Software Engineering and Advanced Applications, Vienna

  • Avidan E, Feitelson DG (2017) "Effects of variable names on comprehension: an empirical study," in Proceedings of the International Conference on Program Comprehension, Buenos Aires

  • Basili VR, Caldiera G, Rombach HD (1994) Goal question metric paradigm, in Encyclopedia of Software Engineering. Wiley, New York, pp 528–532

    Google Scholar 

  • Bauer J, Siegmund J, Peitek N, Hofmeister JC, Apel S (2019) "Indentation: simply a matter of style or support for program comprehension?," in Proceedings of the International Conference on Program Comprehension, Montreal

  • Beniamini G, Gingichashvili S, Orbach AK, Feitelson DG (2017) "Meaningful identifier names: the case of single-letter variables," in Proceedings of the International Conference on Program Comprehension, Buenos Aires

  • Binkley D, Feild H, Lawrie D, Pighin M (2007) "Software fault prediction using language processing," in Testing: Academic and Industrial Conference Practice and Research Techniques, Windsor

  • Binkley D, Feild H, Lawrie D, Pighin M (2009) Increasing diversity: natural language measures for software fault prediction. J Syst Softw 82:1793–1803

    Article  Google Scholar 

  • Borenstein M, Gedges LV, Higgins JPT, Rothstein HR (2009) Introduction to meta-analysis. Wiley, West Sussex

    Book  MATH  Google Scholar 

  • Börstler J, Paech B (2016) The role of method chains and comments in software readability and comprehension - An experiment. Transactions on Software Engineering 9:886–898

    Article  Google Scholar 

  • Bosch J (2014) "Continuous software engineering: an introduction," in Continuous Software Engineering, Springer, pp. 3–13

  • Buse RPL, Weimer WR (2008) "A metric for software readability," in Proceedings of the International Symposium on Software Testing and Analysis, Seattle

  • Buse RPL, Weimer WR (2010) Learning a metric for code readability. Transactions on Software Engineering 36:546–558

    Article  Google Scholar 

  • Butler S, Wermelinger M, Yu Y, Sharp H (2010) "Exploring the influence of identifier names on code quality: An empirical study," in Proceedings of European Conference on Software Maintenance and Reengineering, Madrid

  • Cohen J (1992) A power primer. Psychol Bull 112:155–159

    Article  Google Scholar 

  • DeYoung GE, Kampen GR (1979) "Program factors as predictors of program readability," in Proceedings of the International Computer Software and Applications Conference, Chicago

  • Eaddy M, Zimmermann T, Sherwood KD, Garg V, Murphy GC, Nagappan N, Aho AV (2008) Do crosscutting concerns cause defects? Transactions on Software Engineering 34:497–515

    Article  Google Scholar 

  • Feigenspan J, Apel S, Liebig J, Kästner C (2011) "Exploring software measures to assess program comprehension," in Proceedings of the International Symposium on Empirical Software Engineering and Measurement, Banff

  • Garvin D (1984) What does “product quality” really mean? Sloan Manag Rev 25:25–43

    Google Scholar 

  • Glass RL (2002) Facts and fallacies of software engineering, 1 ed. Addison-Wesley, Boston, Massachusetts, p 224

    Google Scholar 

  • Guerrouj L, Kermansaravi Z, Arnaoudova V, Fung B, Khomh F, Antoniol G, Guéhéneuc Y-G (2017) Investigating the relation between lexical smells and change and fault-proneness: an empirical study. Softw Qual J 25:641–670

    Article  Google Scholar 

  • Hofmeister J, Siegmund J, Holt DV (2017) Shorter identifier names take longer to comprehend, in International Conference on Software Analysis. Evolution and Reengineering, Klagenfurt

    Google Scholar 

  • Hofmeister JC, Siegmund J, Holt DV (2019) Shorter identifier names take longer to comprehend. Empir Softw Eng 24:417–443

    Article  Google Scholar 

  • Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering, in Guide to advanced empirical software engineering. Springer, London, pp 201–228

    Google Scholar 

  • Jørgensen AH (1980) A methodology for measuring the readability and modifiability of computer programs. BIT Numer Math 20:393–405

    Article  Google Scholar 

  • Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49:1073–1086

    Article  Google Scholar 

  • Lawrie D, Morrell C, Feild H, Binkley D (2006) "What's in a name? A study of identifiers," in Proceedings of the International Conference on Program Comprehension, Athens

  • Lawrie D, Morrell C, Feild H, Binkley D (2007) Effective identifier names for comprehension and memory. Innov Syst Softw Eng 3:303–318

    Article  Google Scholar 

  • Lee T, Lee J, In H (2015) Effect analysis of coding convention violations on readability of post-delivered code. IEICE Trans Inf Syst E98-D:1286–1296

    Article  Google Scholar 

  • Lewis S, Clarke M (2001) Forest plots: trying to see the wood and the trees. BMJ 322:1479–1480

    Article  Google Scholar 

  • Lientz BP, Swanson EB, Tompkins AGE (1978) Characteristics of application software maintenance. Communications of ACM 21:466–471

    Article  Google Scholar 

  • Lipsey MW, Wilson DB (2001) Practical meta-analysis - applied social research methods series, vol. 49. Sage Publications, London

    Google Scholar 

  • Martínez-Fernández S, Santos PSMD, Ayala CP, Franch X, Travassos GH (2015) "Aggregating empirical evidence about the benefits and drawbacks of software reference architectures," in International Symposium on Empirical Software Engineering, Beijing

  • Miara RJ, Musselman JA, Navarro JA, Shneiderman B (1983) Program indentation and comprehensibility. Commun ACM 26:861–867

    Article  Google Scholar 

  • Miyake Y, Amasaki S, Aman H, Yokogawa T (2017) "A replicated study on relationship between code quality and method comments," Applied Computing and Information Technology Studies in Computational Intelligence, pp. 17–30

  • Nguyen V, Boehm B, Danphitsanuphan P (2011) A controlled experiment in assessing and estimating software maintenance tasks. Inf Softw Technol 53:682–691

    Article  Google Scholar 

  • Nielebock S, Krolikowski D, Krüger J, Leich T, Ortmeier F (2019) Commenting source code: is it worth it for small programming tasks? Empir Softw Eng 24:1418–1457

    Article  Google Scholar 

  • Olivier J, Bell ML (2013) Effect sizes for 2×2 contingency tables. PLoS One 8

  • Peitek N, Apel S, Parnin C, Brechmann A, Siegmund J (2021) "Program comprehension and code complexity metrics: an fMRI study," in International Conference on Software Engineering, Madrid

  • Petersen K (2010) "Implementing lean and agile software development in industry," Karlskrona

  • Posnett D, Hindle A, Devanbu P (2011) "A simpler model of software readability," in Proceedings of the Working Conference on Mining Software Repositories, Waikiki

  • Raymond DR (1991) "Reading source code," in Proceedings of the Conference of the Centre for Advanced Studies on Collaborative Research, Toronto

  • Ribeiro TV, Travassos GH (2015) "On the alignment of source code quality perspectives through experimentation: an industrial case," in Proceedings of the International Workshop on Conducting Empirical Studies in Industry (CESI/ICSE 2015), Florence

  • Ribeiro TV, Travassos GH (2017) "Who is right? Evaluating empirical contradictions in the readability and comprehensibility of source code," in XX Ibero-American Conference on Software Engineering, Buenos Aires

  • Ribeiro TV, Travassos GH (2018) Attributes influencing the reading and comprehension of source code – discussing contradictory evidence. CLEI Electron J 21

  • Rugaber S (2000) The use of domain knowledge in program understanding. Ann Softw Eng 9:143–192

    Article  Google Scholar 

  • Salviulo F, Scanniello G (2014) Dealing with identifiers and comments in source code comprehension and maintenance: Results from an ethnographically-informed study with students and professionals. EASE '14: proceedings of the 18th international conference on evaluation and assessment in software engineering, p 10. https://doi.org/10.1145/2601248.2601251

  • Santos RMD, Gerosa MA (2018) "Impacts of coding practices on readability," in Proceedings of the International Conference on Program Comprehension, Gothenburg

  • Scalabrino S, Bavota G, Vendome C, Linares-Vásquez M, Poshyvanyk D, Oliveto R (2017) "Automatically assessing code understandability: How far are we?," in International Conference on Automated Software Engineering, Urbana

  • Scalabrino S, Bavota G, Vendome C, Linares-Vásquez M, Poshyvanyk D, Oliveto R (2021) Automatically assessing code understandability. Transactions on Software Engineering 47:595–613

    Article  Google Scholar 

  • Scalabrino S, Linares-Vásquez M, Oliveto R, Poshyvanyk D (2018) A comprehensive model for code readability. J Softw Evol Process 30:1–23

    Article  Google Scholar 

  • Scanniello G, Risi M (2013) "Dealing with faults in source code: abbreviated vs. full-word identifier names," in Proceedings of the International Conference on Software Maintenance, Eindhoven

  • Scanniello G, Risi M, Tramontana P, Romano S (2017) Fixing faults in C and Java source code: Abbreviated vs. full-word identifier names. ACM Trans Softw Eng Methodol 26:1–43

    Article  Google Scholar 

  • Siegmund J, Kästner C, Jörg Liebig SA, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19:1299–1334

    Article  Google Scholar 

  • Tashtoush Y, Odat Z, Alsmadi I, Yatim M (2013) Impact of programming features on code readability. Int J Softw Eng Appl 7:441–458

    Google Scholar 

  • Tenny T (1988) Program readability: procedures versus comments. Transactions on Software Engineering 14:1271–1279

    Article  Google Scholar 

  • Trockman A, Cates K, Cates K, Nguyen T, Kästner C, Vasilescu B (2018) ""Automatically assessing code understandability" reanalyzed: combined metrics matter," in Proceedings of the International Conference on Mining Software Repositories, Gothenburg

  • Woodfield S, Dunsmore H, Shen V (1981) "Effect of modularization and comments on program comprehension," in Proceedings of the International Conference on Software Engineering, San Diego

  • Xia X, Bao L, Xing Z, Hassan AE, Li S (2018) Measuring program comprehension: A large-scale field study with professionals. Transactions on Software Engineering 44:951–976

    Article  Google Scholar 

Download references

Funding

The authors thank CAPES and CNPq for supporting this research. Prof. Travassos is a CNPq Researcher and CNE FAPERJ.

Author information

Authors and Affiliations

  1. PESC/COPPE, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    Talita Vieira Ribeiro & Guilherme Horta Travassos

  2. Department of Applied Informatics, Federal University of the State of Rio de Janeiro, Rio de Janeiro, Brazil

    Paulo Sérgio Medeiros dos Santos

Authors
  1. Talita Vieira Ribeiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paulo Sérgio Medeiros dos Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guilherme Horta Travassos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Talita Vieira Ribeiro.

Ethics declarations

The authors thank the students of the courses in which we conducted the experiments (EEL873 – S1, CPS820 – S2, MAB114 – S3, and MAB121 – S3) at UFRJ for taking part in the three different studies.

The authors declare they have no financial interests and no competing interests to declare that are relevant to the content of this article.

Additional information

This article belongs to the Topical Collection: Special Issue on Code Legibility, Readability, and Understandability.

Appendix A

Appendix A

References for the works mentioned in the Background section.

A

DeYoung and Kampen (1979)

B

Jørgensen (1980)

C

Woodfield et al. (1981)

D

Miara et al. (1983)

E

Tenny (1988)

F

Lawrie et al. (2006)

G

Lawrie et al. (2007)

H

Buse and Weimer (2008)

I

Buse and Weimer (2010)

J

Butler et al. (2010)

K

Feigenspan et al. (2011)

L

Posnett et al. (2011)

M

Tashtoush et al. (2013)

N

Lee et al. (2015)

O

Börstler and Paech (2016)

a

Avidan and Feitelson (2017)

b

Beniamini et al. (2017)

c

Scalabrino et al. (2017)

d

Hofmeister et al. (2017)

e

Scalabrino et al. (2018)

f

Trockman et al. (2018)

g

Nielebock et al. (2019)

h

Bauer et al. (2019)

i

Hofmeister et al. (2019)

j

Scalabrino et al. (2021)

k

Peitek et al. (2021)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ribeiro, T.V., dos Santos, P.S.M. & Travassos, G.H. On the Investigation of Empirical Contradictions - Aggregated Results of Local Studies on Readability and Comprehensibility of Source Code. Empir Software Eng 28, 148 (2023). https://doi.org/10.1007/s10664-023-10360-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-023-10360-5

Keywords

Search

Navigation