Advertisement

Empirical Software Engineering

, Volume 23, Issue 5, pp 2795–2828 | Cite as

Does syntax highlighting help programming novices?

  • Christoph Hannebauer
  • Marc Hesenius
  • Volker Gruhn
Article

Abstract

Program comprehension is an important skill for programmers – extending and debugging existing source code is part of the daily routine. Syntax highlighting is one of the most common tools used to support developers in understanding algorithms. However, most research in this area originates from a time when programmers used a completely different tool chain. We examined the influence of syntax highlighting on novices’ ability to comprehend source code. Additional analyses cover the influence of task type and programming experience on the code comprehension ability itself and its relation to syntax highlighting. We conducted a controlled experiment with 390 undergraduate students in an introductory Java programming course. We measured the correctness with which they solved small coding tasks. Each test subject received some tasks with syntax highlighting and some without. The data provided no evidence that syntax highlighting improves novices’ ability to comprehend source code. There are very few similar experiments and it is unclear as of yet which factors impact the effectiveness of syntax highlighting. One major limitation may be the types of tasks chosen for this experiment. The results suggest that syntax highlighting squanders a feedback channel from the IDE to the programmer that can be used more effectively.

Keywords

Syntax highlighting Source code typography Code colouring IDE interface Program comprehension 

Notes

Acknowledgments

We thank all students enrolled in the course Programming in Java for their participation in our experiment. We also thank Matthias Book, Tobias Brückmann, and Tobias Griebe for useful comments on earlier draughts of this paper. We further thank Florian Stefan and again Matthias Book for their help setting up and supervising the experiment. We thank Stefan Hanenberg for his feedback on the paper and for the discussions about statistics. We furthermore thank the anonymous reviewers for their valuable feedback.

References

  1. Allen E, Cartwright R, Stoler B (2002) Drjava: a lightweight pedagogic environment for java. ACM SIGCSE Bullet. 34(1):137.  https://doi.org/10.1145/563517.563395 CrossRefGoogle Scholar
  2. Baecker RM, Marcus A (1989) Human factors and typography for more readable programs. ACM, New YorkGoogle Scholar
  3. Beelders TR, du Plessis JPL (2015) Syntax highlighting as an influencing factor when reading and comprehending source code. J Eye Mov Res 9(1).  https://doi.org/10.16910/jemr.9.1.1
  4. Bergin S, Reilly R (2005) Programming: Factors that influence success. SIGCSE Bull 37(1):411–415.  https://doi.org/10.1145/1047124.1047480 CrossRefGoogle Scholar
  5. Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code!: Examining the effects of ownership on software quality. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ACM, New York, ESEC/FSE ’11.  https://doi.org/10.1145/2025113.2025119, pp 4–14
  6. Chaffin WW, Rhiel SG (1993) The effect of skewness and kurtosis on the one-sample t test and the impact of knowledge of the population standard deviation. J Stat Comput Simul 46(1-2):79–90.  https://doi.org/10.1080/00949659308811494 CrossRefGoogle Scholar
  7. Crosby M, Stelovsky J (1990) How do we read algorithms? a case study. Computer 23(1):25–35.  https://doi.org/10.1109/2.48797 CrossRefGoogle Scholar
  8. Dieste O, Aranda AM, Uyaguari F, Turhan B, Tosun A, Fucci D, Oivo M, Juristo N (2017) Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study. Empir Softw Eng 22:1–86. ISSN:1573-7616.  https://doi.org/10.1007/s10664-016-9471-3
  9. Dimitri GM (2015) The impact of syntax highlighting in sonic pi. In: Psychology of Programming Interest Group Annual Conference 2015. Proceedings of, pp 59–70Google Scholar
  10. Dragicevic P (2016) Fair Statistical Communication in HCI. Springer International Publishing, Cham, pp 291–330.  https://doi.org/10.1007/978-3-319-26633-6_13 Google Scholar
  11. Erguler K (2013) Barnard: Barnard’s Unconditional Test. R package version 1.3. http://cran.r-project.org/package=Barnard, [accessed 2014-07-22]
  12. Feigenspan J, Kästner C, Apel S, Liebig J, Schulze M, Dachselt R, Papendieck M, Leich T, Saake G (2013) Do background colors improve program comprehension in the #ifdef hell?. Empir Softw Eng 18(4):699–745.  https://doi.org/10.1007/s10664-012-9208-x CrossRefGoogle Scholar
  13. Gellenbeck EM, Cook CR (1991) Does signaling help professional programmers read and understand computer programs?. In: Fourth Workshop on Empirical Studies of Programmers, Ablex Publishing Corporation, Norwood, pp 82–98Google Scholar
  14. Goldstein EB (1995) Sensation and Perception. Wadsworth Publishing, BelmontGoogle Scholar
  15. Grant EE, Sackman H (1967) An exploratory investigation of programmer performance under on-line and off-line conditions. IEEE Trans Hum Fact Electron HFE-8(1):33–48.  https://doi.org/10.1109/THFE.1967.233303 CrossRefGoogle Scholar
  16. Gray KE, Flatt M (2003) ProfessorJ: A gradual introduction to java through language levels. In: Companion of the 18th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, ACM Press, New York, OOPSLA ’03.  https://doi.org/10.1145/949344.949394, pp 170–177
  17. Gruhn V, Hannebauer C (2012) Using wikis as software development environments. In: Proceedings of the the 11th SoMeT_12, IOS Press, Amsterdam, Frontiers in Artificial Intelligence and Applications.  https://doi.org/10.3233/978-1-61499-125-0-3, vol 246, pp 3–20
  18. Hakala T, Nykyri P, Sajaniemi J (2006) An experiment on the effects of program code highlighting on visual search for local patterns. In: 18th Workshop of the Psychology of Programming Interest Group, Proceedings of, pp 38–52Google Scholar
  19. Hannebauer C, Hesenius M, Gruhn V (2017) Lab package for the syntax highlighting experiment. https://www.uni-due.de/~hw0433/sh/
  20. Holden E, Weeden E (2003) The impact of prior experience in an information technology programming course sequence. In: Proceedings of the 4th Conference on Information Technology Curriculum, ACM, New York, CITC4 ’03.  https://doi.org/10.1145/947121.947131, pp 41–46
  21. Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. Springer, Berlin, pp 201–228Google Scholar
  22. Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734.  https://doi.org/10.1109/TSE.2002.1027796 CrossRefGoogle Scholar
  23. Kleinschmager S, Hanenberg S (2011) How to rate programming skills in programming experiments?: A preliminary, exploratory, study based on university marks, pretests, and self-estimation. In: Proceedings of the 3rd ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools, ACM, New York, PLATEAU ’11.  https://doi.org/10.1145/2089155.2089161, pp 15–24
  24. Knuth DE (1984) Literate programming. Comput J 27(2):97–111.  https://doi.org/10.1093/comjnl/27.2.97 CrossRefzbMATHGoogle Scholar
  25. van Laar D (1989) Evaluating a colour coding programming support tool. In: Sutcliffe A, Macaulay L (eds) Proceedings of the Fifth Conference of the British Computer Society Human-Computer Interaction Specialist Group, Cambridge University Press. British Computer Society Workshop Series, Cambridge, pp 217–230Google Scholar
  26. Lientz BP, Swanson EB, Tompkins GE (1978) Characteristics of application software maintenance. Commun ACM 21(6):466–471.  https://doi.org/10.1145/359511.359522 CrossRefGoogle Scholar
  27. LPLUS GmbH (2014) LPLUS-System on-campus. http://lplus.de/lplus.de/index_9_19_2__.html, [accessed 2014-07-16]
  28. Ludbrook J (2008) Analysis of 2 x 2 tables of frequencies: matching test to experimental design. Int J Epidemiol 37(6):1430–1435.  https://doi.org/10.1093/ije/dyn162 CrossRefGoogle Scholar
  29. Mehta R, Zhu RJ (2009) Blue or red? exploring the effect of color on cognitive task performances. Science 323(5918):1226–1229.  https://doi.org/10.1126/science.1169144 CrossRefGoogle Scholar
  30. Microsoft Developer Network (2015) C# coding conventions (c# programming guide). http://msdn.microsoft.com/en-us/library/ff926074.aspx, [accessed 2017-03-08]
  31. Oman PW, Cook CR (1990) Typographic style is more than cosmetic, vol 33.  https://doi.org/10.1145/78607.78611
  32. Oracle Technology Network (1999) Code conventions for the java programming language. http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html, [accessed 2017-03-08]
  33. Prechelt L (1999) The 28:1 Grant/Sackman legend is misleading, or: How large is interpersonal variation really? Interner Bericht, Fakultät für Informatik. Universität Karlsruhe, KarlsruheGoogle Scholar
  34. Rambally GK (1986) The influence of color on program readability and comprehensibility. In: Proceedings of the Seventeenth SIGCSE Technical Symposium on Computer Science Education, ACM, New York, SIGCSE ’86.  https://doi.org/10.1145/5600.5702, pp 173–181
  35. Reijers H, Freytag T, Mendling J, Eckleder A (2011) Syntax highlighting in business process models. Decis Support Syst 51(3):339 – 349.  https://doi.org/10.1016/j.dss.2010.12.013 CrossRefGoogle Scholar
  36. Sackman H, Erikson WJ, Grant EE (1968) Exploratory experimental studies comparing online and offline programming performance. Commun ACM 11(1):3–11.  https://doi.org/10.1145/362851.362858 CrossRefGoogle Scholar
  37. Sarkar A (2015) The impact of syntax colouring on program comprehension. In: Psychology of Programming Interest Group Annual Conference 2015, Proceedings of, pp 49–58Google Scholar
  38. Scintilla Project (2014) Scintilla – a free source code editing component for win32, gtk+, and os x. http://www.scintilla.org, [accessed 2014-07-22]
  39. Seo H, Sadowski C, Elbaum S, Aftandilian E, Bowdidge R (2014) Programmers’ build errors: A case study (at google). In: Proceedings of the 36th International Conference on Software Engineering, ACM, New York, ICSE 2014.  https://doi.org/10.1145/2568225.2568255, pp 724–734
  40. Siegmund J, Kästner C, Liebig J, Apel S, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19(5):1299–1334.  https://doi.org/10.1007/s10664-013-9286-4 CrossRefGoogle Scholar
  41. Tapp R, Kazman R (1994) Determining the usefulness of colour and fonts in a programming task. In: Proceedings of the Third IEEE Workshop on Program Comprehension.  https://doi.org/10.1109/WPC.1994.341265. IEEE Computer Society Press, Los Alamitos, pp 154–161
  42. Tiarks R (2011) What programmers really do: an observational study. Softwaretechnik-Trends 31(2):36–37Google Scholar
  43. Wallis WA (1942) Compounding probabilities from independent significance tests. Econometrica, J Econ Soc 10(3/4):229–248MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018
corrected publication May/2018

Authors and Affiliations

  1. 1.paluno – The Ruhr Institute for Software TechnologyUniversity of Duisburg-EssenEssenGermany

Personalised recommendations