Abstract
Program comprehension is an important skill for programmers – extending and debugging existing source code is part of the daily routine. Syntax highlighting is one of the most common tools used to support developers in understanding algorithms. However, most research in this area originates from a time when programmers used a completely different tool chain. We examined the influence of syntax highlighting on novices’ ability to comprehend source code. Additional analyses cover the influence of task type and programming experience on the code comprehension ability itself and its relation to syntax highlighting. We conducted a controlled experiment with 390 undergraduate students in an introductory Java programming course. We measured the correctness with which they solved small coding tasks. Each test subject received some tasks with syntax highlighting and some without. The data provided no evidence that syntax highlighting improves novices’ ability to comprehend source code. There are very few similar experiments and it is unclear as of yet which factors impact the effectiveness of syntax highlighting. One major limitation may be the types of tasks chosen for this experiment. The results suggest that syntax highlighting squanders a feedback channel from the IDE to the programmer that can be used more effectively.
Similar content being viewed by others
Notes
Microsoft Visual Studio and Eclipse still can print out source code, though. Interestingly, Eclipse retains syntax highlighting in the print-out and Visual Studio removes it.
The *.cxx files in lexer and lexlib contain 38,482 and 908 (LOC), respectively, whereas the whole project comprises 75,182 (LOC) in version 225.
References
Allen E, Cartwright R, Stoler B (2002) Drjava: a lightweight pedagogic environment for java. ACM SIGCSE Bullet. 34(1):137. https://doi.org/10.1145/563517.563395
Baecker RM, Marcus A (1989) Human factors and typography for more readable programs. ACM, New York
Beelders TR, du Plessis JPL (2015) Syntax highlighting as an influencing factor when reading and comprehending source code. J Eye Mov Res 9(1). https://doi.org/10.16910/jemr.9.1.1
Bergin S, Reilly R (2005) Programming: Factors that influence success. SIGCSE Bull 37(1):411–415. https://doi.org/10.1145/1047124.1047480
Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code!: Examining the effects of ownership on software quality. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ACM, New York, ESEC/FSE ’11. https://doi.org/10.1145/2025113.2025119, pp 4–14
Chaffin WW, Rhiel SG (1993) The effect of skewness and kurtosis on the one-sample t test and the impact of knowledge of the population standard deviation. J Stat Comput Simul 46(1-2):79–90. https://doi.org/10.1080/00949659308811494
Crosby M, Stelovsky J (1990) How do we read algorithms? a case study. Computer 23(1):25–35. https://doi.org/10.1109/2.48797
Dieste O, Aranda AM, Uyaguari F, Turhan B, Tosun A, Fucci D, Oivo M, Juristo N (2017) Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study. Empir Softw Eng 22:1–86. ISSN:1573-7616. https://doi.org/10.1007/s10664-016-9471-3
Dimitri GM (2015) The impact of syntax highlighting in sonic pi. In: Psychology of Programming Interest Group Annual Conference 2015. Proceedings of, pp 59–70
Dragicevic P (2016) Fair Statistical Communication in HCI. Springer International Publishing, Cham, pp 291–330. https://doi.org/10.1007/978-3-319-26633-6_13
Erguler K (2013) Barnard: Barnard’s Unconditional Test. R package version 1.3. http://cran.r-project.org/package=Barnard, [accessed 2014-07-22]
Feigenspan J, Kästner C, Apel S, Liebig J, Schulze M, Dachselt R, Papendieck M, Leich T, Saake G (2013) Do background colors improve program comprehension in the #ifdef hell?. Empir Softw Eng 18(4):699–745. https://doi.org/10.1007/s10664-012-9208-x
Gellenbeck EM, Cook CR (1991) Does signaling help professional programmers read and understand computer programs?. In: Fourth Workshop on Empirical Studies of Programmers, Ablex Publishing Corporation, Norwood, pp 82–98
Goldstein EB (1995) Sensation and Perception. Wadsworth Publishing, Belmont
Grant EE, Sackman H (1967) An exploratory investigation of programmer performance under on-line and off-line conditions. IEEE Trans Hum Fact Electron HFE-8(1):33–48. https://doi.org/10.1109/THFE.1967.233303
Gray KE, Flatt M (2003) ProfessorJ: A gradual introduction to java through language levels. In: Companion of the 18th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, ACM Press, New York, OOPSLA ’03. https://doi.org/10.1145/949344.949394, pp 170–177
Gruhn V, Hannebauer C (2012) Using wikis as software development environments. In: Proceedings of the the 11th SoMeT_12, IOS Press, Amsterdam, Frontiers in Artificial Intelligence and Applications. https://doi.org/10.3233/978-1-61499-125-0-3, vol 246, pp 3–20
Hakala T, Nykyri P, Sajaniemi J (2006) An experiment on the effects of program code highlighting on visual search for local patterns. In: 18th Workshop of the Psychology of Programming Interest Group, Proceedings of, pp 38–52
Hannebauer C, Hesenius M, Gruhn V (2017) Lab package for the syntax highlighting experiment. https://www.uni-due.de/~hw0433/sh/
Holden E, Weeden E (2003) The impact of prior experience in an information technology programming course sequence. In: Proceedings of the 4th Conference on Information Technology Curriculum, ACM, New York, CITC4 ’03. https://doi.org/10.1145/947121.947131, pp 41–46
Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. Springer, Berlin, pp 201–228
Kitchenham B, Pfleeger S, Pickard L, Jones P, Hoaglin D, El Emam K, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28(8):721–734. https://doi.org/10.1109/TSE.2002.1027796
Kleinschmager S, Hanenberg S (2011) How to rate programming skills in programming experiments?: A preliminary, exploratory, study based on university marks, pretests, and self-estimation. In: Proceedings of the 3rd ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools, ACM, New York, PLATEAU ’11. https://doi.org/10.1145/2089155.2089161, pp 15–24
Knuth DE (1984) Literate programming. Comput J 27(2):97–111. https://doi.org/10.1093/comjnl/27.2.97
van Laar D (1989) Evaluating a colour coding programming support tool. In: Sutcliffe A, Macaulay L (eds) Proceedings of the Fifth Conference of the British Computer Society Human-Computer Interaction Specialist Group, Cambridge University Press. British Computer Society Workshop Series, Cambridge, pp 217–230
Lientz BP, Swanson EB, Tompkins GE (1978) Characteristics of application software maintenance. Commun ACM 21(6):466–471. https://doi.org/10.1145/359511.359522
LPLUS GmbH (2014) LPLUS-System on-campus. http://lplus.de/lplus.de/index_9_19_2__.html, [accessed 2014-07-16]
Ludbrook J (2008) Analysis of 2 x 2 tables of frequencies: matching test to experimental design. Int J Epidemiol 37(6):1430–1435. https://doi.org/10.1093/ije/dyn162
Mehta R, Zhu RJ (2009) Blue or red? exploring the effect of color on cognitive task performances. Science 323(5918):1226–1229. https://doi.org/10.1126/science.1169144
Microsoft Developer Network (2015) C# coding conventions (c# programming guide). http://msdn.microsoft.com/en-us/library/ff926074.aspx, [accessed 2017-03-08]
Oman PW, Cook CR (1990) Typographic style is more than cosmetic, vol 33. https://doi.org/10.1145/78607.78611
Oracle Technology Network (1999) Code conventions for the java programming language. http://www.oracle.com/technetwork/java/javase/documentation/codeconvtoc-136057.html, [accessed 2017-03-08]
Prechelt L (1999) The 28:1 Grant/Sackman legend is misleading, or: How large is interpersonal variation really? Interner Bericht, Fakultät für Informatik. Universität Karlsruhe, Karlsruhe
Rambally GK (1986) The influence of color on program readability and comprehensibility. In: Proceedings of the Seventeenth SIGCSE Technical Symposium on Computer Science Education, ACM, New York, SIGCSE ’86. https://doi.org/10.1145/5600.5702, pp 173–181
Reijers H, Freytag T, Mendling J, Eckleder A (2011) Syntax highlighting in business process models. Decis Support Syst 51(3):339 – 349. https://doi.org/10.1016/j.dss.2010.12.013
Sackman H, Erikson WJ, Grant EE (1968) Exploratory experimental studies comparing online and offline programming performance. Commun ACM 11(1):3–11. https://doi.org/10.1145/362851.362858
Sarkar A (2015) The impact of syntax colouring on program comprehension. In: Psychology of Programming Interest Group Annual Conference 2015, Proceedings of, pp 49–58
Scintilla Project (2014) Scintilla – a free source code editing component for win32, gtk+, and os x. http://www.scintilla.org, [accessed 2014-07-22]
Seo H, Sadowski C, Elbaum S, Aftandilian E, Bowdidge R (2014) Programmers’ build errors: A case study (at google). In: Proceedings of the 36th International Conference on Software Engineering, ACM, New York, ICSE 2014. https://doi.org/10.1145/2568225.2568255, pp 724–734
Siegmund J, Kästner C, Liebig J, Apel S, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19(5):1299–1334. https://doi.org/10.1007/s10664-013-9286-4
Tapp R, Kazman R (1994) Determining the usefulness of colour and fonts in a programming task. In: Proceedings of the Third IEEE Workshop on Program Comprehension. https://doi.org/10.1109/WPC.1994.341265. IEEE Computer Society Press, Los Alamitos, pp 154–161
Tiarks R (2011) What programmers really do: an observational study. Softwaretechnik-Trends 31(2):36–37
Wallis WA (1942) Compounding probabilities from independent significance tests. Econometrica, J Econ Soc 10(3/4):229–248
Acknowledgments
We thank all students enrolled in the course Programming in Java for their participation in our experiment. We also thank Matthias Book, Tobias Brückmann, and Tobias Griebe for useful comments on earlier draughts of this paper. We further thank Florian Stefan and again Matthias Book for their help setting up and supervising the experiment. We thank Stefan Hanenberg for his feedback on the paper and for the discussions about statistics. We furthermore thank the anonymous reviewers for their valuable feedback.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Sven Apel
Rights and permissions
About this article
Cite this article
Hannebauer, C., Hesenius, M. & Gruhn, V. Does syntax highlighting help programming novices?. Empir Software Eng 23, 2795–2828 (2018). https://doi.org/10.1007/s10664-017-9579-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-017-9579-0